The Bruces came over with the Conqueror. The theory of a Norse origin in a follower of Rollo the Ganger, who established himself in the diocese of Coutances in Manche, Normandy, though not improbable, is but vaguely supported. The name is territorial; and the better opinion is inclined to connect it with Brix, between Cherbourg and Valognes.

The first Robert de Brus on record was probably the leader of the Brus contingent in the army of the Conqueror. His services must have been conspicuous; he died (about 1094) in possession of some 40,000 acres, comprised in forty-three manors in the East and West Ridings of Yorkshire, and fifty-one in the North Riding and in Durham. The chief manor was Skelton in Cleveland.

The next Robert de Brus, son of the first, received a grant of Annandale from David I., whose companion he had been at the English court. This fief he renounced, probably in favour of his second son, just before the Battle of the Standard (1138), on the failure of his attempted mediation between David and the English barons. He died in 1141, leaving two sons, Adam and Robert.

This Robert may be regarded as the true founder of the Scottish branch. He is said to have remained with David in the Battle of the Standard, and, whether for this adherence or on some subsequent occasion, he was established in possession of the Annandale fief, which was confirmed to him by a charter of William the Lion (1166). He is said to have received from his father the manor of Hert and the lands of Hertness in Durham, 'to supply him with wheat, which did not grow in Annandale.' He died after 1189.

The second Robert de Brus of Annandale, son of the preceding lord, married (1183) Isabel, daughter of William the Lion, obtaining as her dowry the manor of Haltwhistle in Tyndale. His widow married Robert de Ros in 1191. The uncertainty as to the dates of his father's death and his own has suggested a doubt whether he ever succeeded to the lordship.

William de Brus, a brother, the next lord, died in 1215.

The third Robert de Brus of Annandale, son of William, founded the claim of his descendants to the crown by his marriage with Isabel, second daughter of David, Earl of Huntingdon, younger brother of William the Lion. He died in 1245.

The fourth Robert de Brus of Annandale, eldest son of the preceding lord, was born in 1210. In 1244, he married Isabel, daughter of Gilbert de Clare, Earl of Gloucester. Next year he succeeded to Annandale, and, on his mother's death in 1251, he obtained ten knight's fees in England, her share of the Earldom of Huntingdon. He took an active part in public affairs. In 1249–50 he sat as a Justice of the King's Bench, and in 1268 he became Chief Justice of England, but Edward, on his accession (1272), did not reappoint him. He served as Sheriff of Cumberland and Governor of Carlisle Castle in 1254–55, and in 1264 he fought for Henry at Lewes, and was taken prisoner.

At the same time, de Brus was a prominent figure in the baronage of Scotland. The alleged arrangement of 1238 whereby Alexander II., with the consent of the Scots parliament, appointed de Brus his successor in the event of his dying childless, was frustrated by the King's second marriage (1239), and the birth of a son, Alexander III. (1241). As one of the fifteen Regents (1255) during the minority of Alexander III., he headed the party that favoured an English alliance, cemented by the young King's marriage with Margaret, daughter of Henry III. At the Scone convention on February 5, 1283–84, he was one of the Scots lords that recognised the right of Margaret of Norway. The sudden death of Alexander III., however, in March 1285–86, and the helplessness of the infant Queen, put him on the alert for the chances of his own elevation.

On September 20, 1286, de Brus met a number of his friends at Turnberry Castle, the residence of his son, the Earl of Carrick. There fourteen Scots nobles, including de Brus and the Earl of Carrick, joined in a bond obliging them to give faithful adherence to Richard de Burgh, Earl of Ulster, and Lord Thomas de Clare (de Brus's brother-in-law), 'in their affairs.' One of the clauses saved the fealty of the parties to the King of England and to 'him that shall obtain the kingdom of Scotland through blood-relationship with King Alexander of blessed memory, according to the ancient customs in the kingdom of Scotland approved and observed.' The disguise was very thin. The instrument meant simply that the parties were to act together in support of de Brus's pretensions to the crown when opportunity should serve. It 'united the chief influence of the West and South of Scotland against the party of John de Balliol, Lord of Galloway, and the Comyns.' There need be no difficulty in connecting this transaction with the outbreak of 1287–88, which devastated Dumfries and Wigton shires. The party of de Brus took the castles of Dumfries, Buittle and Wigton, killing and driving out of the country many of the lieges. There remains nothing to show by what means peace was restored, but it may be surmised that Edward interfered to restrain his ambitious vassal.