It is uncertain whether any of the Norman kings before Stephen made the Tower a place of residence. But in 1140 that monarch, during a gloomy period of private and public affairs, retired to the Tower with a large retinue and kept his court there during Whitsuntide.

“Early in the year,” writes Freeman in his “History of the Norman Conquest,” “after Matilda’s landing, an attempt had been made to make peace. At Pentecost the King held, or tried to hold, the usual festival in London; but this time his court was held to the east and not to the west of the city, not in the hall of Rufus, but in the fortress of his father.”

The custody of the Tower appears, soon after its completion, to have been made an hereditary office, granted by the sovereign to the family of Mandeville. In this year of 1140 the Tower was in the keeping of Geoffrey, grandson of that great Geoffrey de Mandeville, who had accompanied the Conqueror to England, and who had greatly distinguished himself at the Battle of Hastings. Stephen created the grandson Earl of Essex, but being himself taken prisoner soon afterwards at the Battle of Lincoln, the Empress Matilda gained de Mandeville over to her party, during Stephen’s captivity. By a charter, dated from Oxford in 1141, Matilda confirmed the Earl in all the possessions which he had inherited, whether in lands or fortresses, the custody of the Tower being included therein, Essex being given a free hand to strengthen and fortify it. A subsequent charter of the same year gave him the special charge of the Tower, “with all lands, liveries, and customs thereto appertaining” (Dugdale’s Baronage). According to Leland, de Mandeville constantly added to the fortifications of the Tower, but when he was defeated and taken prisoner at the Battle of St Albans he was obliged to surrender the Constableship into the hands of Stephen.

In 1153 the Tower was held for the Crown by Richard de Lucy, Chief Justiciary of England, in trust for Henry, Duke of Normandy, to whom, after Stephen’s death, it reverted.

Matilda had offended the Londoners by refusing to abolish her father’s laws, and by also refusing to restore those granted by Edward the Confessor, and, rising in arms, they drove the Empress from the city. Stephen having recovered his liberty, Matilda’s power ceased shortly afterwards. After her flight the Londoners laid siege to the Tower, but it had been so strongly fortified by de Mandeville that he was not only able to defy the besiegers’ uttermost efforts to effect its capture, but was able to make a sortie as far as Fulham, where he took the Bishop of London prisoner, “as then lodged there, being of the contrary faction” (Holinshed).

It is doubtful whether Henry the First ever lived in the Tower, or whether he added to its fortifications. Thomas à Becket is supposed to have wished to have been made Constable of the fortress as well as of Rochester Castle, which latter he is known to have held.

FitzStephen, in the reign of Henry the Second, describes the “Arx Palatina” as being then, “great and strong with encircling walls rising from a deep foundation, and built with mortar tempered with the blood of beasts.” Probably the sanguinary aspect of the mortar used in the Tower buildings was owing to the use of pulverised Roman red tiles and bricks, of which a large quantity were most likely pounded into mortar.

When Richard Cœur de Lion left England for the Holy Land he entrusted the charge of guarding the Tower to Longchamp, Bishop of Ely, who was his Chancellor. This Bishop strengthened the fortress and deepened the moat. He had good reason for his work upon the fortress, for John, taking advantage of his brother’s absence, besieged the Tower; but the Bishop, thinking discretion the better part of valour, yielded up his trust without attempting to defend it, and fled for safety to Dover Castle. John made over the Tower to the confederated nobles under the Archbishop of Rouen, who occupied it until Richard’s return from the Holy Land.

In 1215, the Barons, who were then up in arms, aided by the London citizens, besieged the Tower, but although it was poorly garrisoned, their attacks were repelled. A year later, whilst the civil war was waging between John and his barons, the Tower was handed over to the French prince Louis by the rebellious nobles, who had invited him to take John’s place as King of England, but Louis does not seem to have taken kindly to the position, and speedily returned to his own land. In 1217, Henry III. was reigning in undisputed possession of the realm, and to him belongs the credit of having done more towards making the Tower worthy of a royal abode, than any of his predecessors or successors upon the English throne. The most stately of its buildings, after the Great Keep, are due to his love of art and architecture. The Royal Chapel, the Great Hall, and the Palace chambers, which he either built or decorated, are frequently mentioned in the chronicles of Henry’s reign, and were the outcome of his taste and love of magnificence.

In 1232 the Tower was given into the custody for life to the famous Hubert de Burgh, Earl of Kent. His constableship, however, was brief, he being supplanted by Peter de Roches, Bishop of Winchester, and imprisoned in the fortress he had formerly governed.