Three kinds of masonry are visible in the construction of the tower, each occupying about a third of its height. The bottom layer consists of small stones, the middle part of larger ones, while those at the top are hewn.
At one time part of the wall connecting the towers was used as a ball-alley (the ground being flagged for this purpose), until the owner of the castle enclosed the ruins with a wall for their preservation.
In 1865 part of the fortress on the north side fell in a thunderstorm, and the tenant of that date procured leave to blast the rest of this wall for fear of accident. The ground is littered with broken masonry.
After Strongbow’s death in 1177, Henry II. bestowed the manor and castle of Ferns upon William FitzAdelm de Burgo. The same year FitzAdelm seized the Black Castle of Wicklow from the three sons of Maurice Fitzgerald, giving them Ferns by way of compensation.
The brothers at once began to rebuild and strongly fortify their new possession, but it was hardly completed before Walter Allemand, a nephew of FitzAdelm, attacked the castle and left it in a ruined condition.
William Marshall, Earl of Pembroke, who married the grand-daughter of Dermot MacMurrough, began to erect a much larger fortress on the same site in 1192. He was succeeded by his son, who had married the daughter of King John, and he completed the stronghold in 1224. He then presented it to the Bishop of Ferns as restitution for Church land which his father had seized.
It remained in the possession of the Church from 1224 to 1364, during which time it was used as an Episcopal Palace.
In 1243 Geoffrey St. John, Vicar-General of Ferns, and Escheator of Ireland, came into residence, and was succeeded by Bishop Lambert in 1282, who died in the castle.
He was followed by Richard of Northampton, who had been Canon of Kildare, while in 1304 Robert Waldrond was consecrated, and took up his abode at Ferns. During this time the neighbourhood was much disturbed, and the next Bishop was arraigned for high treason in 1317, but was pardoned the following year.
In 1331 the clan of O’Toole seized the castle, pillaged, and burned it. Next year the Crown took possession again, and three years later they thought it expedient to appoint Lord Gerald Rochford constable of Ferns Castle. He held office for ten years, and was summoned to Parliament as a Baron.