As the years passed visitors flocked to Ashland in ever-increasing numbers. Many of them were politicians, but more were plain people who were devoted to Clay and could not understand why the country refused to elect him President. In 1844, during his longest period of continuous residence at Ashland, he received word of the disappointing result of the election. After a few days, when he was walking on the turnpike near the house, he was startled by a woman who, on passing him, burst into tears. When he asked her why she wept, she said:
"I have lost my father, my husband, and my children, and passed through other painful trials; but all of them together have not given me so much sorrow as the late disappointment of your friends."
A story is also told of a bride and groom who visited Ashland on the day the news of defeat was received. The journey was continued down the Mississippi River. On the boat the groom was taken seriously ill. The physician who was called to attend him was puzzled to define the ailment until the bride said that the cause was the defeat of Henry Clay. The old doctor threw his arms about the patient's neck and cried, "There is no cure for a complaint like that."
The sting of defeat was forgotten one day in 1845. Mr. Clay was in his bank in Lexington, prepared to pay a part of the indebtedness that had all but swamped him, so that he felt he might have to sacrifice Ashland. The bank told him that about $50,000 had been deposited in the bank by his friends from all parts of the country, enough to pay all his debts. He never knew the names of the generous friends who had made possible the retention of the property.
He thought he was to spend the remainder of his days at home, and that he would die there in peace. One day he said, in an address in Lexington, "I felt like an old stag which has been long coursed by the hunters and the hounds, through brakes and briars, and over distant plains, and has at last returned to his ancient lair to lay himself down and die."
Again in 1848 he tasted defeat, though on this occasion it was in the nominating convention, not in the election. In the trying days that followed he was sustained by his Christian faith. He had been baptized in the parlor at Ashland on June 22, 1847. The reality of his religious convictions was seen one day by what he said to a company of friends who had been talking in a despairing manner of the future of the country. Pointing to the Bible on the table, he said, "Gentlemen, I do not know anything but that Book which can reconcile us to such events."
In 1849 Clay was sent to the United States Senate because the legislature of Kentucky felt that he was needed to help in the solution of questions raised by the Mexican War. He spent three years in Washington, then died in the midst of his work. After a journey that showed what a place he had won in the hearts of the people, his body was taken to Lexington. The catafalque lay in state in Ashland over one night. Next day the body was buried near Lexington.
His son, James B. Clay, who purchased the estate at auction, tore down the house because of its weakened foundations, but rebuilt it of the same materials, on the old site, and on almost the identical plans. Both outside and inside the mansion has practically the appearance of the original.
Before the Civil War Ashland was purchased by the State College, but in 1882 it became the property of Major Henry Clay McDowell, whose widow lived there for many years. She was the daughter of Henry Clay, Jr., whose death at the Battle of Buena Vista was a sore blow to one who was always a fond father.