When Thomas Oliver, Lieutenant Governor and president of George III's provincial council, built his house in Cambridge about 1767, he did not dream that within nine years he would have to abandon it because of his allegiance to the same George III. But so it proved. He was a Tory, and his neighbors would not suffer him to remain among them. On September 2, 1774, he wrote his resignation of the offices he held, adding the statement, "My house at Cambridge being surrounded by five thousand people, in compliance with their command, I sign my name." At his request, made to General Gage and the admiral of the English fleet, troops were not sent to Cambridge, according to plan. "But for Thomas Oliver's intercession," Edward Everett Hale says, "Elmwood would have been the battle-ground of the First Encounters."
After his summary departure the house was used as a hospital by the Continental Army. When the government sold it at auction it became the property first of Arthur Cabot, then of Elbridge Gerry, a Signer of the Declaration of Independence, Governor of Massachusetts from 1810 to 1812, and Vice-President under Madison.
The next occupant was Rev. Charles Lowell, pastor of the West Church of Boston. He bought the property just in time to make it ready for his son, James Russell Lowell, who was born February 22, 1819.
As a boy James never wearied of rambling over the old house and the ten acres of ground, all that was left of the original ninety-five acres. Many of his poems contain references to the memories of these early years. "The First Snowfall," "Music," and "A Year's Life" are, in part, autobiographical. Lines on "The Power of Music" told of the days when he was his father's companion in the chaise, on the way to make a Sunday exchange of pulpits with a neighboring minister:
"When, with feuds like Ghibelline and Guelf,
Each parish did its music for itself,
A parson's son, through tree-arched country ways,
I rode exchange oft in dear old days,
Ere yet the boys forgot, with reverent eye,