During the fourteen years' stay in England many misfortunes came to Penn. He was accused of treason, and his title to the American lands was taken away from him. Later he was acquitted, and his lands were returned.
In 1692 Guli Penn died, and in 1696 Penn married Hannah Callowhill. In 1699, when he returned to America, he brought with him his wife and Letitia, who was then about twenty-five years old.
Evidently the old house was not good enough for the ladies of the family. At any rate they occupied for a time the "slate-roof house," one of the most pretentious buildings in the Colony. When the manor, Pennsbury, twenty miles up the Delaware, was completed, the family was taken there. Great style was maintained at the country estate in the woods. The house had cost £5,000, and was "the most imposing house between the Hudson and Potomac rivers."
The Philadelphia house was transferred to Letitia on "the 29th of the 1st month 1701." At once extravagant Letitia tried to dispose of it. She succeeded in selling a portion of the generous lot, but it was some years before she was able to sell the whole.
In the meantime the Proprietor felt that he must return to England because of the threat of Parliament to change the government of the American Colonies. Mrs. Penn and Letitia, who did not like America, pleaded to go with him. He thought he would be returning soon, and he urged them to remain. They insisted. In a letter to James Logan he wrote: "I cannot prevail on my wife to stay, and still less with Tish. I know not what to do." Later he wrote: "The going of my wife and Tish will add greatly to the expense.... But they will not be denied."
In 1702 Letitia married William Aubrey, who had all of Penn's keenness and none of his genial qualities. Almost from the day of the marriage both husband and wife pestered Penn for money. Aubrey insisted on a prompt payment of his wife's marriage portion. His father-in-law was already beginning to feel the grip of financial embarrassment that later brought him to the verge of bankruptcy, but, on this occasion as well as later, he felt compelled to yield to the insistent demands of the grasping Aubrey.
The only members of the Penn family who ever returned to America were the children of the second wife, to whom most of the property descended.
The Letitia Penn House, as it came to be known, fell on evil days. It was an eating house in 1800, and in 1824 it was the Rising Sun Inn. Later it was called the Woolpack Hotel.
In 1882 funds were raised by public subscription, and the venerable house was taken down and rebuilt in Fairmount Park. Visitors who enter the city by the Pennsylvania Railroad from New York City may easily see it from a right-hand car window, for it is the only house in the corner of the park on the west side of the river.