Following his death in 1799 the house was sold by his executors in 1809 to his son, Samuel N. Lewis, also a successful merchant of great public spirit. In 1817 the younger Lewis sold the house to Samuel Fisher, another merchant and prominent Friend noted for his hospitality and his charity, especially toward negroes and Indians. Because of his neutrality during the Revolution, he was exiled to Virginia from 1777 until 1779, when he was arrested because of a business letter to his partner in New York which was regarded as antagonistic to the government. He was committed to the "Old Gaol", and after refusing bail was tried and because of the clamor of the mob was sentenced to imprisonment for the duration of the war. Soon afterward, however, a pardon was offered him, which he refused, and two years later he left prison by invitation without terms, his health broken. His wedding gift to his daughter, Deborah, on her marriage to William Wharton in 1817, was the Spruce Street house, which has ever since borne Wharton's name.

William Wharton was the son of Charles Wharton, who, with his wife, Hannah, devoted themselves to a religious life among the Friends. Deborah Wharton, William Wharton's wife, became a prominent[46] minister of the Society of Friends, traveling extensively in the interests of Indian welfare and giving generously of her ample means to various philanthropic causes. She was one of the early managers of Swarthmore College, as has been a descendant in each generation of the family since that time. Of her ten children, Joseph Wharton, also a prominent Friend, was owner of the Bethlehem Steel Works and one of the most successful ironmasters in the country. A liberal philanthropist, he founded the Wharton School of Finance and Economy at the University of Pennsylvania and was for many years president of the board of managers of Swarthmore College. On his mother's death in 1888 the Spruce Street house came into his possession and is still owned by his estate. Although rented as a rooming house, it remains in a fair state of preservation.

The Wistar house, at the southwest corner of Fourth and Locust streets, to which architectural reference has previously been made, was built about 1750 and for nearly three quarters of a century thereafter was the scene of constant hospitality and lavish entertainment. Here lived Doctor William Shippen, whose marriage to Alice, the daughter of Thomas Lee, of Virginia, and the sister of Richard Henry and Arthur Lee, was one of the numerous alliances which drew the county families of Virginia and Maryland into close relationship with[47] Philadelphia families. Doctor Shippen's home quickly became the resort of the Virginia aristocracy when visiting the national capital, and in consequence there was a constant succession of balls and dinners during the winter season.