Swartwout was a protege and confidante of Aaron Burr during the period of Burr’s shady adventure in the West when he was accused of treason in an alleged plot to establish an empire in the southwestern United States. And when Jackson’s star began to rise as a Presidential candidate, Swartwout attached himself to the cause of the Tennessean.
Many of Jackson’s friends and followers resented Swartwout’s close association with Jackson because they regarded him as a doubtful character smeared by the tar of the Burr affair. But when Jackson entered the White House in 1828, Swartwout was among the honored guests at the celebrations.
Jackson’s friends were concerned when it became known that the New Yorker had easy access to the office of the President and was seen coming and going as though he were one of Jackson’s intimate advisers—which he wasn’t. The concern became dismay when rumors spread that Swartwout had come to town seeking from Jackson the nomination as Collector of Customs for New York City, a post of no little prestige and political influence in those days. Jackson’s Secretary of State, Martin Van Buren, was so upset by the reports that he refused to admit Swartwout to his office or to enter into correspondence with him.
Jackson must have felt he owed a political debt to Swartwout because on April 25, 1829, during a recess of Congress, he handed the New Yorker the political plum he had been seeking. Swartwout continued in the office until March 29, 1838, with never any public suspicion that he was involved in thefts of money collected by the Customs House in New York. Only when the records were checked by his successor was the discovery made that his accounts were short by $1,250,000.
The scandal which followed broke like a storm over the young Customs Service. Demands were made in Congress for safeguards to prevent any such future looting of the Treasury. Enemies of Jackson attacked the “spoils system” of appointments and centered much of their assault on Customs.
As for Swartwout, he had foreseen the storm that was to come. He had bade his friends farewell and boarded a ship for Europe several weeks before the shortages in his accounts were discovered. He was in France, safely out of reach of the law, when the scandal broke—and he didn’t bother to return.
By 1849, the Customs Service spanned the continent. It reached the coast of California in the person of John Collier, who was appointed as the first Collector of Customs for San Francisco just as the state was clearing the way for entry into the Union. Collier reached San Francisco on November 13, 1849, after a perilous trip across the country. He arrived at the beginning of the gold rush to find the city and the customs situation in a state of disorganization and confusion.
Collier was overwhelmed by the amount of business being carried on in San Francisco, by the number of vessels arriving and leaving the harbor, by the smuggling which was going on, and by the high prices he found in the city. He advised Secretary of the Treasury W. M. Meredith in a long, rambling letter: “I am perfectly astounded at the amount of business in this office.... The amount of tonnage ... on the 10th instance in port, was 120,317 tons; of which 87,494 were American, and 32,823 were foreign. Number of vessels in the harbor on that day, 10th instance, 312, and the whole number of arrivals since the first of April, 697; of which 401 were American, and 296 foreign. This state of things, so unexpected, has greatly surprised me....”
He found that Customs clerks were being paid from $1800 to $3000 per annum but that the salaries were not particularly attractive in a city gripped by the get-rich-quick fever. Flour was selling for $40 per barrel and pork for $60. Board was $5 a day and a room with a single bed was $150 a month. Wood was $40 a cord and prices for other necessities were equally shocking to a man as obviously thrifty as Collier.
Collier moved into the Old Spanish Custom House to set up shop. It was a dark and gloomy building. The roof leaked. Some of the doors were off the hinges. There was no vault in which to place the money he collected. He confided to the Secretary that “owing to the rates for rents, I am afraid to lease a building. One for myself, containing four rooms, two below and two above, without fireplaces, was offered to me on yesterday at $2400 a month....”