As the days passed by and the promised arms and reinforcements for Gen. Spier failed to materialize, he became restless and disheartened. The United States authorities had seized all of the arms and ammunition that could be discovered, and the fact was forced on the deluded General's mind that if he did not leave Canada soon a strong force of British troops would be upon him and annihilate his command. Moreover, the demoralization of his whole army was becoming complete, and both officers and men refused to do duty any longer. Desertions were taking place in a wholesale manner, and in several instances Colonels marched off with their entire commands and re-crossed the line. He therefore convened a Council of War to consider the situation. It was of short duration, as the officers were of the unanimous opinion that there was no other course left for them but to retrace their steps and give up the idea of invading Canada. The reinforcements, arms, provisions and munitions of war that had been so liberally promised, had failed to reach them, and weakened as they were by such wholesale desertions to the rear, it was deemed by old soldiers to be nothing but madness to remain where they were, as they would be wholly unable with such a small force to make even a decent show of a fight, should they happen to be attacked, and it was at once determined to give up the intended invasion, leave Canada, and head back for the United States.

Therefore Gen. Spier ordered Col. Coutri and Col. O'Connor to form up their men and march them back to St. Albans to report to Gen. Sweeny. Both of these officers were deeply affected as they proceeded to carry out their orders, as they wanted to stay and fight it out.

The men were formed in companies, but many went off on their own responsibility, and at 9.30 o'clock on the morning of June 9th, all that was left of the grand "Right Wing" were marching back across the border to the United States. The men had a few rounds of ammunition left in their pouches, and immediately commenced firing off their muskets and rifles in a most promiscuous manner. Arms, plunder and everything else that the men could carry off with them on their retreat were lashed upon their backs or packed in satchels, and quite a number of new suits of clothes, hats, shoes and other valuables which they had pilfered were carried off by them. Several horses were also taken across the line by the marauders.

Generals Spier and Mahon marched on foot among their retreating troops, and were very much downcast. Gen. Spier said that he would rather have been shot than have left Canada in the manner he was obliged to, while Gen. Mahon wept with rage at the thought of having to abandon the invasion. Most of the officers expressed themselves as being ashamed of the affair, and would rather never go home. After all their boasting of how easily they would capture Canada and set up their visionary Republic, the disgraceful manner in which the whole campaign terminated, without so much as a slight skirmish having taken place, was more than they could bear. There were many brave yet deluded men who joined the expedition with a determination to fight, but the majority of them were "nothing more or less than an armed mob, roving about wherever they pleased, robbing the houses and insulting and abusing women and children." as stated by a newspaper correspondent.

When the retreating raiders reached United States territory they found detachments of American troops stationed upon all the roads leading to St. Albans, who had received instructions to seize all the arms the Fenians might have in their possession. As the majority of them had thrown away their muskets, sabres and ammunition on their retreat, there was not much left for the United States troops to gather up, but what little there was left was promptly seized.

Upon arrival on the American side of the line Gen. Spier and his staff surrendered to Col. Livingston, of the United States Army, and were taken to St. Albans and placed under heavy bonds to await trial for violation of the neutrality laws.

A portion of Spier's army who were stationed at a point about eight miles from St. Armand when the main body retreated, were charged upon by 40 men of the Montreal Guides, and in the skirmish several Fenians were killed and sixteen taken prisoners, who were conveyed to Montreal. There were no casualties on the Canadian side.

On the night of the 9th of June a train left St. Albans for the east with nearly 1,000 Fenians bound for their homes, while many others were left skulking around the country in the hope that another raid would soon be organized, whereby they could have an opportunity of securing more booty.

On the 22nd of June a small party of these marauders came on a reconnoitering expedition to Pigeon Hill, and on arriving at the outpost began firing at the Richelieu Light Infantry sentinel who was stationed there. They were in a thick bush off the road, leading across the lines to Franklin County. As soon as they were perceived, the Canadian detachment made an endeavor to get between the Fenians and American territory, for the purpose of intercepting their retreat. But the Fenians fled through a swamp and managed to effect their escape. About twenty shots were fired, but without effect.

This was the last episode of the Pigeon Hill affair, and in another week peace and quietness again prevailed along the Vermont border.