During all these operations the enemy had a signal-station on Three Top Mountain, almost overhanging Strasburg, from which every movement made by our troops could be plainly seen; therefore, early on the morning of the 17th he became aware of the fact that we were retiring down the valley, and at once made after us, and about sundown drove Torbert out of Winchester, he having been left there-with Wilson and Lowell, and the Jersey brigade of the Sixth Corps, to develop the character of the enemy's pursuit. After a severe skirmish Wilson and Lowell fell back to Summit Point, and the Jersey brigade joined its corps at the crossing of the Opequon. This affair demonstrated that Early's whole army had followed us from Fisher's Hill, in concert with Anderson and Fitzhugh Lee from Front Royal, and the two columns joined near Winchester the morning of the 18th.
That day I moved the Sixth Corps by way of Clifton to Flowing Spring, two and a half miles west of Charlestown, on the Smithfield pike; and Emory, with Dwight's and Grower's divisions (Grower's having joined that morning from Washington), to a position about the same distance south of Charlestown, on the Berryville pike. Following these movements, Merritt fell back to Berryville, covering the Berryville pike crossing of the Opequon, and Wilson was stationed at Summit Point, whence he held a line along the Opequon as far north as the bridge at Smithfield. Crook continued to hold on near Clifton until the next day, and was then moved into place on the left of Emory.
This line was practically maintained till the 21st, when the enemy, throwing a heavy force across the Opequon by the bridge at Smithfield, drove in my cavalry pickets to Summit Point, and followed up with a rapid advance against the position of the Sixth Corps near Flowing Spring. A sharp and obstinate skirmish with a heavy picket-line of the Sixth Corps grew out of this manoeuvre, and resulted very much in our favor, but the quick withdrawal of the Confederates left no opportunity for a general engagement. It seems that General Early thought I had taken position near Summit Point, and that by moving rapidly around through Smithfield he could fall upon my rear in concert with an attack in front by Anderson, but the warm reception given him disclosed his error, for he soon discovered that my line lay in front of Charlestown instead of where he supposed.
In the manoeuvre Merritt had been attacked in front of Berryville and Wilson at Summit Point, the former by cavalry and the latter by Anderson's infantry. The exposed positions of Merritt and Wilson necessitated their withdrawal if I was to continue to act on the defensive; so, after the army had moved back to Halltown the preceding night, without loss or inconvenience, I called them in and posted them on the right of the infantry.
My retrograde move from Strasburg to Halltown caused considerable alarm in the North, as the public was ignorant of the reasons for it; and in the excited state of mind then prevailing, it was generally expected that the reinforced Confederate army would again cross the Potomac, ravage Maryland and Pennsylvania, and possibly capture Washington. Mutterings of dissatisfaction reached me from many sources, and loud calls were made for my removal, but I felt confident that my course would be justified when the true situation was understood, for I knew that I was complying with my instructions. Therefore I paid small heed to the adverse criticisms pouring down from the North almost every day, being fully convinced that the best course was to bide my time, and wait till I could get the enemy into a position from which he could not escape without such serious misfortune as to have some bearing on the general result of the war. Indeed, at this time I was hoping that my adversary would renew the boldness he had exhibited the early part of the month, and strike for the north side of the Potomac, and wrote to General Grant on the 20th of August that I had purposely left everything in that direction open to the enemy.
On the 22d the Confederates moved to Charlestown and pushed well up to my position at Halltown. Here for the next three days they skirmished with my videttes and infantry pickets, Emory and Cook receiving the main attention; but finding that they could make no impression, and judging it to be an auspicious time to intensify the scare in the North, on the 25th of August Early despatched Fitzhugh Lee's cavalry to Williamsport, and moved all the rest of his army but Anderson's infantry and McCausland's cavalry to Kerneysville. This same day there was sharp picket firing along the whole front of my infantry line, arising, as afterward ascertained, from a heavy demonstration by Anderson. During this firing I sent Torbert, with Merritt's and Wilson's divisions, to Kerrteysville, whence he was to proceed toward Leetown and learn what had become of Fitz. Lee.
About a mile from Leetown Torbert met a small force of Confederate cavalry, and soon after encountering it, stumbled on Breckenridge's corps of infantry on the march, apparently heading for Shepherdstown. The surprise was mutual, for Torbert expected to meet only the enemy's cavalry, while the Confederate infantry column was anticipating an unobstructed march to the Potomac. Torbert attacked with such vigor as at first to double up the head of Breckenridge's corps and throw it into confusion, but when the Confederates realized that they were confronted only by cavalry, Early brought up the whole of the four infantry divisions engaged in his manoeuvre, and in a sharp attack pushed Torbert rapidly back.
All the advantages which Torbert had gained by surprising the enemy were nullified by this counter-attack, and he was obliged to withdraw Wilson's division toward my right, to the neighborhood of Duffield's Station, Merritt drawing back to the same point by way of the Shepherdstown ford. Custer's brigade becoming isolated after the fight while assisting the rear guard, was also obliged to retire, which it did to Shepherdstown and there halted, picketing the river to Antietam ford.
When Torbert reported to me the nature of his encounter, and that a part of Early's infantry was marching to the north, while Fitzhugh Lee's cavalry had gone toward Martinsburg, I thought that the Confederate general meditated crossing his cavalry into Maryland, so I sent Wilson by way of Harper's Ferry to watch his movements from Boonesboro', and at the same time directed Averell, who had reported from West Virginia some days before, to take post at Williamsport and hold the crossing there until he was driven away. I also thought it possible that Early might cross the Potomac with his whole army, but the doubts of a movement like this outweighed the probabilities favoring it. Nevertheless, to meet such a contingency I arranged to throw my army on his rear should the occasion arise, and deeming my position at Halltown the most advantageous in which to await developments, my infantry was retained there.
If General Early had ever intended to cross the Potomac, Torbert's discovery of his manoeuvre put an end to his scheme of invasion, for he well knew that and success he might derive from such a course would depend on his moving with celerity, and keeping me in ignorance of his march till it should be well under way; so he settled all the present uncertainties by retiring with all his troops about Kerneysville to his old position at Bunker Hill behind the Opequon, and on the night of the 26th silently withdrew Anderson and McCausland from my front at Halltown to Stephenson's depot.