After his discharge Major Kelly returned to his farm and family, and during the three succeeding years the Indians were troublesome to the settlers on the West Branch. He became colonel of the regiment, and it was his duty to guard the valley against the incursions of the savages.

When the “Big Runaway” occurred following the Wyoming massacre, Colonel Kelly was among the first to return. For at least two harvests reapers took their rifles to the field, and some of the company watched while others wrought.

Colonel Kelly had the principal command of scouting parties in the valley, and very often he went in person. Many nights he laid on the branches of trees without a fire, because it would have indicated his position to the enemy. He was skilled in Indian mode of warfare and was a terror to their marauding bands.

So greatly was he feared by the savages that they determined on his destruction and, being too cowardly to attack him openly, sought his life by stealth. One night he apprehended they were near. He rose early next morning and, looking through the crevices of his log house, he ascertained that two at least, if not more, were lying with their arms so as to shoot him when he should open his door. He fixed his own rifle and took his position so that by a string he could open the door and watch the Indians. The moment he pulled the door open two balls came into the house and the Indians rose to advance. He fired and wounded one, when they both retreated. When safe to do so he followed them by the blood, but they escaped.

After the capture and destruction of Fort Freeland, Colonel Kelly with a company of men went to the scene of the battle and buried the dead.

For many years Colonel Kelly held the office of Justice of the Peace, and, in the administration of justice, he exhibited the same anxiety to do right, which had characterized him in his military service. He would at any time forego his own fees, and, if the parties were poor, pay the constable’s costs, to procure a compromise.

While he was a devout Presbyterian he entertained an intense hatred for an Indian. When the Presbytery of Northumberland called on Colonel Kelly for a contribution to be used to evangelize the savages, he refused to give one cent, but said he would cheerfully subscribe any sum required to buy ropes to hang them.

Toward the end of a long and active life, Colonel Kelly became by disease incapable of much motion; and seldom left his home. He died February 18, 1832, aged eighty-eight years. He was greatly respected by his neighbors and friends, and it is little wonder that a monument was unveiled to his memory three years later.

The spring of 1856 the monument, together with his remains, were removed to the new and beautiful cemetery on the western border of the Union County seat.

The old colonel was survived by his wife, seven sons and two daughters. One son, James, was the father of United States Senator James K. Kelly, of Oregon.