The verb ought, when used to express past duty or obligation, is followed by what is called the perfect infinitive—a use peculiar to itself because ought has no past form.
Example: I ought to have gone yesterday.
Other verbs expressing past time are used in the past tense followed by the root infinitive.
Example: I intended to go yesterday.
Composition and Rhetoric. By Lockwood and Emerson. Ginn & Co., Boston.
The Art of Writing and Speaking the English Language. By Sherwin Cody. The Old Greek Press, Chicago.
The Writer's Desk Book. By William Dana Orcutt. Frederick Stokes Company, New York.
A Manual for Writers. By John Matthews Manly and John Arthur Powell. The University of Chicago Press, Chicago.