According to Vine's "The Morphology of Italic", all the infinitive endings originated with the third/consonant conjugation, and were extrapolated from there.
Many consonant-stem verbs in Latin used to share a root with an S-stem noun: in other words, the verb was formed by putting (thematic) verb endings directly on the PIE root, and the noun was formed with an extra -s before the noun endings.
For example, the Old Latin verb gen-ō "give birth to" shares its root with the S-stem gen-os-s, gen-os-is "birth" (later genus, generis). The locative singular of this noun was gen-es-i "in birth", which became gen-er-e "to give birth to" by regular sound change.
Over time, this got re-analyzed as gene-se: in other words, a suffix -se attached to the present stem of a verb. And since it was really useful having an infinitive form, this was extrapolated to all the other conjugations, as well as other forms that didn't have S-stem nouns in the first place: habu-is-se "to have held", es-se "to be".
*(The alternation between S and R is due to rhotacism: sometime in the Old Latin period, S between vowels turned into Z, then into R. This is why habē-re has an R in the suffix while es-se has an S.)
Other Italic languages, though, formed their infinitives somewhat differently—and it's likely that pre-Old-Latin followed some of these patterns too. The passive was never quite as common, so it took longer to settle down and never got fully regularized.
- The oldest infinitive pattern, with cognates in Indo-Iranian languages, goes back to a PIE suffix like *-dʰyēi. In Italic this became *-ðjē, which we see in words like Umbrian pihafei (on the Iguvine Tablets—stem cognate with Latin piāre).
- In Oscan, an extra passive -r was stuck on the end, the same one seen in most other passive endings (-or, -tur, -ntur, etc). This formed the passive infinitive sacrafīr (stem cognate with Latin sacrificāre).
- Latin did the same thing, but turned the *ð into a z between vowels and deleted it after consonants. So the "original" Latin passive infinitive was -ier as in ag-(z)ier, amā-(z)ier, etc, which led to the archaic-but-Classically-attested forms like agier, amārier: these used to be far and away the most popular infinitives in the third conjugation.
- Another option came from root nouns in the third conjugations, where noun endings were stuck on the PIE root with no adulteration: instead of gen-os-s, the root noun would be simply gen-s. Root nouns were never very prominent in Italic, but Vedic used the dative singular of root nouns as a passive infinitive.
- And in fact, the dative singular of a root noun in Latin would look like ag-ī "to be driven" (< *ag-ej "to/for driving"), which is exactly what we see in the third conjugation!
- Once the -rier passive infinitive fell out of fashion, this is the formation that stuck in the third conjugation. But it never spread to the other conjugations, which didn't have root nouns to work with.
- Yet another option was to use the dative singular, but from the S-stem noun instead of the root noun. This ending would be *-s-ej, becoming Latin -s-ī, and this was quickly applied to the first, second, and fourth conjugations. It looked a lot like the -se active infinitive so it was easy to remember and that was a big advantage. Hence, forms like amārī "to be loved".
- But, this new form never managed to displace the root-noun -ī infinitive in the consonant conjugation, which stuck around and gave us the unexpected third-conjugation forms.
