In Old Latin domus (then domos) was usually declined as a second-declension noun, though it was feminine even then. It's probably the case that it was shifted into the fourth declension in an attempt to make its unusual gender make sense—so it's not the retained second-declension forms that are unusual, it's the new fourth-declension ones.
This actually happened to a greater or lesser extent for a lot of feminine second-declension nouns: e.g. nurus 'daughter-in-law' has fully joined the fourth declension (cf. its Greek cognate νυός, still a second-declension feminine), while fagus 'beech' and laurus 'laurel' are still in the second declension but also have an attested acc. pl. in -us. Feminine second-declension nouns seem to have confused Romans nearly as much as they confuse students today. (Conversely, quercus 'oak' is a native fourth-declension noun that has an attested gen. pl. quercorum.)
For domus, this shift was already underway in Old Latin, and pretty much completed in Classical Latin, apart from the occasional now-irregular form and archaism.
As for why domos was feminine in the first place, that's less clear. We can reconstruct perhaps three basic Proto-Indo-European words built on the root *dem- in the o-grade meaning 'house' based on reflexes in daughter languages:
- Athematic root noun *dṓm < *dóm-s: Greek δῶ (neuter), Armenian town (no gender), Sanskrit dám (masculine)
- Thematic (o-stem) *dóm-o-s: Greek δόμος (masculine), Sanskrit dáma (masculine)
- Athematic u-stem *dóm-u-s: Old Church Slavonic domŭ (masculine), Lithuanian nãmas (masculine), Vedic Sanskrit dámūnas- (a derived adjective, 'of the house'), Armenian tanow-tēr (a compound, 'house-lord')
PIE o-stems gave rise to Latin's second declension and u-stems to its fourth, but as domus's membership of the fourth declension seems to be an innovation we can disregard influence from PIE *dómus (which is certainly masculine anyway). PIE o-stems in *-os were overwhelmingly masculine, and the Greek and Sankrit seem to confirm *dómos was too; by rights a Latin domus inherited from PIE *dómos should be masculine.
The gender of PIE athematics is less predictable (much like the Latin third declension, into which most of them went), and it's very inconvenient none of the high-profile reflexes of *dṓm is feminine. Mainly on the strength of Latin (!), *dṓm is still usually taken to be feminine, though, and if that's assumed, then Latin domus could have retained its feminine gender from it, after it either merged with *dómos or (in my view more likely) independently became thematised (in which case it effectively hopped declensions from the third to the second to the fourth over the centuries).
Some sources worth your time (though they don't say anything else that's relevant here, I don't think):
- Michael Weiss, Outline of the Historical and Comparative Grammar of Latin.
- Andrew L. Sihler, New Comparative Grammar of Greek and Latin.
- Michiel De Vaan, Etymological Dictionary of Latin and the other Italic Languages.