A synchronic analysis: deletion of coronals before s within a syllable
In "Latin Rhotacism for Real," Kyle Gorman describes the deletion of /t/ in the nominative form of words like mons, montis as part of a larger pattern of deletion of coronal consonants which also applies to words like pes, pedis and pollis, pollinis. The main argument of the paper is that in Classical Latin, nouns with variation between r in some forms and and s in the nominative should be analyzed according to this deletion rule, rather than according to a separate rule of intervocalic rhotacization. So if Gorman is right, the deletion rule was still active in Classical Latin.
He notes that
The sequence [ts] (from /ts/, or /ds/ and DEVOICING) does not occur in
word-final position (Devine and Stephens 1977:129), and its medial
occurrences are limited to syllable contact, allowing a rule of
deletion to be stated generally (cf. Heslin 1987:142).
This is the following "pre-/s/ deletion" rule:
[CORONAL, +ORAL, -CONTINUANT] → ∅ / _s]σ
Gorman extends this rule to nouns with stems ending in n such as pollis and sanguis by removing the [+ORAL] specification.
[CORONAL, -CONTINUANT] → ∅ / _s]σ
In order to prevent this extended rule from improperly deleting multiple consonants (such as the n in mons) Gorman proposes that the rule is "non-iterative" and "rightward-applying":
In processing /mont-s/, the rule “sees” /t-s/, but the resulting [ns]
is not scanned by the rule.
This is the basis for his argument that forms like mos, moris can be analyzed using deletion of r before s, even though r is not deleted in the nominative forms of words with stems ending in rt such as ars, artis.
Etymological analysis
Gorman's rule is clearly not based on the etymology of the words, since historically, the alternation of s and r did in fact originate from rhotacism and not from a historical process of r-deletion. (I don't know the historical origin of the nouns like pollis that have n in the stem but not in the nominative.)
Nonetheless, for nouns with with stems that end in -t- and -d-, the historical source of this variation seems to be similar to Gorman's synchronic rule. In some ancestor to Latin, a phonetic change occurred where word-final [ts] was eventually simplified to [s].
Etymological consonant stem nouns
I found a paper that describes some theories about when this happened and what the intermediate steps were: "The Proto-Indo-European *-VTs# clusters and the formulation of Szemerényi’s Law," by Dariusz Piwowarczyk. The main example word he uses actually has "d" (*pod-s) but it seems that all scholars agree that underlying /ds/ was already devoiced to [ts] in PIE. There is disagreement about whether this [ts] was simplified to [s] in PIE or independently in the later branches of Indo-European.
Etymological i-stem nouns
TKR pointed out in the comments that Classical Latin also has many words that lose t in the nominative that come from Proto-Indo-European i-stem nouns. Mons and sors are two examples. The nominative forms of words like these seem to originate from syncope of i, followed by simplification of the cluster [ts] to [s]. Since this syncope was a Latin-specific development, and preceded the cluster simplification, these seem to require a Latin-specific rule at some point that changed [ts] to [s] in this context.