I agree with the emphasis on morphology in the comments and in user786's answer. The Greek second-declension terminations -ον, -ος are etymologically and morphologically equivalent to the Latin second-declension terminations -um, -us, which surely contributed greatly to their interchange in loaned words.
However, I wanted to post about some phonetic and phonological factors that may have also been relevant. Some of the changes in the Latin > Greek direction especially seem to be fairly natural from a phonetic viewpoint.
Also, when researching this topic, I came across some information about variation in transcription practices that I thought was interesting.
The use of Greek <ν> for Latin <m> in word-final position was probably related to Greek phonotactics, and possibly to phonetic details about the pronunciation of Latin word-final -m
The phoneme /m/ (<μ>) does not occur in word-final position in any Greek words. As mentioned in the comments, Latin final m is cognate to Greek final /n/.
Furthermore, it is thought that Latin words ending in <m> were not pronounced with [m] in most contexts, but instead with either a word-final nasalized vowel, or with a vowel followed by some kind of nasalized offglide like [w̃]. Latin final -m is thought to have been phonetically realized as a nasal consonant only before a following plosive or nasal: [m] before [p, b, m]; [n] before [t, d, n]; and [ŋ] before [k, g]. Ancient Greek final /n/ most likely showed the same kind of place assimilation to a following consonant across word boundaries.
Given all that, it makes sense that Greek speakers would have adapted Latin words ending in m with Greek /n/ (<ν>).
However, there are some inscriptions with word-final <μ>
There do in fact seem to be inscriptions of Latin in the Greek alphabet that use word-final <μ>. My guess is that spellings like these were based on the Latin spelling, rather than being based on how Greek speakers perceived the pronunciation of Latin words ending in <m>.
"Greek/Latin Bilingualism", by Luca Lorenzetti, mentions three examples of <ουµ>, from Latin -um, being used as a spelling of the genitive plural termination: αννωρουµ, µησουµ, διου[µ] (IGVR 1040), which are described as evidence for "the existence of a Greek-Latin diasystem, a sort of gradient of bilingual varieties based on an extensive network of phonological, morphological and lexical conversions from one language to another" (p. 148, Encyclopedia of Ancient Greek Language and Linguistics, Vol. 2, G-O).
Latin short ŭ had a somewhat similar sound to Greek <ο>, and there are transcriptions that equate them in other positions as well
It might not be quite true to say that Latin <u> was phonetically closer to and corresponded to Greek <ου> in all other cases. Attic Greek <ου> originally represented a long back vowel, which came to have a high value something like [uː] (thought to have developed from earlier [oː]). Latin <u> had two values, short and long. The long value [uː] was certainly best represented in Attic Greek by <ου>.
But the short value of Latin <u>, aside from differing from <ου> in length, most likely also differed in having a lower quality: [ʊ]. The quality of [ʊ] is either as close to [o] as it is to [u], or possibly even closer to [o]. Since Attic <ο> is thought to have been pronounced as [o], the sound of Greek <ο> would probably have been a reasonably good match for the sound of Latin ŭ.
Greek <ο> for Latin short ŭ
In fact, Latin ŭ was not only transcribed with Greek <ο> in non-final syllables: this transcription of ŭ can also be seen in other positions in some Greek transcriptions, although the frequency of its use relative to <ου> seems to have varied between different times and places.
"Notes on the inscriptions of Delos: the Greek transliteration of Latin names", by Francesco Rovai (2015) explores this topic. Here is a quote from a relevant section:
If, from the beginning of 1st c. AD, when the sons of the Roman aristocrats were regularly sent to Athens, both Latin ŭ and ū are transcribed as ‹ου›37, this comes as no surprise. At this date and in this setting, the Athenian draftsmen encountered a definitely standardised variety of urban Latin, in which ŭ = [u ~ ʊ] (Allen, 1978: 49-50)38. On the contrary, if in the 2nd
and 1st c. BC, the Latin ŭ produced in Delos by non-aristocrat negotiatores of both urban and non-urban provenance, is instead transliterated with ‹ο›, this can be assumed as a genuine clue of its [o]-like pronunciation. At least, it was a (mid-)high vowel open enough to be perceived by the Greek drafters as more similar to their native /o/ (= ‹ο›) than to their native /u/ (= ‹ου›).
37Cf. Ιούλιος, Λουκίλιος, and Ῥοῦφος, with ū, and Σουμπτουάριος, Πουδέντα, and ἀννώρουμ, with ŭ; these and other examples can be drawn from Threatte (1980: 220-223).
38This corresponds to a higher level of standardisation also in the script, because, in this way, the Latin graphemic identity between short and long vowels is restored also in Greek. Only when followed by a consonant cluster, some examples of Latin ‹u› transliterated into Greek as ‹ο› are maintained alongside the normal spelling ‹ου›, but they all cease about 100-110 AD (Threatte, 1980: 220-221)
(Studi e Saggi Linguistici, Special Issue: Ancient Languages Between Variation and Norm, edited by Giovanna Marotta - Francesco Rovai, pp. 177-178)
This passage is followed by a table ("Table 2. Greek spellings of Latin back high vowels") that shows Latin ŭ to be transcribed with Greek <ο> in all 22 of its occurrences in the Delian inscriptions:
Καλπόρνιος (1), Κλούιος (1), Δεκόμιος (2),Φολούιος (7),Ποστόμιος (6), Σπόριος (1), Οὐετόριος (1), Οὐολόσιος (3)
Latin <u> for Greek <ο>
I'm not sure how many examples there are of Latin <u> being used to transcribe Greek <ο> outside of grammatical endings like -us and -um, but Allen (1978) says that there are some. His examples are purpura, gummi, and empurium for Greek πορφύρα, κόμμι and ἐμπόριον. These three examples don't give as much of an impression of systematicity to me as the Delos inscriptions that Rovai talks about: for example, gummi has the additional oddity of using <g> for <κ> (which some, including Allen, suggest could be explained by Latin /k/ being somewhat more aspirated than Greek /k/, but which regardless is not a usual correspondence), and the spelling emporium also exists (which Allen attributes to the influence of Greek spelling) (Vox Latina, 2nd Ed., p. 49). Allen's coverage of this topic is relatively brief, and there might well be a paper that goes into more detail and that could provide more examples.