I've been looking through some etymologies and it seems to me that cardinals past trēs aren't inflected. Is this correct, and if so, what's the logic in forming words with indeclinable numbers? Take, for example, the words "duumviratus," "triumviratus," and "quīnquevirātus"; meaning, respectively, a group of two, three, or five men. What confuses me is that "duum" is an inflection of "duo," and "trium" is an inflection of "trēs," but "quinque" is just... There? Sorry if I'm out of my gourd, I'm legitimately not very experienced with Latin.
Cardinal numbers in Latin have some very peculiar rules. As you say, unus, duo and tres are inflected, but beyond those, from quattuor, four, to centum, hundred, they are indeclinable.
The cardinals from 200 (ducenti) to 900 (nongenti) are declined like the plural of adjectives such as bonus, agreeing in number and gender with the noun — sescenti, sescentae, sescenta and so on.
Then we have 1,000, mille, an adjective, but indeclinable; but in the plural it becomes milia, a (plural) neuter substantive, declined like cubilia. Thus, a mile is mille passus, a thousand (Roman) paces, while for two miles the paces are put into genitive plural, duo milia passuum.
The whole business of Roman numerals takes some effort to understand fully and use. In compound numbers, the above rules are mostly applied to the several parts. As far as xxix, the order is the same as in English. For the rest, unus almost always stands first, but otherwise we find either the smaller followed by et and the larger next (sex et sexaginta), or in reverse without the conjunction (sexaginta sex). The complications change and continue after one hundred, but I think that's probably enough for this answer.
Your examples should have -viri instead of -viratus : duumviri, a [board] of two men, each man being a duumvir — only the -vir being declined; similarly with triumvir. Thereafter the numeral part is indeclinable, so we see, for instance, a decemvir as a member of the board of ten, decemviri.