Can I use "Domus optima" or "Domus optimus" as the equivalent (I do not need a literal translation) of "The best house"? Should "optima" be used with a noun "domus" or both are correct?
In Latin, every noun has a gender, either masculine, feminine, or neuter. Adjectives modifying that noun need to have the same gender.
Most nouns ending in -us are masculine, but domus is an exception: it's feminine. So the proper translation of "the best house" is domus optima.
This is the singular (a single best house, not several best houses) and nominative (the subject of the sentence, like "I" as opposed to "me") form. If either of those assumptions is wrong (e.g. you're talking about several best houses as the object of a sentence), the form will have to change.