I have always been told that all Latin words with a Z are ancient Greek loanwords.
Why doesn't Latin have any native words with a Z?
Z is not originally a Latin letter! In fact, the letter we call Z wasn't Greek either, but rather had either an "sd" or "dz" sound (the jury is still out on which is correct, and there may have been regional variations).
But with Latin, it's not just that no words started with Z, but that the letter itself is Greek. They borrowed the letter to represent the Z sound in Greek words. The same is true for Y, which is why it is at the end of the alphabet.
(Incidentally, Latin didn't have J, U, or W, which are medieval additions.)
The standard reference for pronouncing Latin and its alphabet is W. Sidney Allen's Vox Latina (2nd ed., Cambridge 1989).
Z, allegedly, has a strange story. It was a Latin letter, then it became obsolete and was removed. It was then added back to accommodate words derived from Greek
A B G D E V
H Θ I K L M
N Ξ O P Ś Q R S T Y X Φ Ψ
During the Roman conquest of Greece, the letter was added back in (along with
y) to work with the words being added to the language derived from Greek.
Rex Wallace argues that the letter z “remained part of the alphabetic series until the third century BCE even though it seems to have been used sparingly – if at all – in Very Old and Old Latin inscriptions” (Wallace 2011: 15). He also cites Colonna 1980, who reportedly mentions a late 7th century BCE graffito ZKA on a fragment of a ceramic. Interestingly, in his table 2.4. “Comparison of archaic Etrsucan and Latin letterforms,” Wallace says “not attested” for zeta:
Wallace also includes a reproduction of Latin abecedarium that was incised on a ceramic plate (3rd century BCE), where the letter Z “appears in its proper position following F” (p. 15):
Eventually, the letter Z was replaced by G and we can’t find it in the Latin alphabet of the late republican period.
Weiss 2009/2011 writes that the first epigraphical attestation of Z is on a Roman denarius of 81 BCE. He also mentions evidence coming from ancient grammarians (Velius Longus, Varro, Martianus Capella) but there is no epigraphical evidence to confirm this. Weiss also hypothesizes that originally Z was probably pronounced as [ts].
For a more scholarly discussion we’d have to consult Wachter 1987 (chapter 2; caveat: it’s in German) but I don’t have a copy at hand now.