In this case, lūna is not only the subject of the infinitive "esse": it's the subject of the entire phrase "'nova' esse dīcitur". That is, the structure of this clause is parallel to the English translation "The moon is said to be new". The word lūna is nominative here because "dīcitur" takes a nominative subject. Compare "Faunus cornua habēre dīcitur": it is clear why "Faunus" is nominative here?
We would have to use the accusative in sentences like "lūnam esse novam dīcimus" or "Faunum cornua habēre dīcimus" where the infinitive does not have the same subject as the main verb.
It is possible to use an accusative + infinitive with a passive verb if the verb is construed as an impersonal passive: Sean Gleason gives the example "Dicitur te venisse" translated thus: ‘It is said that you have come.’ (Personal versus impersonal passive in Latin infinitival clauses: Some diachronic considerations) Gleason says that this type of impersonal construction "is found only from Cicero in the first century BC onwards", and that in older Latin we only find sentences of the type Tu diceris venisse 'You are said to have come.'
In the case of "esse", we are dealing with what is called a copular verb, so the other, non-subject word ('nova') (called the predicative complement) agrees in case with the nominative subject.
Copular verbs are often combined in this way with passive verbs or with verbs like volō, with the resulting combination taking a nominative subject and predicative complement. A previous question that touches on this topic: Is the complement of esse in nominative or accusative when esse is a subject?