Is it correct that “Finit hic, deo” translates into “God ends here” like they say in the movie “The Nun”?
(The scene in the movie where the phrase is seen and the translation is given can be viewed here.)
No, that is not a correct translation, but the root problem is probably not a mistranslation from Latin to English, but rather from English to Latin. The Latin sentence was very likely created specifically for the motion picture – as far as I can see, it does not appear anywhere else – and, as they say, mistakes were made.
There are a few problems here:
As C Monsour has pointed out in the comments, it the word order is infelicitous, because it suggests the reading “This god ends.” It is be better to draw the hic to the front.
In conclusion, the correct form would be: Hic finit(ur) deus.
What would the sentence mean as it appears in the movie? It is not ungrammatical, but the subject is missing. The deo could be interpreted as a dativus (in)commodi, indicating that God is affected by the action. In that case the meaning would be:
He (or she or it) ends (something) here for God (i.e., to his advantage or disadvantage).
For example (since it is my understanding that the sentence is inscribed on a door in the movie), the sentence could mean that the door ends the passage here for God.
It could also be interpreted as an ablativus instrumentalis:
He (or she or it) ends (something) here using God.
I see no way to read the sentence that would excuse the comma, though. The comma does suggest that the sentence is directed at God, in which case we would expect the vocative case. The classical vocative of deus is one of the great mysteries of the Latin language, but in the Christian context it is deus. It is definitely not deo.