There is no limit to how many infinitives you can have in a sentence.
Of course, it will become silly after some amount, but having two infinitives does not make your translation wrong or bizarre.
Consider this artificial example:
Credo te velle me potuisse intelligere rem ita fuisse.
I believe that you want me to have understood that the thing was so.
(More directly: I didn't understand the thing was the way it was. I think it would have been better for you if I had understood.)
Your translation is very good, but I would recommend some changes:
- Rather than using retro (which may in fact be correct but sounds a little off to me), I would use a prefixed verb.
I would replace dare retro with reddere (≈ re- + dare).
- The word order is a little unusual.
Latin word order is mostly free, but is often most fluent to use SOV unless you want to stress something or add variation to your sentence structures.
Notice that these suggestions are stylistical; I see no grammatical error.
My suggestion is this variation of yours:
Nauta se intellexit debere pecuniam puellae reddere.
This is not the only correct choice!
This is just one suggestion that I think would make your sentence more fluent.
I would also consider moving debere all the way to the end after reddere.
To emphasize "himself" you can add an ipsum after se.