My suggestion with complex sentences is always to try to identify the core and to rewrite it into a simpler independent sentence.
Here the core, as far as your question is concerned, has to do with ordering and loving.
Let us compare two descriptions of orders:
Te iubes amare.
You order yourself to love.
Te iubes amari.
You order yourself to be loved.
Both are valid, but the meaning is of course inverted.
When the passive is used, the semantic subject can be indicated by an agent with a(b).
This brings in the a me:
Te iubes amari a me.
You order yourself to be loved by me.
This is very similar to "you order me to love you", but for some reason it has been composed with passive.
Using active would lead to a potential ambiguity of two accusatives: Me iubes te amare.
I can't tell whether this was the reason to choose a passive, but it probably contributed to the choice.
This would be expanded by what follows after nisi but let me ignore that here to focus on amari.
Now all of this goes into a subordinate clause.
The ut clause turns the predicate iubes to conjunctive:
Quid tibi sum ut te iubeas amari a me?
What am I to you so that you would order yourself to be loved by me?
I hope this helps you grasp the structure.
You can say bonus sum or amo.
In indirect use these becomes dixit me bonum esse and dixit me amare.
But you cannot say amare sum or amari sum.
To say "I am to be loved" you need to say something like amandus sum or amari debeo or credo me amatum iri or volo amari.
(These are not identical in meaning, but give ideas for expressing similar thoughts in Latin.)
Your last suggestion is grammatical:
Domine, iubes te amari a me.
Lord, you order yourself to be loved by me.
This would be a little clumsy to convert to direct speech, as you would need a passive imperative of the first person singular.
The speaker changes so me has to become a third person pronoun.
So perhaps:
Dominus dixit: Utinam amer ab illo.
≈ The lord said: I hope to be loved by him.
The reflexive pronoun se is often used in indirect speech to refer to the subject of the dominant clause.
But it can only refer to the third person, so *iubes se amari a me does not work.
If you want to emphasize "you" to "yourself", change te to te ipsum but not to se.
