I think the ablative of agent in your example sentence is a red herring, and you can state your question in more general terms. In fact, I think the simplest case to look at is a noun + adjective, since the amata in your case is functioning as an adjective.
The most important thing to know about a relative pronoun is that it begins a relative clause. A relative clause needs to be a full sentence with a verb, just as in English.
So, let's say we want to talk about a "strong girl." There are a few things we could do:
- Attributive adjective
Here, we simply have an attributive adjective modifying puella. In the nominative, it indicates that a "strong girl" is the subject, but it needs a verb or something else to be a complete sentence, e.g. Puella fortis pugnat = "The strong girl fights."
- Predicative adjective
Puella fortis est.
Here, you are expressing a complete thought: "The girl is strong."
- Relative pronoun
Puella quae fortis est...
Here, you are starting a relative clause after puella. You cannot say puella quae fortis, because quae fortis is not a complete thought. In this case, the English equivalent captures the disfluency pretty well: we don't say, "The girl who strong." Rather, we have to say, "The girl who is strong..." Note that this sentence still isn't complete: it's like the first example, puella fortis, and the main clause still needs to be completed, e.g. Puella quae fortis est pugnat. = "The girl who is strong fights."
To return to your example case, you would need to do two things:
- Complete the relative clause
- Complete the main clause
For example, you could write:
Puella quae a puero amata est... = "The girl who was loved by the boy..."
(Note that amata est is the perfect passive: the present passive would be expressed by amatur.)
You would only write quam if the girl was the direct object of the sentence in the relative clause. For instance, you could write:
Puella quam puer amat... = "The girl whom the boy loves..."
In short: treat the relative clause like an independent sentence. Just as "puella amata" is not a complete sentence, so "quae amata" is also incomplete: it needs a verb.
Latin does sometimes omit the verb, particularly copulae like est, but this would be confusing in a relative clause. There's at least one example to be found in the Aeneid (quoted in the linked A&G grammar section):
accipe quae peragenda prius. (Aen. 6.136)
My intuition here is that the gerundive makes this construction more permissible. I could also see instances in a Ciceronian speech where a verb would be omitted, e.g. "quae facta, quae omissa bene scis..." but for your purposes I would recommend avoiding that kind of construction.