Consider this a supplement to @brianpck's answer. This is an explanation of why making the possessive pronoun agree with the thing possessed and not the possessor makes sense in Latin.
Agreement
In Latin, as in all languages, we are mainly concerned with linking things we are talking about with concepts that describe them. This happens in two ways: asserting a claim about something, and narrowing down what you're referring to by combining it with another concept. A sentence asserts some predicate of its subject: "Mārcus ridet" asserts of Marcus, the subject, that he is laughing, the predicate. In "Fīlius improbus rīdet", improbus distinguishes one son from the other. To understand a sentence, we have to know which modifier goes with which modified thing. This is the job of grammar.
In English, we mostly depend on word order to link a modifier with what it modifies, but in Latin this is done mainly by grammatical agreement between them. In this sentence:
Iūlius ex hortō silēns vēnit.
silēns, in the nominative case, agrees with Iūlius. So, it means "Julius, silent, came out of the garden." In this sentence:
Iūlius ex hortō silēnte vēnit.
silēnte, in the ablative case, agrees with hortō. So, it means "Julius came out of the silent garden."
To "get" Latin, you have to train yourself to track grammatical agreement until eventually it becomes unconscious. It takes a while, but notice that English exploits grammatical agreement, too, so you already have some experience with this approach. Notice the difference between "John and Stephen met Henry after he was sentenced" and "John and Stephen met Henry after they were sentenced." Grammatical agreement in English is mostly limited to subjects agreeing in number and person with their verbs, and pronouns agree in number and case with their antecedents. In Latin, nouns also agree with their adjectives, in number, case, and gender. These are things that you use for clarity and expressiveness.
Possessive pronouns
If a possessive pronoun—a modifier—agreed not with the thing possessed but with the possessor, it would throw grammatical agreement into confusion. That's probably hard to believe if you're confused now, but here's why it would result in more confusion. What happens if the possessor and the thing possessed have different cases?
Iūlius mihi clāvēs suārum lectīcārum dedit. (Julius gave me the keys to his sedans.)
If we tried to change the grammar so the possessive pronoun agreed with the possessor, what would suārum change to? Iūlius is singular, nominative, and masculine, so suus would be needed to agree with it. But lectīcārum is plural, genitive, and feminine, requiring suarum.
What are you going to do? The only reasonable way, which doesn't suffer constantly from internal contradictions, is to make the possessive pronoun agree with the thing it modifies—the thing possessed—in all three ways: number, case, and gender. If you're constantly tracking grammatical agreement in order to follow a sentence, agreeing with the possessor would feel like a bizarre exception—a confusing inconsistency.
This is particularly important when using non-standard word order to make a point or shift emphasis:
Iūlius mihi clāvēs lectīcārum dedit suārum. (Julius gave me the keys to his sedans.)
Nōn Aemiliae Iūlius mihi clāves lectīcārum dedit sed suās. (It wasn't Emily's keys to the sedans that Julius gave me, but his.)
Se
Here's one other clue that helps: the possessive pronouns in suus are adjectival forms of the reflexive pronoun se, which means "himself, herself, itself". If you let the s remind you of se, you'll be in the Latin groove even without thinking about what should modify what. Of course suam in "villam suam" refers to Iulius: he's the subject of the verb.
