The full sentence is 'Quintus no longer enjoyed his studies,' and I've translated it as 'Quintus non longior gaudebat studiorum.' Should 'studiorum' be genitive since it expresses possession?
To answer the main question, no—the owner goes into the genitive case, not the thing being owned. It's like the 's in English: you say Quintus's studies, with "Quintus" getting the special marker. So if you wanted to be very explicit about whose studies they are, you could say studia Quintī. You could also use the possessive adjective suus -a -um "their own": studia sua "his own studies".
But in this sentence, I don't think I'd use any sort of genitive or pronoun there. In English, almost every noun needs to have a determiner: a word like "the" or "a" or "his" or "that" that comes before it. Latin, on the other hand, doesn't. So while you can say specifically "these studies", or "his own studies", you don't have to; in many cases, I would just ignore the English determiner when translating, because the context already makes it obvious that these are Quintus's studies, not anyone else's.
You have a few other issues in that sentence, too. Non longior is "no longer" in the sense of "not having a greater length", not in the sense of "previously but not now"; I would recommend non jam instead. Gaudeō is closer to "rejoice" than "enjoy", so I'd replace that with something like fruor. And either way, the object of that verb should be in the ablative. But these are separate from the main thing you're asking about.