Unfortunately you do need to memorize the perfect stem for each verb you learn.
Many verbs are similar, and it helps a lot that many first conjugation verbs have the -v- in perfect forms.
But not all have, and I can't think of a reliable way to tell when a first conjugation verb is going to have an irregular perfect stem.
Especially when it comes to the third conjugation, there is no way to guess the perfect stem.
Or more precisely, there are rules but you can't predict which rule applies to a specific verb.
Is there reduplication?
Is there a nasal augment in the present stem?
Does a vowel get longer?
Knowledge of such phenomena helps memorize the perfect forms once you know quite a few of them, but I would consider it a bad study strategy to try to find rules for each one of them.
But do remember the other side of things:
You only need to remember the perfect stem or any single form formed from it.
The Latin (active) perfect is completely regular and identical for all verbs: once you know the perfect first person singular, you know all forms in perfect, pluperfect and future perfect tenses.
And this regularity goes beyond the perfect stem (albeit a little less nicely), as apart from a small amount of irregular verbs it suffices to remember just a couple of forms of each Latin verb to reproduce all forms.
Not all languages are so easy.