Apparently not always, but the exceptions seem to be rather marginal and hard to find.
"The morphome [sic] vs. similarity-based syncretism", by Donca Steriade, says that -sor agent nouns invariably correspond to -sus perfect participles, but gives examples in (23) of -tor agent nouns that don't correspond to -tus participles: converritor vs. versus from verrī 'to sweep', Imporcitor (the proper name of a god) vs. imporcātus from imporcāre, infenditor 'advocate' vs. -fensus from -fendere, libritor 'hurler' vs. librātus from librāre, favitor (alongside fautor) vs. fautus from favēre, and deversitor vs. deversātus from deversārī (p. 131, The Morphome Debate, edited by Ana R. Luís and Ricardo Bermúdez-Otero, 2016).
Furthermore, page 141, Steriade lists lūdor 'player' as an example of a word where the suffix takes the form -or rather than -tor or -sor; the regular variant lūsor also exists.
"The Suffix -tor-: Agent-Noun Formation in Latin and the Other Italic Languages", by Margaret M. T. Watmough (1995/1996), says that the PIE rule for forming -tor nouns was accented full grade of the verbal root + -tor, but describes forming these nouns on the perfect participle stem as an "Italic rule" that developed later on. Many words apparently have a form that is compatible with either rule (p. 84).