W. Sydney Allen, Vox Latina, 12–13, contends that the voiceless plosives in Latin were, compared to English, "relatively unaspirated," but that some aspiration may have been tolerated.
First, evidence for a lack of aspiration can be seen in Greek transcriptions of these letters: π, τ, κ were used for p, t, and c/qu ("e.g., Καπετωλιον, Κοιντος for Capitolium, Quintus"), which is significant because these "Greek letters can only stand for unaspirated plosives." Allen also mentions the unaspirated nature of these consonants in Romance languages as evidence for a lack of aspiration in Latin.
However, because Latin did not contrast unaspirated and aspirated plosives, Allen argues that "some degree of aspiration could theoretically have been tolerated." He also suggests that the way some words were borrowed from Greek is indirect evidence that some aspiration was employed.
For example, the Greek word πύξος begins with a voiceless unaspirated consonant, π. This word is borrowed into Latin, however, as buxus, not puxus. Something similar happens with κυβερνω (which becomes guberno) and κόμμι (gummi).
The reason for this can be illustrated in English. English speakers associate aspiration with voicelessness, so they tend interpret a lack of aspiration as voice, even though none is there. So the unaspirated voiceless p and k sound as though they are voiced, and are heard as b and g, respectively.
Apparently, then, Latin speakers, hearing words like πύξος and κόμμι, understood the initial consonants like English speakers might, and transcribed them with b and g. Thus, like English speakers, they must have associated aspiration and voicelessness to at least some extent, possibly indicating that their voiceless plosives p, t, c, and qu had some aspiration.
One should probably not insist too strongly on the complete avoidance of aspiration in Latin.