As far as I am aware, the classical pronunciation of -gn- (as in magnus) is not [gn] but [ŋn]. How do we know that this is in fact how -gn- was pronounced?
We don't know for sure how -gn- was pronounced in Classical Latin. There are a few arguments for reconstructing the pronunciation of -gn- as [ŋn]. (More specifically, I believe the most common reconstruction is [ŋ.n], with a syllable break between the two consonants. This explains why the preceding syllable is always metrically "heavy"—or in the misleading old-fashioned way of putting it, the preceding vowel is "long by position". The idea that vowels before -gn- were always "long by nature" is probably erroneous; see Nathaniel's answer to Are vowels long before “gn”?)
But there are also some complications with this reconstruction, and possible reasons to prefer reconstructing the pronunciation of -gn- as [g.n] (with a voiced oral velar plosive) for at least some time periods or some speakers.
As usual, a good place to start for questions about Classical Latin pronunciation is W. Sidney Allen's Vox Latina.
There are several pieces of evidence pointing towards a pronunciation [ŋn]
Allen (p. 23) describes several reasons for thinking that -gn- was pronounced as [ŋn] in Classical Latin.
ĕ > ĭ before [ŋ]
Apparently, a vowel change of short e to short i is commonly seen before the clusters ng, nc, and gn; we can unify the description of these environments as "before [ŋ]" if we assume gn represented [ŋn].
This sound change does not seem to have applied to ĕ before -gm-. Allen says that this is commonly viewed as a sign that -gm- was not pronounced with [ŋ] (which is perhaps surprising from a theoretical point of view, as both M and N are nasal consonants, and they would generally be expected to have similar effects on the pronunciation of preceding consonants).
It is generally assumed that g did not have a nasal pronunciation before m, as in tegmen, segmentum, since the change of ĕ to ĭ does not occur in these words. However, since original gm seems to have given mm (e.g. flamma from flag-ma, cf. flagro), all examples of gm may have arisen later, e.g. by syncope, after the change of ĕ to ĭ was operative (cf. the unsyncopated forms tegimen, integumentum). The possibility of a pronunciation of g as [ŋ] here is therefore not entirely excluded-though it cannot be safely recommended.
(Allen 1978, p. 25)
This sound change seems to be one of the strongest pieces of evidence for reconstructing the pronunciation of -gn- as [ŋn]. I believe there are some exceptions to the sound change, but I don't know what they are (and I don't know how many good examples of the sound change we have).
Prefixing in- or con- to gn- gives ign- or cogn-
Words like cognatus and ignobilis contain the prefixes con- and in-, which typically end in a nasal consonant that assimilates in place to a following plosive or nasal (as in compugno, impossibilis, committo, immortalis). If word-medial gn in Latin was pronounced as [ŋn], then the use of gn in these words can be explained as the result of the usual assimilation of /n/ > [ŋ] before /g/, followed by a simplification of the tautosyllabic onset cluster /gn/ to /n/, with a heterosyllabic cluster [ŋ.n] as the result.
If gn was just pronounced as [gn], it would mean that the nasal consonant from the end of the prefixes was dropped entirely in these words. That seems a bit implausible to me, but I guess it's not impossible. In fact, I did find a source from the 19th century that argues for this interpretation of prefixed words containing gn, and rejects the [ŋ.n] interpretation:
Had the Romans retained the n of the prepositions before gn, they would have felt bound to pronounce ing-gnotus, cong-gnatus, but would practically have pronounced ing-notus, cong-natus. But they did not retain the n and write ingnotus, congnatus, but ignotus, cognatus. What is the explanation ? Does this represent a pronunciation ing-notus, cong-natus, or inyotus, conyatus? Mr Munro (and I agree with him) holds that it does not. I account for it by supposing the Romans to have reduced the mass of consonants, the whole of which they were unable to pronounce, by omitting as usual (§ 31) the former n, that of the preposition, rather than the later and radical n.
(Roby 1887, p. lxxxi)
I am aware of a somewhat similar phenomenon to what Roby suggests (loss of a coda nasal because of phonotactic constraints) in the allomorphy of the Cypriot Greek definite article. According to Ringen and Vago (2011):
In Cypriot Greek the definite articles ton (masculine) and tin (feminine) lose their final nasal consonant if the next word begins either with a consonant cluster or a geminate., motivated by a *CCC constraint. The facts are as in (11) (Muller 2001).
(11) Cypriot Greek definite article allomorphy
a. Final nasal stays before V or C
ton ápparon 'the horse'
ton tixon 'the wall'
b. Final nasal deletes before CC
ti psačín 'the poison'
to flókkon 'the mop'
c. Final nasal deletes before G
to pparán 'the money'
to ttaván 'the stew'
Regressive assimilation to nasality occurred in other Latin sound changes
Comparison with other clusters in Latin and their history suggests that *kn, *gn > [ŋn] is a plausible sound change. Allen gives as an example the change of original *-pn- to -mn- in Latin words such as somnus. Although abn-, with the letter B rather than M, occurred at the start of Latin words prefixed with ab-, Allen says that amnegauerit actually exists as an inscriptional variant of abnegauerit.
Note that etymological comparison with other languages and internal reconstruction shows that in some cases Classical Latin gn seems to have originated from some kind of partial assimilation of *k to a following *n (for example, Wiktionary says that the word dignus comes from the Proto-Indo-European root *deḱ-).
Assimilation of plosives to nasals before nasal consonants is known to be a sound change that has occurred in other languages; for example, in Korean, plosives like p t k were nasalized to m n ŋ before nasal consonants.
In contrast, it's difficult for me to think of languages that had a sound change of /k/ to oral [g] before /n/. In my accent of English, there may be a single isolated example of historical /kn/ to [gn] in the word "acknowledge" (I made an ELU post about it here), but that's not a regular sound change (and I have [kn] in many other words.)
I can only think of a few things that seem like they might be comparable in Latin:
The word segmentum is said to be etymologically related to secō: this looks like an example of *k voicing to oral [g] before m in Latin. Sen (2015) gives a somewhat complicated explanation for the use of [g] rather than [ŋ] here: he attributes it to differences in the timing of the pronunciation of /n/ and /m/. Apparently, /n/ is articulated more quickly than /m/, which is supposed to have caused more overlap with the preceding /g/, resulting in greater perception of [ŋ] which eventually led to phonologization of [ŋ] as a conditioned nasal allophone of /g/ before /n/, but not before /m/ (pp. 183-184). Sen suggests that the existence of variation between word-forms with /gm/ and /gVm/ in Latin provides some additional evidence for a longer transition between /g/ and /m/ than between /g/ and /n/: the specific examples are drachma~drac(h)uma and tegmen~tegimen~tegumen.
There may be an example of coda /k/ assimilating to become voiced, but non-nasal g before the resonant /l/:
Sen (2011a) argues that the sonorants /r, l, m, n/ were phonologically specified as [+voice] ... in syllable-initial position ... Hence, we see regressive voice assimilation triggered by a syllable-initial sonorant in *nek̲-l̲egoː > neg̲l̲egoː ‘I neglect’, *sek̲m̲entom > seg̲m̲entum ‘piece’
(Sen 2012, p. 38)
However, I'm not sure that the explanation that Sen gives here for the presence of /g/ in neglego is correct, because we also find neg- before vowels in the word negotium, and perhaps in nego.
Hackstein (2017) says that nego may be derived from nec > neg, and offers two alternative explanations of the voicing: either it could have been conditioned by the word-final position of the plosive, as in ab < PIE *apo, or it could have originally been the result of assimilation in voicedness to a following voiced plosive ("*nekwe deikō > *neg dīcō like *apo-doukō > abdūcō (cf. Sommer 1948:275)") with the form neg later generalized for some reason before sounds other than voiced plosives (and then eventually lost again in favor of nec, except for in old compounds) (pp. 2-3). I'm left with the impression that we don't actually know exactly how neg- came to be used in words like neglego.
There were spelling variants with -NGN- instead of -GN-
In previous versions of this answer, I forgot to mention another piece of evidence that some sources say supports the reconstruction [ŋn]: apparently, the spelling NGN is sometimes used instead of GN. Ward (1944) gives a couple of examples:
On inscriptions, even late ones, we find some spellings with ngn, as singnifer, dingnissimē
It does seem natural to interpret ngn as indicating that the preceding vowel was followed immediately by a nasal consonant, rather than by an oral plosive.
However, Romance reflexes are often thought to point to [gn] rather than [ŋn]
As you can see above, there are various pieces of evidence that support the reconstruction [ŋn]. But on the other hand, Allen says (p. 24) that some Romance words, such as French lein or Southern Italian liunu from Latin lignum, show the same development of coda g to a semivowel that occurs in non-nasal environments. This has been seen as evidence that the pronunciation [gn] may have been used after Classical times, possibly due to influence from the spelling. (Maiden (1995) also suggests this is a piece of evidence against reconstructing [ŋn] in Proto-Romance.) I'm a bit confused by this argument, though, because I'm not sure why [ŋ] is thought to be less likely than [g] to vocalize to [j] or [w].
No Latin grammarians seem to mention a pronunciation of -gn- as [ŋn]
In addition to the supposed problems with explaining how the Romance reflexes of -gn- could have developed from [ŋn], Allen notes that Latin grammarians "are strangely silent about any nasal pronunciation of g" (p. 24). In contrast, there are descriptions of the pronunciation of n as [ŋ] before g or c.
Linguistic sources do tend to classify -gn- as containing the phoneme /g/
To be clear, all of the discussion in the previous sections is about the phonetic realization of gn. As far as I know, from a phonemic perspective it's standard to analyze gn as /gn/, and it seems to behave this way in contexts like the formation of diminutives: for example, the nouns signum, tignum correspond to the diminutive forms sigillum, tigillum where the g would have been pronounced as [g]—a voiced velar plosive, not a nasal. (I don't know how old these diminutive formations are thought to be.)
2017. Hackstein, Olav. "Latin negō *‘not I’." The 36th East Coast Indo-European Conference. Cornell University, Ithaca, New York.
2015. Sen, Ranjan. Syllable and Segment in Latin.
2012. Sen, Ranjan. "Reconstructing Phonological Change: Duration and Syllable Structure in Latin Vowel Reduction"
2011. Ringen, Catherine O. and Vago, Robert M. "Geminates: Heavy or Long?" Handbook of the Syllable, edited by Charles Cairns, Eric Raimy.
1995. Maiden, Martin. A Linguistic History of Italian.
1978. Allen, W. Sidney. Vox Latina: A Guide to the Pronunciation of Classical Latin. Second Edition. (First edition published 1965)
1944. Ward, Ralph L. "Afterthoughts on g as ŋ in Latin and Greek." Language, Vol. 20, No. 2 (Apr. - Jun., 1944), pp. 73-77.
1887. Roby, Henry John. A grammar of the Latin language from Plautus to Suetonius.