As far as I am aware, the classical pronunciation of -gn- (as in magnus) is not [gn] but [ŋn]. How do we know that this is in fact how -gn- was pronounced?
We don't know for sure how -gn- was pronounced in Classical Latin. There are a few arguments for reconstructing the pronunciation of -gn- as [ŋn], or more specifically [ŋ.n], with a syllable break between the two consonants. (This syllabification would explain why the preceding syllable is always metrically "heavy"—or in the misleading old-fashioned way of putting it, the preceding vowel is "long by position". The idea that vowels before -gn- were always "long by nature" is probably erroneous; see Nathaniel's answer to Are vowels long before “gn”?)
But there are also some complications with this reconstruction, and possible reasons to prefer reconstructing the pronunciation of -gn- as [g.n] (with a voiced oral velar plosive) for at least some time periods or some speakers.
As usual, a good place to start for questions about Classical Latin pronunciation is W. Sidney Allen's Vox Latina.
Several pieces of evidence point towards a pronunciation [ŋn]
Allen (p. 23) describes several reasons for thinking that -gn- was pronounced as [ŋn] in Classical Latin.
ĕ > ĭ before [ŋ]
A vowel change of etymological short e to short i is seen before the clusters ng, nc, and gn; we can unify the description of these environments as "before [ŋ]" if we assume that gn represented [ŋn].
This sound change does not seem to have applied to ĕ before -gm-. Allen says that this is commonly viewed as a sign that -gm- was not pronounced with [ŋ] (which is perhaps surprising from a theoretical point of view, as both M and N are nasal consonants, and they would generally be expected to have similar effects on the pronunciation of preceding consonants).
It is generally assumed that g did not have a nasal pronunciation before m, as in tegmen, segmentum, since the change of ĕ to ĭ does not occur in these words. However, since original gm seems to have given mm (e.g. flamma from flag-ma, cf. flagro), all examples of gm may have arisen later, e.g. by syncope, after the change of ĕ to ĭ was operative (cf. the unsyncopated forms tegimen, integumentum). The possibility of a pronunciation of g as [ŋ] here is therefore not entirely excluded-though it cannot be safely recommended.
(Allen 1978, p. 25)
This sound change seems to be one of the strongest pieces of evidence for reconstructing the pronunciation of -gn- as [ŋn].
Examples: dignus, ilignus, lignum, signum (etymologies given below).
Prefixing in- or con- to gn- gives ign- or cogn-
Words like cognatus and ignobilis contain the prefixes con- and in-, which typically end in a nasal consonant that assimilates in place to a following plosive or nasal (as in compugno, impossibilis, committo, immortalis). If word-medial gn in Latin was pronounced as [ŋn], then the use of gn in these words can be explained as the result of the usual assimilation of /n/ > [ŋ] before /g/, followed by a simplification of the tautosyllabic onset cluster /gn/ to /n/, with a heterosyllabic cluster [ŋ.n] as the result.
If gn was just pronounced as [gn], it would mean that the nasal consonant from the end of the prefixes was dropped entirely in these words. That seems a bit implausible to me, but I guess it's not impossible. In fact, I did find a source from the 19th century that argues for that interpretation of prefixed words containing gn, and rejects the [ŋ.n] interpretation:
Had the Romans retained the n of the prepositions before gn, they would have felt bound to pronounce ing-gnotus, cong-gnatus, but would practically have pronounced ing-notus, cong-natus. But they did not retain the n and write ingnotus, congnatus, but ignotus, cognatus. What is the explanation ? Does this represent a pronunciation ing-notus, cong-natus, or inyotus, conyatus? Mr Munro (and I agree with him) holds that it does not. I account for it by supposing the Romans to have reduced the mass of consonants, the whole of which they were unable to pronounce, by omitting as usual (§ 31) the former n, that of the preposition, rather than the later and radical n.
(Roby 1887, p. lxxxi)
I am aware of a somewhat similar phenomenon to what Roby suggests (loss of a coda nasal because of phonotactic constraints) in the allomorphy of the Cypriot Greek definite article. According to Ringen and Vago (2011):
In Cypriot Greek the definite articles ton (masculine) and tin (feminine) lose their final nasal consonant if the next word begins either with a consonant cluster or a geminate., motivated by a *CCC constraint. The facts are as in (11) (Muller 2001).
(11) Cypriot Greek definite article allomorphy
a. Final nasal stays before V or C
ton ápparon 'the horse'
ton tixon 'the wall'
b. Final nasal deletes before CC
ti psačín 'the poison'
to flókkon 'the mop'
c. Final nasal deletes before G
to pparán 'the money'
to ttaván 'the stew'
*kn seems to have turned into gn, which might make more sense if gn = [ŋn]
Internal reconstruction and etymological comparison with other languages show that in some cases, Classical Latin gn seems to have originated from some kind of partial assimilation of *k to a following *n. Examples: dignus, īlignus, and gnixus, an old spelling of nīxus, which is related to nītor (etymologies given below).
The apparent replacement of *kn- with gn- would be somewhat surprising if gn were pronounced as [gn] with oral [g]. In general, we expect languages that have words that start with [gn] to also have words that start with [kn]: this is discussed in Sen (2015), which says
On the basis of a typological assessment of attested initial sequences, Stephens (1978) comes to [the conclusion that] initial <gn> in Latin was pronounced [ŋn], since the data examined by Greenberg (1965) point to an implicational universal whereby languages which have initial [gn] also have initial [kn]; as early Latin shows no evidence for [kn], the sequence written <gn> cannot have been [gn] and therefore must have had its word-internal value of [ŋn].10 [...]
10Cser (2011: 70-71, 78), however, points out that initial [ŋn] would be the only onset cluster containing (i) a nasal and (ii) two sonorant consonants, and concludes that in the intermediate stage between very early initial [gn] and eventual simple [n] (when, he argues, the lexical representations of these words contained a stem-initial floating C-Place node), the pronunciation 'must have involved a degree of variation [between [gn] and [n]] whose details are no longer fully recoverable'.
Even in word-internal position, it's difficult for me to think of languages that have had a sound change of /k/ to oral [g] before /n/. In my accent of English, there may be a single isolated example of historical /kn/ to [gn] in the word "acknowledge" (I made an ELU post about it here), but that's not a regular sound change (and I have [kn] in many other words.)
I can only think of a few things that seem like they might be comparable in Latin:
The word segmentum is said to be etymologically related to secō: this looks like an example of *k voicing to oral [g] before m in Latin. Sen (2015) gives a somewhat complicated explanation for the use of [g] rather than [ŋ] here: he attributes it to differences in the timing of the pronunciation of /n/ and /m/. Apparently, /n/ is articulated more quickly than /m/, which is supposed to have caused more overlap with the preceding /g/, resulting in greater perception of [ŋ] which eventually led to phonologization of [ŋ] as a conditioned nasal allophone of /g/ before /n/, but not before /m/ (pp. 183-184). Sen suggests that the existence of variation between word-forms with /gm/ and /gVm/ in Latin provides some additional evidence for a longer transition between /g/ and /m/ than between /g/ and /n/: the specific examples are drachma~drac(h)uma and tegmen~tegimen~tegumen.
There may be an example of coda /k/ assimilating to become voiced, but non-nasal g before the resonant /l/:
Sen (2011a) argues that the sonorants /r, l, m, n/ were phonologically specified as [+voice] ... in syllable-initial position ... Hence, we see regressive voice assimilation triggered by a syllable-initial sonorant in *nek̲-l̲egoː > neg̲l̲egoː ‘I neglect’, *sek̲m̲entom > seg̲m̲entum ‘piece’
(Sen 2012, p. 38)
However, I'm not sure that the explanation that Sen gives here for the presence of /g/ in neglego is correct, because we also find neg- before vowels in the word negotium, and perhaps in nego.
Hackstein (2017) says that nego may be derived from nec > neg, and offers two alternative explanations of the voicing: either it could have been conditioned by the word-final position of the plosive, as in ab < PIE *apo, or it could have originally been the result of assimilation in voicedness to a following voiced plosive ("*nekwe deikō > *neg dīcō like *apo-doukō > abdūcō (cf. Sommer 1948:275)") with the form neg later generalized for some reason before sounds other than voiced plosives (and then eventually lost again in favor of nec, except for in old compounds) (pp. 2-3). I'm left with the impression that we don't actually know exactly how neg- came to be used in words like neglego.
Nasalization from regressive partial assimilation occurred in other Latin consonant clusters
Comparison with other clusters in Latin and their history suggests that *kn, *gn > [ŋn] is a plausible sound change. Allen gives as an example the change of original *-pn- to -mn- in Latin words such as somnus. Although abn-, with the letter B rather than M, occurred at the start of Latin words prefixed with ab-, Allen says that amnegauerit actually exists as an inscriptional variant of abnegauerit.
Assimilation of plosives to nasals before nasal consonants is known to be a sound change that has occurred in other languages; for example, in Korean, coda plosives like p t k were nasalized to m n ŋ before nasal consonants.
There were spelling variants with -NGN- instead of -GN-
Apparently, the spelling <NGN> is sometimes used instead of <GN>. Ward (1944) gives a couple of examples: "On inscriptions, even late ones, we find some spellings with ngn, as singnifer, dingnissimē" (p. 73).
It does seem natural to interpret <NGN> as indicating that the preceding vowel was followed immediately by a nasal consonant, rather than by an oral plosive. However, I'm not sure how common or how significant this variant spelling is. Uncommon variant spellings aren't always closer to the standard pronunciation of a word than the standard spelling: for example, "renpent" exists as a rare misspelling of the English word "repent", even though the typical pronunciation of this word contains no corresponding nasal consonant sound before the /p/.
Lindsay (1894) says "Singnifer, on a soldier's grave (C.I.L. vi. 3637)" (§69, p. 70) is best explained as an example of more general variation in inscriptions in the use of the letters N and M before consonants: "Occasionally a nasal is wrongly inserted in plebeian and late inscriptions, m before Labials, n before Dentals and Gutterals, e.g. semptem, singnifer" (§64, p. 66).
However, Romance reflexes are often thought to point to [gn] rather than [ŋn]
As you can see above, there are various pieces of evidence that support the reconstruction [ŋn]. But on the other hand, Allen says (p. 24) that some Romance words, such as French lein or Southern Italian liunu from Latin lignum, show the same development of coda g to a semivowel that occurs in non-nasal environments. This has been seen as evidence that the pronunciation [gn] may have been used after Classical times, possibly due to influence from the spelling. (Maiden (1995) also suggests this is a piece of evidence against reconstructing [ŋn] in Proto-Romance.) I'm a bit confused by this argument, though, because I'm not sure why [ŋ] is thought to be less likely than [g] to vocalize to [j] or [w].
Allen does say that the Sardinian reflex of -nn- makes sense as a development from [ŋn]. And Sihler (1995) views the Romance evidence as pointing towards a Latin pronunciation with [ŋn]: he treats the Romanian outcome of -mn- as a particularly strong piece of evidence in favor of [ŋn], but he also says that Italian [ɲː], as in legno, is "easiest to understand if traced to an original [ŋn]", although "it admits of other explanations" (§220a).
No Latin grammarians seem to mention a pronunciation of -gn- as [ŋn]
In addition to the supposed problems with explaining how the Romance reflexes of -gn- could have developed from [ŋn], Allen notes that Latin grammarians "are strangely silent about any nasal pronunciation of g" (p. 24). In contrast, there are descriptions of the pronunciation of n as [ŋ] before g or c.
Linguistic sources do tend to classify -gn- as containing the phoneme /g/
To be clear, all of the discussion in the previous sections is about the phonetic realization of gn. As far as I know, from a phonemic perspective it's standard to analyze gn as /gn/, and it seems to behave this way in contexts like the formation of diminutives: for example, the nouns signum, tignum correspond to the diminutive forms sigillum, tigillum where the g would have been pronounced as [g]—a voiced velar plosive, not a nasal. (I don't know how old these diminutive formations are thought to be.)
dignus: from Proto-Indo-European *deḱ-, according to Wiktionary and Sihler 1995 (§220).
īlignus 'oaken': from the same root as īlex, īlicis 'holm-oak', according to Sihler 1995 (§220).
lignum: from Proto-Indo-European *leǵ-no-, according to Wiktionary
nītor: from Proto-Indo-European *kneygʷʰ-, according to Wiktionary and Sihler 1995 (§220). The related word nixus (=nīxus) is supposed to have an old spelling gnixus.
signum: from PIt. *seknom, according to Wiktionary
2017. Hackstein, Olav. "Latin negō *‘not I’." The 36th East Coast Indo-European Conference. Cornell University, Ithaca, New York.
2015. Sen, Ranjan. Syllable and Segment in Latin.
2012. Sen, Ranjan. "Reconstructing Phonological Change: Duration and Syllable Structure in Latin Vowel Reduction"
2011. Ringen, Catherine O. and Vago, Robert M. "Geminates: Heavy or Long?" Handbook of the Syllable, edited by Charles Cairns, Eric Raimy.
1995. Maiden, Martin. A Linguistic History of Italian.
1995. Sihler, Andrew L. New Comparative Grammar of Greek and Latin.
1978. Allen, W. Sidney. Vox Latina: A Guide to the Pronunciation of Classical Latin. Second Edition. (First edition published 1965)
1944. Ward, Ralph L. "Afterthoughts on g as ŋ in Latin and Greek." Language, Vol. 20, No. 2 (Apr. - Jun., 1944), pp. 73-77.
1894. Lindsay, W. M. The Latin Language: An Historical Account of Latin Sounds, Stems and Flexions.
1887. Roby, Henry John. A grammar of the Latin language from Plautus to Suetonius.