It's actually amazingly simple.
To raise > to raise up from its place > to remove > to destroy by removing > to destroy
The visual metaphor here is astounding: imagine you want to topple a temple. What better way to do so than by uprooting from its foundations the columns upon which the temple depends? If you remove the columns, the temple is destroyed.
Auferre works on the same principle, though that shouldn't be surprising since the perfect principle parts of ferre is the old perfect principle parts of tollere (and why tollere has the sus- instead)!