Logeion, s.v. impudicus, gives Martial 6.70.5 which mind you isn't specified as being the middle finger. But there's an interesting and amusing commentary on that passage in an article (CJ 47:67) on Roman Elementary Mathematics by J. Hilton Turner, which makes it pretty clear that it was. Elsewhere in the same article, there's a translation of a chunk of Bede in which he talks of the middle finger (clear from the context, despite no Latin original in the article) as impudicus.
In Muratori (XI, part 3, p126), in the anonymous Liber de computo sive kalendario attributed to Cyril of Alexandria, §138 there's another digitus impudicus; here too in the context of representing numbers on one's fingers.
Roger Pack, "Catullus, Carmen V: Abacus or Finger-Counting?", AJP 77:47-51 at JSTOR builds on another part of the Turner article, by suggesting that the Catullus poem maybe is best viewed not on an abacus with Turner, but in finger-counting, and there's a bit about the impudicus in his article as well.