It seems like memoriam meam fugit or memoria me fefellit would be good candidates. Here are some relevant passages I've found:
...et si qua sunt alia, quae nunc memoriam meam fugiunt (Columella, De Re Rustica 18.104.22.168)
nisi memoria me fallit (Aulus Gellius, Noctes Attica 20.1.14)
This longer passage from Quintilian also has a couple of relevant phrases:
pleraque gratiora sunt si inventa subito nec domo allata sed inter dicendum ex re ipsa nata videantur, unde illa non iniucunda schemata: "paene excidit mihi" et "fugerat me" et "recte admones" (Quinitilian, Institutio Oratoria 4.5.4).
Translation: there are many points which can be produced in a more attractive manner, if they appear to be discovered on the spot and not to have been brought ready made from our study, but rather to have sprung from the requirements of the case itself while we were speaking. Thus we get those not unpleasing figures such as "It has almost escaped me," "I had forgotten," or "You do well to remind me."