Partial answer!
As far as I can tell, most mathematical discourse would be done in Greek. Latin was used for engineering purposes, but speaking unambiguously about mathematics became rather awkward.
For example, Vitruvius's description of the Pythagorean formula, from De Architectura IX.6:
namque si sumantur regulae tres e quibus una sit pedes III altera pedes IIII tertia pedes V, eaeque regulae inter se compositae tangant alia aliam suis cacuminibus extremis schema habentes trigoni, deformabunt normam emendatam. ad eas autem regularum singularum longitudines si singula quadrata paribus lateribus describantur, quod erit trium latus, areae habebit pedes VIIII, quod IIII, XVI, quod V erit, XXV. ita quantum areae pedum numerum duo quadrata ex tribus pedibus longitudinis laterum et quattuor efficiunt, aeque tantum numerum reddidit unum ex quinque descriptum.
Translation:
For if three straight rods are acquired (and from these let one be three feet, the other four feet, and the third five feet) and these rods are arranged together so that they touch each other at the very tip, making the shape of a triangle, they will form a perfect right angle. Then, if squares are marked out, with sides equal to the lengths of each of the rods, then the [square with the] side [length] of three will have an area of nine feet, the four, sixteen, and the five, twenty-five. Thus how many feet of area two squares with sides of three and four feet in length contain, is equal to how many one [square] marked out from five [feet] contained.
In other words: a² + b² = c², expressed geometrically and in far too many words.
Frontinus' De Aquis shows similarly how unwieldy numbers could be in Roman numerals. A small excerpt starting from I.50:
Fistula quadragenaria: diametri digitos septem 𐆑 𐆒 ℈III, perimetri digitos XXII 𐆐 𐆐 𐆑, capit quinarias XXXII S 𐆑.
Fistula quadragenum quinum: diametri digitos septem S 𐆒 ℈ octo, perimetri digitos XXIII S 𐆐 𐆑 𐆒, capit quinarias XXXVI S 𐆑 𐆒 ℈ octo; in usu non est.
Fistula quinquagenaria: diametri digitos septem S 𐆐 𐆐 𐆑 𐆒 ℈ quinque, perimetri digitos XXV 𐆒 ℈VII, capit quinarias XL S 𐆐 𐆒 ℈V.
Fistula quinquagenum quinum: diametri digitos octo 𐆐 𐆐 ℈ decem, perimetri digitos XXVI 𐆐 𐆑 𐆒, capit quinarias XLIIII S 𐆐 𐆑 𐆒 ℈II; in usu non est.
Translation from LacusCurtius, since I'm not familiar enough with the fractions:
The 40‑pipe: 7 digits plus 1/12 plus 1/24 plus 3/288 in diameter; 22 digits plus 5/12 in circumference; it has a capacity of 32 quinariae plus 1/2 plus 1/12.
The 45‑pipe: 7 digits plus 1/12 plus 1/24 plus 8/288 in diameter; 23 digits plus 1/2 plus 3/12 plus 1/24 in circumference; it has a capacity of 36 quinariae plus 1/2 plus 1/12 plus 1/24 plus 8/288; is not in use.
The 50‑pipe: 7 digits plus 1/2 plus 5/12 plus 1/24 plus 5/288 in diameter; 25 digits plus 1/24 plus 7/288 in circumference; it has a capacity of 40 quinariae plus 1/2 plus 2/12 plus 1/24 plus 5/288.
The 55‑pipe: 8 digits plus 4/12 plus 10/288 in diameter; 26 digits plus 3/12 plus 1/24 in circumference; it has a capacity of 44 quinariae plus 1/2 plus 3/12 plus 1/24 plus 2/288; is not in use.
So it is more likely that educated Romans preferred to speak about mathematics in Greek—especially as the most important mathematical texts at the time had all been written in Greek.
As Vitruvius's description shows, mathematics at that time was more focused on geometry than on formulas. Rather than calculating a number, the goal would be to construct a line segment of that length, or a triangle of that area, etc. This made certain things far more difficult: numerical methods can approximate the cube root of 2 to arbitrary precision, but with only a compass and straightedge it is impossible to construct exactly. Plato famously mocked this failing when the geometers couldn't double the volume of a cube, as related by Vitruvius, Plutarch, and others.
However, this also had the advantage of making it much easier to read out loud. Vitruvius's description of the Pythagorean formula may be harder to visualize than a diagram, but unlike a diagram it can be spoken without losing any information. As another example, from Euclid's Elements:
κγʹ. Παράλληλοί εἰσιν εὐθεῖαι, αἵτινες ἐν τῷ αὐτῷ ἐπιπέδῳ οὖσαι καὶ ἐκβαλλόμεναι εἰς ἄπειρον ἐφ ̓ ἑκάτερα τὰ μέρη ἐπὶ μηδέτερα συμπίπτουσιν ἀλλήλαις.
Translation:
- Lines which are in the same plane, and are projected infinitely in both directions, but touch each other in neither direction, are "parallel".
That's pretty much the same way I would describe parallel lines to someone if I didn't have pen and paper on hand.
(More to come when I find more examples: later usage, for instance.)
