Your guesses are very close: Greek terms for "vocabulary" or "glossary" are λέξεις and γλῶσσαι (both nominative plurals).
On λέξις, LSJ says:
Gramm., a word peculiar in form or signification: hence λέξεις is the older term for a glossary, Ῥοδιακαὶ λέξεις a glossary of Rhodian phrases
and on γλῶσσα:
obsolete or foreign word, which needs explanation, Arist. Rh.1410b12, Po.1457b4, Plu.2.406f: hence Γλῶσσαι, title of works by Philemon and others
The lexicon of grammatical terms in Eleanor Dickey's Ancient Greek Scholarship implies something similar, defining γλῶσσα as "dialect, language, obsolete or dialectal word" and λέξις as "word, phrase, speech, diction, style, peculiar word (hence λέξεις 'glossary'), text of an author (as opposed to commentary)". So it seems that of the two, insofar as there was a difference in usage, λέξεις may have been the more general term for "vocabulary", and γλῶσσαι a more specific term for vocabulary consisting of obscure words.