I've seen both, so obviously both were used, but are there circumstances that determine which option to use? Or is it something that changed over time?
Allen and Greenough (§50) refer to vesper as a second declension noun, except that it takes the ablative vespere and the locative vesperī.
Gildersleeve and Lodge provides more detailed guidance, citing the 2nd, 3rd, and 1st declensions:
Vesper, evening, has Acc. vesperum; Dat. Abl. vesperō; Pl. Nom. vespera of the Second Declension; Acc. vesperam; Abl. vesperā of the First; Gen. vesperis; Abl vespere; Loc. vespere, vesperī of the Third. (§68)
Theophilus Hall keeps it simple:
Vesper (also vespera), the evening has, according to the Second Declension, the acc. vesperum, but the Abl. usually according to the Third Declension, vespere, verperī. (§53)
Leonhard Schmitz is the only source I've found that suggests a different declension depending on meaning. In §81 he suggests that the meaning evening uses the second declension except for the ablative's third declension, while the meaning evening star always uses the second declension.
Overall, the general guidance is that the second declension is preferred except in the ablative, where the third declension is normally employed.