For my answer, I will use material from a 1931 article written to address this very issue: "The Use of Forem and Essem" by Winnie D. Lawrence, available on JSTOR.
While essem was always a good literary word, forem, after Plautus, gave evidence of decline in this respect. In later writers, especially in Sallust and Tacitus, it was an affected archaism, but in the poets, metrical convenience also was often the deciding factor in its use. The etymology of forem suggests a future meaning and those who employ it most frequently have a strong feeling for this quality. Syntactically, forem is met most often in conditional sentences, being employed in this way in 42% of its occurrences, while essem is so used in only 15%.
Here are some general points sprinkled with the findings of Lawrence's study, which includes a fairly rigorous statistical analysis:
forem has a future tendency
forem comes from the PIE root "BHEU", essem from "ES." "BHEU" survives in other forms like fu-, fore, and the -ba- and -bo- endings of the imperfect and future (and also English's be!). A rough equivalent "future subjunctive" survives in Spanish (fuere) and Portuguese (for). This future force was recognized by Roman authors, even though it can also be granted to essem:
Servius, commenting on the Aeneid, remarks with surprise when Vergil uses forem without any future kind of meaning:
modo praeteritum significat tempus, quod rarum est, saepius enim futurum significat. (Servius on Aen. III, 417)
Ilio tria fuisse audivi fata quae illi forent exitio. (Pl. Bacch. 953)
Sallust is especially prone to this kind of usage, using forem 33 times with a future sense (e.g. as a substitute for ero in indirect discourse) and essem only once. One quote in particular even juxtaposes essem and fores with these two different senses:
At Catilina ex itinere...litteras mittit: se...in exitium proficisci, non quo sibi tanti sceleris conscius esset, sed uti res public quieta foret... (Sal. Cat. 34, 2)
Note that this is only a tendency with multiple exceptions. However, as Lawrence remarks, "those who use forem most often have a strong feeling for its future quality."
forem is more archaic and colloquial
Earlier writers, such as Plautus, use forem comparatively frequently (about 25% of the time). Lucretius uses forem more often than essem.
Later, more formal writers tend to shy away from heavy usage of forem. The most prominent example of this is Vergil, who does not use forem at all in the Eclogues or Georgics and uses it only 15 times in the Aeneid (we will see why in the next section).
Sallust used the word heavily, but was also often accused of over-employing archaisms.
forem is great for meter
In my opinion, the most cogent conclusion of Lawrence's study is that forem is chosen by poets not for any difference of meaning but because of metrical expediency. Forem is ideal for metrical schemes ending in an iamb, for instance. Likewise, forem is often placed after words ending in a vowel which would cause an elision. Some striking figures:
- Terence uses forem 8 times: 7 are at the end of a line requiring an iamb, and 1 is after a vowel that would otherwise be elided.
- Lucretius: 21 out of 27 cases avoid elision
- Catullus: 7 out 9 avoid elision: in all 9 cases using essem would violate the meter
- Notably, Vergil, who is known for his elision, uses essem much more often.
forem fits better with certain constructions
Besides the previous cases, there is one particularly one construction in which forem predominates: conditionals. This is very likely a matter of sound, especially for poets, since si essem is unpleasant.
- Half of Terence's forem's are in conditionals
- Quintilian, Seneca Philosophus, and Pliny--who otherwise use forem very rarely--use forem at a much higher percentage in conditionals
- Tacitus also provides numerous examples, e.g. "Si in Britannia foret" (Ag. 40), "si mali forent" (Hist 1.49), "si vera forent" (Ann 2.8)
This is likely a question of sound and feel, since a sensitive prose or poetry writer will carefully balance his usage of s's. This seems to explain, for instance, some of the few times when Vergil opts for forem.
At the end of the day, though, we must always appeal to that invisible, inexplicable "choice" an author makes between two roughly equivalent words: the two words evidently have different "feels", but I would hesitate to formulate any kind of rules that apply to all particular cases.