This answer is based on the discussions in Weiss and Sihler.
The regular development is oi > ū: e.g. ūnus : Gk. οἴνη "one (on a die)". There are inscriptions up until the second half of the 2nd century BC that still show OI, e.g. LOIDOS for later lūdus.
However, there are a few words that have oe instead of ū: e.g. moenia, poena, Poenus, foedus "ugly", foedus "treaty". In all of these words the oe is preceded by a labial and followed by a coronal. (In other words with oe, e.g. coepī, proelium, the oe is secondary and results from the combination of a prefix ending in -o with the vowel e.) But, apparently it is not a question of simple phonetic conditioning, as Weiss says that "this cannot be the whole story since evidently oe did monophthongize in this environment in some words". Examples include some words from the same roots as the words above, e.g. Pūnicus, mūnīre. Sihler mentions the theory that the words which retained oe are "technical or literary" words, but this does not seem to be true of all of them. Neither Weiss nor Sihler mentions the possibility that these words are loans from a non-standard dialect.
So it looks like there's no complete answer: the outcome oe is partly phonetically conditioned, but there may be something more going on that we haven't figured out.