In the classical period, olive oil was considered a must-have piece of equipment for an athlete. The exact details of its use aren't known perfectly, but it appears that it was coated on an athlete's body and then covered with fine sand, in preparation for competition (especially wrestling). This would protect the body, especially from heat.
Athletes in Greece (yes, the original Olympians) were especially common users of the mixture, in a variety of events. Thus, if a person shied away from olive oil like this, it implied that they were cowardly, and not ready for competition. This practice was carried over the Latin-speaking Romans (this history of garlic and olive oil mentions to Roman use) - but Horace may still have been alluding to the Greek practice.
After the competition, special instruments were used to clean off the mixture (one being a strigil, and another being a common sponge). Olive oil use has even made its way into art, as A Companion to Sport and Spectacle in Greek and Roman Antiquity notes (see also the points of general use by Eutruscan athletes).
Source: The Olympic Games in Antiquity
Another important detail is that other translations of that verse exist. One version instead goes
Like poison loathes the oil
Indeed, looking at the passage as a whole, we see many context clues that point out the central idea of sport - some, interestingly enough, references to Greece:
