What is the difference between the two adjectives ἀρχαῖος and παλαιός?
In particular, what word would fit the best to mean "history" between ἀρχαιολογία and παλαιολογία?
The standard word for historical study was in fact archaeology. While Thucydides' "Archaeology" may be a conventional term, Josephus' Antiquitates in Greek was the Archaiologia (ἀρχαιολογία), and historical works by Cleanthes, Hieronymus the Egypytian inter alia were all titled similarly.
That said, palaios is not semantically incorrect, and Appian uses the verbal form of palaiologos once.
As far as the difference goes, they essentially both developed enough extended meanings to be nigh synonymous. Both have been used for old and ancient people and things, though archaios is generally not used for "old people," in the sense of aged to a long life, while palaios can.
I post this rather irrelevant bit as an answer for practical reasons:
The House of Palaiologos (Greek: Παλαιολόγος), also found in English-language literature as Palaeologus or Palaeologue, was a Byzantine Greek noble family that rose to power and produced the last and longest-ruling dynasty in the history of the Byzantine Empire.
The origins of the Palaiologos family are unclear. According to several later oral traditions, the family had originated in Italy, supposedly in the city of Viterbo. As per this version, the family name Palaiologos (Palaios logos, lit. "old word") was a Greek translation of vetus verbum, a dubious etymology of Viterbo.
The etymology of their family name was believed to be "ragman" by Soviet-American Byzantinist Alexander Kazhdan, possibly referencing humble origins, whereas the French Byzantinist Jean-François Vannier believes the correct etymology to be "antique collector".