If I interpret you correctly, you're saying that there appears to be a sort of syntax-meaning mismatch involved here: on the one hand, as you point out, the syntax clearly forces us to interpret the nominal predicates as ablatives of description ("with such weakened age and with such infirm power"), but, on the other, a more natural interpretation (at least for you) would be the one whereby the participle/adjective should modify not the ablative noun but the nominative subject (contra the original text above and in agreement with "nemo Agrigenti neque aetate tam affectus neque viribus tam infirmus fuit qui...").
Accordingly, as you point out, a good translation of this text would be "No one in Agrigentum was either so weakened by age, or so infirm in strength...". Well, in my opinion, this is a good translation, which is, by the way, very similar to the one given by C. D. Yonge (1903): "No one in Agrigentum was either so advanced in age, or so infirm in strength". However, translations are translations and grammar is grammar...
In fact, it is important to point out that Latin offers the two grammatical possibilities, as can be shown by comparing your example above with the following one, also from Cicero:
Tertium genus est aetate iam adfectum, sed tamen exercitatione robustum; quo ex genere iste est Manlius cui nunc Catilina succedit (Cic. Cat. 2.20).
Furthermore, as noted above, the modification of aetas by the participle affecta is also very natural. This combination is not restricted to a context of non-prepositional "ablative of description" but can also be found in other contexts like the one exemplified below (by the way, cf. viribus infirmis in your text with infirmissima valetudine in the text below and cf. aetate affecta in both texts, which can lead us to conclude that this parallel modification with these two identical (!) adjectives was very pleasant to Cicero):
Est enim sine dubio domus iuris consulti totius oraculum civitatis; testis est huiusce Q. Muci ianua et vestibulum, quod in eius infirmissima valetudine adfectaque iam aetate maxima cotidie frequentia civium ac summorum hominum splendore celebratur. (Cic. De orat. 1.200)
Interestingly, Augustus S. Wilkins provided a grammatically relevant comment on adfectaque iam aetate in his Cicero. De oratore (OLMS, 1990: p. 184) by including the following nice text from A. Gellius, which, by the way, provides a very useful native intuition to learn the aspectual value of Latin prefixes (e.g. cf. ad-fecta aetate with con-fecta aetate, etc.).
Adfecta enim, sicuti Marcus Cicero et ueterum elegantissimi locuti sunt, ea proprie dicebantur, quae non ad finem ipsum, sed proxime finem progressa deductaue erant. (Gell. III.16.19)
To conclude, Latin offers both grammatical possibilities (see the original one in 1. & the invented/constructed one in 2. below): e.g. cf. (nemo) aetate affectā ((no one) 'with a weakened age') and (nemo) affectus aetate ((no one) 'weakened by age'); aetate tam affectā is an ablative of description in 1., whereas aetate in aetate affectus is an ablative of cause ('weakened by age') or, alternatively, an "ablative of specification": 'weakened with respect to age' in 2. Ditto for viribus: this noun is an ablative of description in 1. but is an ablative of specification/respect in 2.
"Nemo Agrigenti neque aetate tam affectā neque viribus tam infirmis fuit qui..."
"Nemo Agrigenti neque aetate tam affectus neque viribus tam infirmus fuit qui..."
NB: interestingly, some old dictionaries do give the reading infirmus (nominative sg.), hence modifying the subject nemo, instead of infirmis (ablative pl.), which modify viribus: e.g. see here & n.1 here. See also the comment below by Cerberus and my answer to it.
One could say: "ok, Mitomino, nice long story but, in my opinion, the interesting part of the question raised by the OP remains unanswered: why is it the case that some translators (even excellent ones such as C. D. Yonge) translate Cicero's text in 1. as it was the one in 2.? That is to say, you have emphasized the (otherwise quite obvious) morphosyntactic & semantic differences between 1. & 2. but you have not addressed the linguistic commonalities that can allow one to translate 1. as 2.".
Short(er) answer: quite probably, this is allowed by the fact that both aetate and viribus are inalienable possessed nouns, nouns that are obligatorily possessed by their possessors. Please see this link for some relevant discussion on this linguistic notion: note that the list includes "possessed noun originates from the possessor" (e.g. sweat, voice, strength, etc.) and "attribute of a known possessor" (e.g. name, age, etc.), i.a. If so, a minor qualification could be made on the following general statement above: "translations are translations and grammar is grammar." Despite the indisputable morphosyntactic differences between 1. & 2. above, in this particular case the OP's translation of Cicero's sentence and the one provided by C.D. Yonge can be said, to some extent (i.e. to the "extent" mediated by inalienable possession), to be grammatically motivated as well.
