The syntax and meaning of the expression "ad conventus agendos" are actually both quite settled. What is not clear, as you have discovered, is how you get from one to the other. To explain this I'll have to go through several levels of explanation and two seeming digressions. I'm basing my explanation primarily on Woodcock's A New Latin Syntax, pp 160-163.
First, gerundives and infinitives have basically the same semantics, which are roughly equivalent to the gerund in English. For example, "cantare me delectat" ("singing pleases me") and "veni cantandi causa" ("I came for the purpose of singing"). The difference between the two is that they are in complementary distribution. Infinitives must be used when the nominative is called for, and the gerund must be used for all the other cases.
Latin has many ways to express purpose. One of those that works well for short phrases is using "ad" with a gerund. By this logic, the phrase in question should rightly be "*ad conventus agendum." "Agendum" is a gerund in the accusative case as the complement of "ad," and "conventus" is in the plural accusative as the object of the verbal noun "agendum." The literal meaning is "for (the purpose of) holding the (judicial) meetings."
As I understand it, "*ad conventus agendum" is good Latin; however, the use of a gerund with an accusative object was avoided during the period of Classical Latin. Apparently, speakers/writers thought it odd to have two different uses of the accusative right next to each other and with endings that did not agree. To remedy this oddness, they had to change the expression, by using a gerundive instead of the gerund.
Here is the first digression.
Gerundives, despite the similarity in names and occasional usage, may have had a different origin from gerunds. The endings used to create gerundives were originally used even with adjectives and intransitive verbs, such as "iucundus," "moribundus," and "secundus." When used with transitive verbs, the ending created a kind of participle associated with modal meanings. Usually it is said that the meaning has to do with necessity, as in "Delenda est Carthago!" (Carthage must be destroyed!); however, the meaning can be vaguer, including fitness (e.g., "nunc est bibendum", an impersonal neuter nominative and not a gerund in the nonexistent nominative case.) ("Now is the time to drink"), and even mere passive progressive or passive future ("quod erat demonstrandum").
End of first digression.
Speakers/writers wanting to avoid the clash of two different accusatives with different agreement in the expression "*ad conventus agendum" could turn to the similar sounding gerundive and change the expression to "ad conventus agendos." This expression solves the problem by having two nominal expressions in agreement as part of the complement of the preposition "ad," but seems to create three new problems: (1) Gerundives are active in meaning, whereas gerundives are passive, (2) gerundives have this meaning of necessity/fitness/futurity that doesn't apply to gerunds at all, and (3) the object of "ad" used to be a verbal idea (i.e., "holding"), but now it is just a mere concrete noun modified by something akin to a participle (i.e., "meetings to be held"),
Here's how each of these problems is resolved.
(1) The switch from the active to the passive is momentous syntactically, but unimportant semantically, since in either case the same "object,"--i.e., the "meetings"--is involved. As an example, saying either "I have things to do" or "I have things to be done" basically expresses the same meaning, regardless of whether we use an expression that is active or passive in form. Latin does this all the time by using past participles in ablative absolutes, where active participles would logically seem more appropriate. Greek has such participles and uses them often. Latin generally doesn't and so just uses equivalent expressions in the passive. In the case of deponent verbs, where Latin actually does have "active" past participles, it doesn't hesitate to use them.
(2) When the gerundive is used in place of the gerund, the gerundive's special modal semantics can simply be dropped and replaced by those of the plain gerund. In expressions of purpose, such as the one in question, the semantics of fitness and futurity are even somewhat helpful to express purpose and so don't need to be completely ignored in this case.
(3) In the gerund expression "*ad conventus agendum, the complement of "ad" is the verbal idea of "holding"; while in the gerundive expression "ad conventus agendos," the complement of "ad" is simply the meetings themselves. This seems in appropriate on the surface, since Caesar was not simply going for meetings, but for the purpose of holding meetings himself. In Latin, the semantics are clear that Caesar will hold the meetings and not someone else. The solution is how Latin often treats nouns accompanied by participles.
Begin second digression.
Latin uses and expression called the ablative absolute to express the circumstance that accompany or lead to something else. For instance, you can say: "Urba condita, rex factus est Romulus" ("With the city having been founded, Romulus was made king"). Notice that there is no word that expresses circumstance; nevertheless, the meaning is not "With the founded city, Romulus became king." The phrase "urbe condita" is promoted beyond a nominal idea ("the founded city") to a verbal idea ("the founding of the city").
This type of "promotion" is not limited to ablative absolutes. For instance, one way of identify the year was to calculate it from the founding of the city, i.e., "ab urbe condita." Again this does not mean "from the founded city," but from the founding of the city. The nominal expression has been promoted into a verbal idea.
This type of promotion also occurs in certain expressions in English, where you can say "The city having been founded is something that pleases me (even though I think the city itself is actually ugly and unappealing)." ("Urbs condita me delectat...") Even though the syntax makes it look like your talking about the city, your are actually talking about the founding of the city.
Even when we translate the ablative absolute "urbe condita" literally as "with the city having been founded," we really mean "with the founding of the city." Again, its the verbal idea and not the mere modified noun that we are referring to.
End second digression.
In the expression "ad conventus agendos," "agendos" acts like a progressive/future passive participle. Although syntactically "conventus" is the complement of "ad," semantically it is the verbal idea of "holding the meetings" that is the true complement, just as "ab urbe condita" doesn't mean "from the founded city," but rather "from the founding of the city." Thus "ad conventus agendos" means "for the purpose of holding (judicial) meetings. (By the way, a more technical translation of "conventus" in this case would be "assizes," which is basically a term for periodic, pop-up courts that could be held in varying locations. Apparently, Caesar had to hold court every year to judge whatever cases were pending within his jurisdiction.)
This long explanation was a very complicated way of saying that speakers/writers of Classical Latin found it awkward to use an accusative gerund with an accusative object and so replaced it with a gerundive in gender and number agreement with the object, since they found that the gerundive had semantics that were similar to the gerund.
