1. Yes, it's ambiguous.
This is only a hypothesis, inspired somewhat indirectly by this Marginalia blog post. I don't have nearly enough experience with the language to give this hypothesis much credibility—or to reject it, so here goes. Hopefully someone more knowledgeable can debunk it or corroborate it in comments. Do not take this as in any way authoritative. This post is essentially me thinking out loud.
Se refers to the subject currently having things said about it—not necessarily the grammatical subject, but the current grammatical "topic", regardless of its grammatical case.
The ambiguity occurs when multiple antecedents can reasonably be understood as the current topic—as happens inside a subordinate clause. The main clause has a topic and the inner clause has a topic. Consequently se is "attracted" simultaneously to both antecedents.
A variety of pressures make each antecedent attract the se more or less strongly. When one antecedent exerts a much stronger pressure than all others, there is no ambiguity. When two antecedents exert nearly equal pressures, there is ambiguity. So, there is no rule, only a competition in the listener's mind—and different antecedents may win in different listeners' minds.
Some pressures, each of which can be opposed or overridden by others:
The topic of the main clause is the topic of the whole sentence, and normally maintains that dominance even inside the subordinate clause—at least, it tends to attract se more strongly by default.
The beginning of a clause typically establishes its topic. Regardless of its case, the topic is the anchor to which the sentence gradually attaches information, usually culminating in the end of the clause, where the most important new thing is stated. Upon reaching the first word of a clause, normally the scope of what a pronoun can refer to is limited to subjects already mentioned in the sentence, though this is easily overridden at the start of a main clause (when there's nothing yet to refer to).
Analogous word orders, especially those felt to be canonical, add weight to analogous choices of antecedent.
Alterations of canonical word order add weight to choices of precedent that are not analogous to the canonical one.
Interpretations that don't make sense repel rather than attract.
Context: if preceding discourse has put a question "on the table", an antecedent of se that makes the sentence answer that question is favored.
Juxtaposed ipse, proprius, and semet, sese and the like add weight favoring finding the antecedent in the subordinate clause.
Irony, poetic meter, flair, whimsy—stylistic choices that run against conventional word order can reverse any of the above pressures. These depend on a listener's ability to "get it" and willingness to "play along".
Quintus dixit se Marcum odisse.
Se, coming first in the subordinate clause, here establishes its topic: Quintus said something about himself, namely that he hates Marcus. Pressures (1), (2), and (3) prevail, and there is practically no ambiguity (absent contextual pressures).
Quintus dixit Marcum se odisse.
The familiar and canonical word order Aliquis dixit aliquem se infinitivum, where se is an object of the infinitive verb and refers to aliquis, pulls se toward Quintus (1, 3). But Aliquis se odit is also a reasonable sentence, and it pulls se toward Marcum almost as strongly (2, 5). Absent other pressures, Quintus will most likely be heard as the antecedent, but it's not certain. A speaker and listener might even resolve the ambiguity differently, without even noticing that there is an ambiguity.
2. Here are some ways to disambiguate it.
Specifically, here are some ways to overpower (1), helping se refer to the subject of the subordinate clause—though, since there are no rules, there are no guarantees, either.
Marcum se odisse, Quintus dixit.
Here Marcum is the only available antecedent when se arrives (2). If Quintus is to own se, it'll have to steal it from an already entrenched defender (perhaps another pressure). Cicero may have used this trick in De Finibus 5.20, though other pressures are at work, too (3, 5, 6):
Nam voluptatis causa facere omnia, cum, etiamsi nihil consequamur,
tamen ipsum illud consilium ita faciendi per se expetendum et honestum et solum bonum sit, nemo dixit.
No one [neither Aristippus nor Hieronymus nor Carneades] has claimed that everyone acts for the sake of pleasure in the sense that
even if we accomplish nothing, the mere intention of so acting
is, because of itself alone, to be desired or honorable or the sole good [not because of Aristippus et al.)].
Another way out of the ambiguity is to use the passive voice:
A Quinto dicitur Marcus se odisse.
The passive voice makes Marcus the subject of both the main clause and the subordinate clause. Perhaps contextual pressure could still make se refer to Quinto, though. Moving Marcus to the beginning of the sentence and a Quinto to the end would add more pressures favoring Marcus as the antecedent.
Quomodo se aestimat Marcus? Non bene, puto. Quintus tamen dixit Marcum se odisse.
Here context pulls se toward Marcum (6), probably, but not certainly, excluding Quintus. A speaker who meant that Quintus said that Marcus hates himself would likely choose this wording spontaneously, without noticing the ambiguity. A listener is more likely to notice, but is still likely to take Marcum as the antecedent. (N.b. I am making this up as I am writing it. It's only a guess.)
Quintus dixit Marcum odisse se.
Here, moving se out of its canonical position and into the most important spot, the end of the sentence, suggests that the speaker wants you to understand that se refers to Marcum (4). A listener could still easily hear it the other way, though.
Here's an illustration of throwing in an ipse to overcome other pressures:
Quintus dixit cavum se fodisse. (Surely the speaker is emphasizing the fact that Quintus is talking about the hole: Quintus said of the hole that he had dug it.)
Quintus dixit cavum seipsum fodisse. (Well, maybe the speaker really does mean that Quintus said that the hole dug itself.)
I only made that up, but Allen & Greenough §300 (thanks, TKR!) says that using ipse, or even is, instead of the reflexive pronoun can resolve the ambiguity, with ipse having a stronger pull toward the main clause's subject (maybe contra what I just said):
[Caesar centuriones incusavit:] …Aut cur de sua virtute aut de ipsius diligentia desperarent?
[Caesar berated the centurions:] …Why should they despair either of their own courage or of Caesar's diligence?
So there's yet another pressure: separate use of both se and ipse suggests distinction, driving ipse to the main clause and se to the subordinate clause:
Quintus dixit Marcum se odisse atque ipsum dilexisse.
Of course, you could always drop the reflexive pronoun, though this sure sounds weird:
Quintus dixit Marcum Marcum odisse.