What is the grammatical difference between saying something like Bonam Fortunam (in the accusative) and Bona Fortuna (in the vocative) to another person? I have always heard the former, and I do not quite understand how it differs from the latter. Is the accusative used for addressing someone? If so, could you please explain why in "O mater, es pulcherrima" the mater is in the vocative...
The vocative is the case used for addressing someone. If you said to your friend Mike, "Hey, Mike, I think your sister is swell," "Mike" would be in the vocative case. Or if you found someone in your seat at a bar and said, "Hey, buddy, do you want to move?" "buddy" would be in the vocative case. That's why mater is in the vocative case in O mater, es pulcherrima: it's the equivalent of saying, "You know, mom, you're totally gorgeous." The easiest way to think of the vocative is to imagine it preceded by an imaginary "Hey." (This is easier in Greek, where the vocative is generally preceded by ὦ or O, which is more or less translated "Hey.")
The accusative, on the other hand, is used for the direct object of a verb. If you said, "I'm having a good day" or "I'm reading a boring book," "good day" and "boring book" would be in the accusative (diem bonam and librum tædiosum, respectively). Bonam fortunam ("good luck!") is actually an implied direct object, since it's essentially an abbreviation of Bonam fortunam tibi exopto ("I wish you good luck"). If you were to say Bona fortuna instead, you'd be saying, "Hey, good luck, [I think your sister is swell]"—it would indicate that you were directly addressing good luck. But you're actually addressing your friend Fred, and you're wishing him good luck, so the accusative is called for.
Note that there's another kind of accusative called the accusative of exclamation, that's used when you're making an exclamation that consists of a noun or noun phrase without a verb. If you wanted to say "What a man!" or "Oh, miserable me!" you'd use the accusative: Quem virum! or O me miserum! respectively. I don't actually think that bonam fortunam is an accusative of exclamation; for that to be going on, you'd need to mean, more or less, "Wow, what good luck!" which isn't really what's going on.