There are three different sorts of nouns to worry about here!
The simplest, and most common, are gendered nouns. These have one form for the masculine, and a different form for the feminine; they're extremely common, since most adjectives work this way, and adjectives can be used freely as nouns. For example, a beautiful man is καλός, while a beautiful woman is καλή. And as Cosmas says, this is how μαθηματικός/η would work—it's also an adjective being used as a noun ("mathematical man"/"mathematical woman").
EDIT: I haven't found any mention of these in Greek grammarians, though it probably exists. But Priscian calls them mobilis, "movable", since you can move them from one gender to another by changing the ending. That's probably a better term than "gendered".
Next are the common nouns (or more properly, "nouns of common gender", nomina generis communis). These have only a single form, but swap their gender based on what they're referring to. This is how παῖς works: the form is always the same, but it takes ὁ if the child is male, and ἡ if the child is female. Cosmas suggests the additional example of φιλόσοφος, another adjective-used-as-noun, which doesn't change its form for gender (it's an "adjective of one termination").
Finally, there are epicene nouns. These also have only a single form, but they also have a fixed gender that can't be changed. An ἀλώπηξ, for example, can be a male or a female fox, but it always takes ἡ, never ὁ.
As for how to know—for most words, unfortunately, you just have to look it up. There's no way to know that παῖς is common and ἀλώπηξ is epicene without checking a dictionary. But any good dictionary will note this; the distinction between "common" and "epicene" goes back to Dionysius Thrax if not further, so it's not a new innovation.
