I would say that the difference is much like the one between the English "in" and "at" (for being somewhere) or "into" and "to" (for going somewhere).
When you are in the store, you are actually inside it.
When you are at the store, your are probably somewhere near but not quite inside.
For movement, ad + accusative means going near and in + accusative means going in.
For location, ad + accusative means being near and in + ablative means being inside.
It's not a question of which place requires which pronoun, but of where you are in relation to the place.
If I plan to meet a friend in front of their house, I could say ad domum tuam venio.
If I actually planned to go inside, I could say in domum tuam venio.
(Adding an adjective or a possessive pronoun often turns locatives and other non-prepositional expressions for domus and cities into normal prepositional phrases.)
Both in ludum and ad ludum make sense with venire but mean different things.
Are you going to the schoolyard or all the way into the building?
This is of course a simplified picture, but should give you a good overall idea of the difference.
One cannot cover everything quite so briefly.
I will also point out that I took the point of view of buildings, not so much places like market squares and in particular not people.