What would be the way to say that you don't care about people you don't know? Something like "I don't care about those I don't know" or "I ignore the faceless masses."
Cicero has a fun, "pile-on" quote in Pro Fonteio 32:
potestis igitur ignotos notis, iniquos aequis, alienigenas domesticis, cupidos moderatis, mercennarios gratuitis, impios religiosis, inimicissimos huic imperio ac nomini bonis ac fidelibus et sociis et civibus anteferre?
Using him as a model, you could say:
notos ignotis antefero.
I prefer people I know to people I don't know.
It puts a slightly more positive spin on your phrase than you may want, but it has a nice ring to it!
There are many words for "unknown", but many of them have tones you might want to avoid. For example, ignobilis implies low birth and alienus implies being foreign or perhaps unwelcome, although both could be used for unknown. The most neutral word I could think of is ignotus. The masculine plural is a good choice for unknown people in general.
The verb contemnere can be translated as "to consider a person or thing as unimportant or of small value", and I think that's a pretty good fit.
Combining these, I would suggest this phrase:
I consider unknown [people] to be unimportant.
The English translation is a bit clumsy, but attempts to give a good idea of the meaning. A more fluent translation might be "I ignore the unknown", but I shall not endeavor a better English phrase here since the question was about Latin.
Catullus XCII is the contemptuous and snappy put-down of Julius Caesar, nil nimium studeo, Caesar, tibi velle placere, nec scire utrum sis albus an ater homo, or 'I've not the least wish to please you, Caesar, or even to know whether you are a white man or black'. Leaving out the personal stuff, you might reduce this to
Ignotis nil nimium studeo,
— or 'I don't pay too much attention to those I don't know'.