When you quarter something, you divide it into four, so that you have four separate parts. This notion of separation is what developed into the sense of enclosure or separation or removal. That's why most languages have a word like quarter, quartier, kwartier, Viertel meaning "a fourth part, a separate part of the city, a quartered-off neighbourhood".
That a fourth part should have developed this sense, and not a fifth or a sixth party, is perhaps arbitrary. But perhaps it was common to divide cities into northern, eastern, southern, and western parts? We also do that in mathematics with quadrants.