When I go abroad, on holiday, I don't expect anyone to speak English. If they do it's a bonus, if they don't then I always prepare in advance of the trip and I always find a way to communicate in their language and it's always appreciated.
If you see yourself as someone who is tolerant, then why can you not show more tolerance towards how uncomfortable some people feel, about walking through their town centre and hearing so many foreign accents ? But I can't afford to move to The Lake District and live next door to Tim Farron can I ?
The people of this country were never asked if they wanted Freedom of Movement forced on them, and for good reason as it would have been overwhelmingly rejected. Hence why the majority of the Electorate voted to Leave the EU.