Most Remain voters don't view immigration as a bogeyman, it creates challenges but the country should be able to adapt and reap the benefits (as the story is far from exclusively negative).
So there is no equation to balance for Remain voters, we were, by and large, ok with the levels of immigration especially when weighed up against the benefits of being in the EU, including being in the single market. We were happy enough, within reason, with how things were from that point of view - the issues are fundamentally around domestic government policies over many, many years (and under both Tory and Labour governments). It's not a question that I for one can answer without indulging in hypotheticals where I imagine I believe in an argument that I actually disagree with, which doesn't seem especially worthwhile.