On one hand there there are the fears of over population, cultural violence (on both sides) and conflicting religious ideals. On the other you have notions of peace, open borders and multi-culturalism.
I personally would prefer to live in an open world, fearing "terrorism" than in a closed world fearing my government.
Also there are quite large differences between the many different Muslim populated countries, expecting or assuming all Muslims act the same and want the same thing is like thinking that all Christians worship at the Westboro baptist church.