Both sides hold in their minds the best example of the worst person, or that everyone they disagree with can become such an individual and allow the worst to happen. I think when people do that, they fail to see humans on the others and become what they fear.
The problem is that people only see black and white nowadays...but the truth is gray, people need to be open-minded for the different and also understand the other.