I don't believe it's that bad. Somehow people survived for centuries, despite being around smokers. In the founding days of the USA, people planted tobacco in cracks in their sidewalks because it was so popular. And just a few decades ago, smoking was allowed indoors in public, in restaurants, even in college classes. People who didn't like the smoke would leave the room or open a window, or ask the smoker to go somewhere else...no big deal.
I think people just always want to have someone to blame....and smokers are an easy target.