because it does not matter how you consume calories, whether via fat, carbs, sugars, protein, if you are not doing enough exercise to burn it off, it all gets processed into fat.
You are correct that body-builders do eat a large amount of calories, but these are dedicated people who do lots of hours in the gym each day. They don't sit around chatting like many gym users, they work there. The average member of the public spends at most 1 hour working out. With many people this is in fact only 30 minutes with 30 minutes chatting.
The majority of people do not wish to be huge, they want to be slim. To achieve this, you need a healthy balanced diet, consuming only the calories that your body needs and no more. There is also a vast difference between body-building and fitness. I know someone who is a man-mountain but he cannot run for toffee.