You need to replace the cap and have it checked out by a mechanic.
The problem is oil reacts with oxygen when it gets hot, but the oil cap reduces the amount of oxygen available so it doesn't react as much.
The oil 'burns' (but without flames) which is why you get the smoke.
The burned oil doesn't lubricate the parts of the engine the way it should, so excessive wear could quickly occur.
Again, have a mechanic check it out, and you should probably also change the oil.
Zeba
http://www.JigsawADay.com