Yes, I gather that physicists don't communicate too well.
It sounds like you're playing games as much as he is. Like you're just gearing yourself up to cheat on your boyfriend with him and blame it on the guy's "manipulation" - i.e. creating a situation where you get to do something that you know is wrong, but get to externalize the blame so you don't feel guilty. As a psych major, what do you think of that theory?
I would advise you to make a decision one way of the other; either drop your boyfriend and try your chances with a guy who you well know is just in it for the chuckles, or make the decision that he's not worth losing your existing boyfriend for and tell him to hit the road. That way you get to decide for yourself what your priorities are and don't hurt anyone else in the process.
As Master Betty said, guys are eternal optimists and will always interpret any marginally positive signal as a "yeah, she wants me, she's just waiting for me to hit the right butons" - and that's definitely the message you seem to have sent to him. (the guy that is, not Master Betty.)
(Well, maybe Master Betty too - I dunno!)