Either that or he doesn't want to move into a relationship that fast as, you semi-work together.
He also might be unsure if you're interested or not and probably doesn't want to bore you with the same questions you may get asked by other guys.
There's a whole bunch of possibilities of what you and him could be thinking, don't limit yourself to just.
"Was he just unsure what to talk about because he doesn't know me that well? Or was he just annoyed?"