I think it's likely that society has us pairing it with something negative and with a negative tone. You hear a bit of disgust in the tone of some people when they say the word lesbian...a hesitant sort of speech around that word...
It does sound a bit harsh though with the "b" in the middle of the word. It's so abrupt... (took a poetry course, which is why I can now look at each letter in the word)