No, this does not mean that you are not good enough for him. A lot of people, both men and women share this fantasy. The swinging lifestyle isn't for everyone, but for those who are secure in their relationship, it can add a level of passion and excitement that many find very rewarding. Having said this, if you are not okay with the idea, and your boyfriend continues to push the issue, I suggests that he doesn't respect your feelings about the matter. If that is the case, then I would say it's time for you to consider ending the relationship and finding someone who will treat you better. Remember, for a relationship to succeed, it must include mutual love, respect and trust. If your relationship with your boyfriend doesn't involve all three of these things it will never last. Best of luck to you and your boyfriend.
I hope i helped.