A child is often cause for a family to rediscover their faith: rejoice that this could be your call back.
To be honest, it will depend on the priest and the parish, as well as how you come across. The priest might be concerned that you aren't taking it seriously: why didn't you speak to the priest about finding the person difficult, or seek a different Mass for Sunday, or another local parish? But he also might be unconcerned, and just happy that you want Catholicism for your child. Either way, be honest and treat this as an opportunity to come back.
Also, while it is ostensibly about the abuse scandals, I found this article helpful in my own struggles with the Church when I was finding it hard to attend my local parish where everyone knew me. Not that anyone was nasty to me. http://www.catholic.com/documents/a-crisis-of-saints
In the words of St Francis de Sales, "While those who give scandal are guilty of the spiritual equivalent of murder, those who take scandal-who allow scandals to destroy their faith-are guilty of spiritual suicide."