the baby is your cousin. It depends on how you are related to your cousin as to what level this cousin is to you.
If you are first cousins with this cousin then her child is your first cousin once removed and your children and her children will be second cousins.
if you are second cousins then her child is your 2nd cousin once removed and your children and her children will be 3rd cousins
if you and she are first cousins once removed (meaning she is your mom's or dad's first cousin) then her children will be your 2nd cousins.
but if you are first cousins once removed because she is your 1st cousin's child then her children will be your first cousins twice removed.
it's just easier to say the kid is your cousin. all the other designations tell you is exactly how you are related and to what generation you belong within your family.
You're only the aunt of the baby if it was your sister or brother who had a baby. The baby is your second cousin because the person who had the baby is your first cousin. They'll always be your cousins.