Well it sounds like you could work on your friendship with the father so that he doesn't feel so uncomfortable visiting his child. Could there be unnecessary tension in the room? Not good if there is because the child can sense this,. You need to do whatever you can do to make it a positive experience for the child. Even if it means being nice to the father for the short time he's around.