If the baby is not born yet he has every right to be part of the child's life when it is.Any issue you have with him you need to resolve . You got to think about grandma ,grandpa, aunt, uncle, they have nothing to do with your problems.
You may need to go to family court or your local DSS?