Do you work with others who have children especially babies/young children?

I'm the only one in my group that has a young child (my boss has tween/teen kids), and sometimes I find it a little hard, especially if I have to decline certain work events or travel. If you don't work with others who have children, does it ever affect work relations?