Just to kind of sum up what everyone else said, if you're asking if the bio father can ever go to court to get a visititation or custody order, then no, he can't.
However, if the bio mother says it's ok for the bio father to visit with the childern, then yeah, that's fine. It's no different than if she were to let other family members and friends visit with them but that's about the extent of it. He's playing by the mother's rules now so whatever she allows is what he can do and that's it.
"Say the Bio dad signed his rights away to a stepparent adoption but the bio mother still wanted the children to know their bio dad and allowed the bio family visitation rights. Is there anything that can be done on the bio dads part to get back his rights?"
You did mention the bio family's rights too, so I don't know if you're asking if they have rights or not. If you are, that would include extended family such as grandparents. Evidently in GA, bio grandparents can file for visitation rights but there is no guarantee they would get it and they would only be able to file under what GA says they can file under and certain things are taken into consideration like if it's in the child's best interest, previous relationship, etc.