I was reading an item from the Reserve Officers Association that I believe deals with a similar situation. The military member had custody of the child and due to being deployed arranged for the grandparents to take care of the child. ROA said: "The court gave you custody of the boy, but the court did not and cannot delegate to you the authority to determine an alternative custodial arrangement if for any reason you will not be able to exercise custody of the child for an extended period. You simply do not have the legal power or right to turn the child over to your parents upon your deployment." It sounds like this is exactly what the father did in your case.
https://cdn.ymaws.com/www.roa.org/resource/resmgr/LawReviews/2009/09051-LR.pdf
https://cdn.ymaws.com/www.roa.org/resource/resmgr/LawReviews/2009/09051-LR.pdf