You and your wife together could make the decision to move to another state with your children. There certainly would be nothing "criminal" involved with that, nor nothing civil either.
However you could not unilaterally move your children to another state without your wife's agreement or the blessing of the courts, without potential reprisals. If you move JUST your children to another state, to live with a non-parent, then you risk even more.
I wish you would explain a little more about what it is the grandparents are attempting to do. Its really difficult to give you good advice in such a vaccumn.