I think that's presenting a worst case, unlikely scenario (and OP's attorney seems to agree with me).
Realistically, if OP agrees to something and the judge orders it, that is every bit as solid as if OP doesn't agree and the judge orders the same thing. Grandma has no grounds to ask for more in either case.
The appeal argument has some validity, but let's say that OP agrees to supervised visitation of 2 hours every other week. If the judge orders that, why would OP want to appeal it? If the judge orders something MORE than what OP agreed to, then an appeal is still possible. So thinking about it from the basis of appeals doesn't really change my view.
My argument is that judges deal with intransigent people every day - people who are completely unreasonable and refuse to budge an inch. If Grandma comes in all sweet and nice and play the relationship card and then says "I'd be willing to settle for any chance to see my sweet little grandkids, but OP won't even discuss it with me and won't let me even talk to them on the phone", it could look bad for OP. Possibly bad enough that the judge would award grandma more than OP would have agreed to. OTOH, if OP says "your honor, I really don't think that it's a good environment for the kids due to the drug use and convicted criminals there as well as grandma's previous violation of restraining orders (I think that was in there somewhere). However, in order to facilitate the relationship of the kids with their grandmother, I could agree to a max of 2 hours of visitation every other week with a supervisor of my choosing and expenses paid by grandma", it sends an entirely different message.
I'm not suggesting that OP agree to anything that he's truly uncomfortable with. But if he can find SOME amount of relationship between kids and grandma that he's comfortable with, I believe it shows the judge that he's placing the kids' needs above his own - and that has to have a positive effect.
Still, it's a style difference. I won't say that I'm right and you're wrong - because there is no right or wrong. But I think that my approach is every bit as reasonable as yours.