Perhaps you have a friend or relative who can accompany your son.
You're really looking at this the wrong way. If your son tends to wander on school field trips, then it's for his own safety that someone accompany him. Would you prefer that the teacher take her chances that he won't disappear in a crowd when he wanders away from the group?
As far as the frazzled teacher is concerned, she may have a similarly less-than-kind opinion of you.