I think that's the point I made.
You said that the permit says no such thing. I disagree. The designs are incorporated as a part of the permit.
Boys, boys, let's not argue. I think you are arguing about semantics.
I know in my beck of the woods that you need permits (plural) depending on what you want to do, and where you want to do it, but before you can even start you need the approval of the engineering department and the planning board. Among others.
All we know is that OP doesn't like the location of the house on the lot. OP doesn't indicate anywhere that there it differs from the plans submitted in the permitting process.