You'll need a land use attorney. The "address" may not mean anything. I've had problems with two properties with similar issues.
One was a rather irregular shaped lot (roughly L shaped) and while I had 100' of frontage on the road that my address was on, the house sat turned sideways.at the bend in the L. (about 200' back from the road). I have in my files a determination from the county as to what is the "front" of my house as it pertains to setbacks. I learned more about variance plats, surveys, etc...just trying to turn my deck which was falling down into a screen porch.
The other was building my current house on what (I thought) was also a corner lot. A local ordinance says that the front door has to be parallel to the street (note that this could mean that your front door faces 180 degrees away from the street which for other reasons many houses here do). I suppose this was to keep people from shoving trailers into lots end-on. It also presumably avoids the above problem of houses that are set at odd angles to the street. I added a vestibule to the front of my house (actually, quite architecturally fitting) that turned my front door 90 degrees. Two years later when we were finalling the construction they said "where's the front door" and I point out my relocated door. "It faces the garage," they say. "It only needs to be parallel to the street according to the ordinance and you approved this design." "No problem, we'll call this other door your front door for ordinance purposes." "This door that you have to walk through the back yard, up a flight of steps, into a screen porch, out of a screen porch, across another deck, and then enter, is the front door? OK whatever floats your boat." A year later, they signed off on a neighbors house whose front door indeed faces (via a covered breezeway) his garage.