The executor has a fiduciary duty to the estate beneficiaries. Among other things, that includes the duty to file and apy tax returns in a timely manner. Unless the executor had reasonable cause to incur the (I assume) interest and penalties, he/she can be held personally liable for this.
As far as the fees, I'm guessing that he has already reported them on the estate's 706 and will have to file an amended return with increased taxes if he cannot collect those fees. However, in any case, the fee must be reasonable. If this is in the context of a probate estate, you have a right to object to those fees in court and have a hearing. While not necessarily required, actual time spent is an important factor. You may ask him to justify his fees by provided you with an itemization of actual time spent by him.
And, when he says you will lose a great deal in taxes, my question is will that outweigh the additional money avaiable for distribution due to the decrease in executor's fees. My guess is only the executor loses out.