You are right. The employer should have to pay the taxes that they should have withheld. If you can not come to an agreement with them to do so, I suggest you file your income tax return and include a note to the IRS explaining the facts.
I do not know for sure, but the employer probably would have a right to repayment from you of the amount of taxes the employer would have to pay on your behalf. However, the employer would have no right to reimbursement of any interest or penalties involved.
Following are the rules pertinent to your situation:
Employers are required to have a Form W-4, Employee's Withholding Allowance Certificate, on file for each employee. They must all new employees to give you a signed Form W-4 when they start work. If a new employee does not give the employer a completed Form W-4, the employer must withhold tax as if the employee is single, with no withholding allowances.
Under IRC Sec. 3402, unless an exception applies, every employer making payment of wages must withhold taxes. Under Treasury Regulation Sec. 31.3403-1 , every employer required to deduct and withhold the tax under section 3402 from the wages of an employee is liable for the payment of such tax whether or not it is collected from the employee by the employer. If, for example, the employer deducts less than the correct amount of tax, or if he fails to deduct any part of the tax, he is nevertheless liable for the correct amount of the tax.