Can they FORCE me to do a job I said I wouldn't?

Before I was hired, during an interview, I told the manager that I was unwilling to work in the Dairy, or any *cold* dept. because I cannot stand the cold, knowing this I was hired for *dry grocery* dept.
Lately they have been telling me i HAVE to work in the dairy dept. when they are shorthanded,(today they wanted me to work there for my entire 12 hour shift) and today after they once again forced me to go there for 3 hours I refused to go back after my break, I was still very cold and my fingers were numb. The assistant manager then told me to get a sweater off the store shelf, which is against store policy, so I refused. I told them I was not willing to go back and they told me to either go back or go home, I opted to go home and told them I would be calling the store manager in the AM because they are wrong to do what they did.
I feel they are wrong and I should not be forced to do ANY job I told them before my hire date that I was unwilling to do.
Who is right?
This is in the State of New Hampshire.

you get cold working in the dairy dept. on ocassion and you live in New Hampshire???? Why don't you get a job at a company that doesn't have a dairy dept.? If the company is shorthanded, a reliable employee would normally pitch in and help. I can understand why you would be sent home. There is good and bad in every job as it is in life in general.