Very probably. Their 'allowing' you to advance your vacation was in essence a loan of hours (money). As such, they could claim that this was a loan.
Some random thoughts:
1) You have to consider that this company (that you are considering 'stiffing' for the 'loan') will very probably be asked for a reference from any prospective employer. Do you really want to risk their negative comments?
2) It would be normal for them to deduct this 'vacation advance' from your final pay. Have they given you your final pay yet??
3) It is likely that the employer has more 'legal assets' (money) to spend in pursuing this, than you have in defense assets.
My suggestion, you 'borrowed' the time/money, now you need to pay them back.