What do you mean
"if we let this go on once..."?
You have no say in the matter if the Board and/or the Association voted and passed the special assessment.
It is up to you to attend all condo HOA meetings and learn what is happening. Why is the HOA fees being increased and why is there a special assessment. What is the assessment for ie. attorneys fees for construction defect litigation, repairs to common areas, reserves, management fee increase, resident manager raise etc.