Yes. If your actions violated the terms of the trust or breached your fiduciary duties to the beneficiary,you could be held liable whether or not it caused him damages. But, see an attorney. I don't really have any idea what your relationship really is.
But your fiduciary responsibilities only apply to the trust. SSA has no authority over the trust. So SSA could not hold you liable for anything.
Several thousand ($5000) is not really a life altering amount of money. It is enough for a homeless person or a young person starting out to set up a home and furnishings and maybe buy a beat up old car that could run a few more years. It also could give someone money to have a good time for a few months while making foolish decisions. $50,000 could pay off debt, buy a new car and a really nice TV, or be enough to survive on for several years. Now $500,000 could be life altering money and could provide an income stream that may be enough to live on without invading the principal.
I have never totally understood the desire to stay living at poverty level (on SSI) and having money in a special needs trust that doesn't get spent on the disabled person. I did see a severely mentally disabled SSI recipient with a special needs trust. Daughter was payee and trustee. Daughter complained that SSI was not enough money for him to live on. Daughter also had not spent any of the special needs trust money. Daughter was also the beneficiary of the trust so that whatever money she didn't spend on her father, she would get to spend on herself when he died.
But if you breach your fiduciary responsibility by doing what the beneficiary wants and not follow the terms of the trust, what purpose are you serving? Why even be a trustee? You are the one who agreed to follow the trust documents, so figure out a way to do so or resign.
