It was always your son's money. Any "agreement" is not binding as there wasn't consideration in the first place and the purported agreement to hold money in this way would probably have to be in writing to be enforceable anyway.
But, there are a few problems that can be had for son. One, is the account format. If the bank allows the son to remove the money on his own, you can't really prevent him from doing so. But, if the bank requires both of your signatures, I don't think there would be a penalty for not signing--even if your signature was withheld unreasonably. Litigation over exactly what one had to do in such a situation would be pretty difficult and expensive. (Making it unlikely son could afford it.) But, at the end, he would probably get the money.
The second is the joint titling. Depending on the exact facts and purported agreements, there could be some compensation issues related to management of son's funds. Not only could that shift some actual ownership of some funds, but it could arguably be the thing you give up for consideration to make an agreement regarding the funds. Again, making litigation expensive and difficult thus reducing the likelihood of it being a realistic option. (Depending on the amount we're talking about.)