The new home is in my name only. We signed no agreement. But, he did pay me with a check. I guess my brother is a witness.....
You'll really need to see an attorney.
Worst case scenario is this: Husband denies that you had any deal for him to take the one house and him to buy you out. In that case, he would claim that the money he gave you was marital money and the downpayment was marital. Since the house is in your name only, the money from your brother might be considered a gift.
Now, your attorney will argue that the quit claim is evidence of a swap and you getting a check from your stbx and him refinancing is further evidence that a deal took place. Your brother will be a tainted witness, but given the quit claim and refinancing, that might be enough to convince the judge. No guarantees, but it certainly seems plausible.
Best scenario is that you have a relatively non-confrontational divorce, mediate all the issues and stbx doesn't try to cheat you on the housing issue. That is what I'd work towards so you won't have to worry about who would win in court.