are withholding fees legal

What is the name of your state (only U.S. law)? Virginia
I signed a contract and paid $2500 up front to an attorney who now does not want to represent me. My understanding was that any money paid up front to an attorney was considered retainer money to either be pulled from deducting an hourly wage or as money that could be applied to an outstanding bill if the attorney withdraw as a representative of a case. Is charging an individual $2500 just to say he or she will represent a person ethical. This attorney then charged an hourly fee of $250 and hour plus $15 a month for legal assistant fees. I now owe an attorney who is not willing to proceed with my cast 2
$2,900 and he will not apply the $2500 I paid him up front to the bill. He said I understood this when I signed his contract. He also insists that no other attorney would only charge $250 an hour for work. I know from several phone calls to surrounding attorneys that this information is not true. Does this attorney have an ethical and legal right to charge people $2500 for his work.

I assume you read the contract thoroughly before you signed it. Does it mention anything about withholding fees?

If you feel the attorney has acted incorrectly, contact the State Bar.