If I understand your situation correctly, then I see no ethical dilemma. You are employed by a consulting company. The company does a job for client X. Subsequently client X asks you to come work for them. Your question asked if this was "ethical" for you to work for client X. This of course is an ethical, not legal question. Thus, more than ever, that is for you to decide since so many people have divergent opinions as to what is ethical. Moreover, the majority of Americans have chosen to blur the distinction between mores and morals.
------------------
Mark B. Replogle
