Again, it depends.
I've hinted at it before, but without knowing what confidential information he is using against you that he gleaned during his previous representation, there's really no way for us to know if it's a conflict of interest.
For example: If you gave him your address, and he know is using your address to serve you documents, that is not a conflict of interest. However, if you told him that you like to sleep in pink nighties and play with barbies, then using that previously unknown information could be seen as a conflict.