In order to have a common law marriage, you must hold yourselves out to be husband and wife (in addition to whatever other criteria the state may have). You already said he's your ex BOYFRIEND and you have not filed for divorce. Simply living together does not make you common law married. You are like any other unrelated pair of people owning property together.
Now, you said both names are on the mortgage. Are both names on the DEED? If so, then he can't refinance or take your name off the deed without your permission.
Now, you said both names are on the mortgage. Are both names on the DEED? If so, then he can't refinance or take your name off the deed without your permission.