Relative adoption

i live in new york, but in 1992 i lived in florida and after months of my mother hassling me, allowed my father to adopt my one year old son so that my father would receive government money for him and i could go back to school. Eventually, i believed i would no longer need the money that my father provided. (social security.) my son has always resided with me, i have put him in school, took him to the doctors, everything the "normal" parent does. my son has never actually resided with just my father, it was either myself and my father or just myself. i have been told that my father has "legal custody" and i have "physical custody". my problem now is over the years every time that my father gets upset with me he threatens to take my son away from me. he has never done it, but this time he is extremely upset. i now live with my fiance', whom my father despises and four of our children. we have just bought a house a year ago. what are my legal rights? the adoption papers did not state that i relinquished all parental rights - they said that i gave him the right to adopt my son. Can someone please help? thank you.