I agree wholeheartedly. The longer you let the bad apple roll around in your barrel, the worse conditions get in your barrel. If you rescind the job offer or let the person go very quickly after the hire date, you have lost very little, you have paid out very little in wages, and your connection to this person is very short. People who are dishonest are usually systemically dishonest. They'll fake expenses, they'll steal paper from the copy machine, they'll shoplift grapes at the grocery store
Three years down the road, you're pressing charges, trying to get your company finances back in shape, have a list of complaints against this person that is thick as the Old Testament, and you're so frustrated because you've terminated the person for so many reasons that there's no one clear cut one, and they get approved to draw unemployment from your account.
What kind of references did this person have? Can you do a background check of some kind? Caveat employerium