I think you should probably not say you "refused work." Your part time job is not what the unemployment certification is interested in anyhow.
You were out of town for how long? A whole week? Two or three days? That is not "refusing work." This may, if you missed a whole week and were out of town and were in no way looking for jobs at the location where you were shopping grad schools, mean that you were not "able and available" for work. That is where you would report that if you did. The most honest thing to do when you are not available is stop the claim for a week. Then you re-open it in order to begin drawing benefits again. You do not lose the week's benefits, they are stuck on at the end of the claim.
Refusing work is when you are formally offered a job, and you refuse it for whatever reason. You would be supposed to report this on your certification for that week as yes, I refused work. Then they will stop your claim, call you and obtain the particulars of the situation. They would, if the job were determined to be unsuitable due to the commute, the pay, the hours, whatever, issue a "refusal of work" decision and you would, if they decided it was not unreasonable for you to refuse the job, back pay you and begin paying benefits to you again.
The little interchange you discussed with the employer where you and he talked, and he said you'd be working thus and thus and thus, and then you said you didn't want to start until two weeks later, and then he rescinded the statement....let me put it this way. DON'T APPLY FOR JOBS YOU DON'T WANT!
Since the employer rescinded the job offer, I think you are probably out of the woods as far as it being reported to the unemployment office that you refused the job. I don't feel that you have to report it, under these circumstances. Though you walked a fine line on making yourself sound so unattractive to the employer that he decided to take back the job offer.
My reasoning on this is that the job sounds as though it was spectacularly unsuitable for you, it was very probably not an equivalent job to the one you are looking for, the one you lost, and you are already working at a part-time job to supplement your unemployment benefits anyhow. If you report that you refused a job, they will stop your claim, you will go through weeks of paperwork, and eventually, they will issue you a decision approving you to continue to receive benefits. Sort of an exercise in futility, since the job offer was rescinded and very likely will never be reported to the unemployment office.
I am sure the workers at unemployment office in your state would tell you the same thing I am about to tell you, if they're being very frank and not being sticklers for the law. Okay. DO NOT APPLY FOR JOBS YOU DO NOT WANT! You are not required to apply to obviously unsuitable jobs. When you are in these places looking for graduate schools, apply for a job. By the way, did you also apply, when submitting your application to the school, for a graduate assistantship or work study program? If so, you do not say you were not able and available for work while out of town looking at a grad school, you were seeking work out of town if you decided to move there.
About your part time job. The employer seems to have been cooperating with you, and that is good. We certainly assume you are reporting your wages each week to unemployment, and working your hours he has available for you as you two work out between you. If you were to up and quit this job, you would be supposed to report it to the unemployment system that week, and yes, it would probably stop your benefits all together if you quit the job without a good cause. If you were quitting the job to go to grad school, you would not be able to draw while in school anyhow, under most circumstances.
In any case, that's the time to tell the unemployment system what is going on.
Many people on unemployment, I'll have to say, IMHO try to be terribly honest. In fact, they over-think and agonize and report too much. This increases the work load of the unemployment system considerably. That's why I say don't feel you have to apply for every job you can think of, especially one you know is unsuitable to begin with. If you are formally offered the job, you can be nailed for refusing it. You will be able to figure out whether or not it was a bona fide job offer.
But as I said, it is not dishonest to look for a job out of town. It is not dishonest to work your part time hours around your desire to travel somewhere for a couple of days while on unemployment. A two week vacation in Europe isn't what I mean here, because you know you're not looking for work there, and you're not able and available. You can in such cases stop your claim by not certifying for those weeks, and then will have to re-open it when you are again ready to qualify.