Your doctor is correct about the transmission of HSV-1 from your mouth to your genital area. If you already have it, it's not likely you'll become reinfected in another area.
However, another possible
scenario is that your partner may have given you HSV-2, even by the minimal contact you decribed. Very often cultures come back negative because the sore has healed to the point where it's difficult to get a good specimen.
How are you doing with this today? Is it starting to subside? There are a few options to get closer to a good diagnosis: You can get a good HSV antibody test if you wait 12 weeks from the time this happened.
Also, if this doesn't go away soon, you may also want to ask your doctor about the possibility of a yeast infection, BV or even HPV.
One last thing, could this also be that you've gotten yourself so worried that you're looking at the area quite a bit and overthinking this? We all tend to do that from time to time.
I hope this helps!