Good job on figuring it out on your own. A Doctors guidance is not going to hurt... especially since you want to change your diet.
My doctor has a food challenge he told me to try when testing foods that have been removed. Its a 4 day cycle, in the morning of the first day eat a small amount the food then wait for a hour and eat a some more. Then if you notice anything as you go about your day take notes. Three days later try again.
It might be a good way for you to find out on your own if there's a real allergen.
He's name is Dr. Philip Ranheim in Lake Stevens and I think is worth the drive.
Stay strong and good luck.