Hello – I’m not familiar with swollen glands being specifically related to Graves’, but honestly, if it was me, I would make an appointment to get this checked out ASAP.
Obviously, there is *something* going on, so you are definitely not a hypochondriac!
And I would really encourage you to do whatever it takes to get in a place where you can be consistent with managing these health issues. Whether that is finding a different doctor that you really like, setting timers/reminders for medications, or something else that will suit you. Being inconsistent with your treatment is not likely to be helpful, and it could one day make your health issues even more difficult to deal with.
Take care!