How do you know when to be concerned about bad headaches during pregnancy and preeclampsia?

I had bad headaches during my last pregnancy, and had them even worse with this one. I mentioned t to my midwife, but basically same advice as always, drink more water. They've been better for a few weeks but a new one started last night, woke up at 5 with a headache so severe I wondered if I should go to the ER- but two Tylenol and another hour of sleep did the trick so I probably was overreacting. Exactly 4 hours later it came back with such force it brought me to tears, but eating and drinking and a few more Tylenol brought it back to a just barely tolerable level.

I don't actually think this is a real problem, other than being really uncomfortable, but how do you know when to actually be concerned? I'll ask at my next midwife appt too.

(And I just had an ultrasound this week and everything looked perfect and baby has been moving around plenty this morning).