Do any of you avoid caffeine while pumping? Or wait a certain amount of time to have it before you pump?

DS doesn't sleep much, and I work in a BUSY emergency room. I need my caffeine! I've just been limiting to a teeny cup of coffee right when I get to work so i have 3 hours before I pump. Do I need to be so cautious?