I think the term has been getting more and more negative, so I no longer use it. What got me was seeing how many people were affected by all those South Park episodes about Cartman, starting with "Ginger Kids."

Redheads have long been discriminated against:
http://en.wikipedia.org/wiki/Red_hair#Prejudice_and_discrimination_against_redheads

Overall I am not sure if the term is considered a pejorative term or not though. What's your take on the term "ginger"?