I don't really. I am not afraid to cook things in butter (like pancakes or eggs, yum!) and eat whole milk yogurt. When I get a latte, I get it with whole milk. I also LOVE refried beans.
I try to stay away from processed foods though, which usually have lots of fat in them.
I am most conscious of my carb intake and stay away from refined flour, and basically anything "white."