My DH says this almost every day.
I think it saddens him a LOT that our LO won't remember these "good times" we're having right now.

It doesn't bother me all that much. Especially since I'll remember and cherish these precious moments. I also have about 500 youtube videos of her thus far (she's 9 months old today). So she can "remember" through the videos.

Are you sad that your LO won't really start remembering until s/he's 3-4 years old?