I realize now, that everyone else knows this, but I had never seen anyone consistently while they were growing up and it wasn't until I ran into a kid from my 2nd grade class in college, who looked exactly the same only taller and with facial hair, that it clicked. My only frame of reference for aging before that moment was movies and TV. Well, and my sister, but maybe it was just really good casting. Okay, fine, I have no explanation.
I have a solution though; I think this confusion could be avoided for other people if we just had actors keep a collection of clones of themselves at different ages, just in case their movie requires a scene that flashes back to childhood.