1. We don't remember everything that we experience.
In essence, memory can be thought of as a sequence of information-processing stages with three basic tasks:
1. get information in (encoding)
2. save information over time (storage)
3. get information out (retrieval).
Things that we experience go into our sensory memory and stay there for a few seconds. If we pay attention to them, they'll get lodged in our short-term memory – which can hold a few chunks of information for about 20 seconds. But if you don't then use or rehearse the information in your short term memory, you'll forget it.
That's why, if a day has no particular significance, or you have no reason to recall it in the near future, you won't necessarily remember the details of where you were or what you did.
If you do rehearse the information in your short-term memory – calling a phone number that someone just told you, for example – it can enter your long-term memory and you might be able to recall it much later.
2. Lots of factors can influence how someone remembers something.
3. Long-term memory can last a really, really long time.
"Neurons combine so that each one helps with many memories at a time, exponentially increasing the brain's memory storage capacity to something closer to around 2.5 petabytes. [...] You would have to leave the TV running continuously for more than 300 years to use up all that storage."
So if what happened years ago makes it into your long-term memory, it could potentially be there forever. But there's no guarantee that a memory you have is real...