Here's the maths:
You have two biological parents, four grandparents, eight great-grandparents, and so on. The number doubles every generation – after six generations, for example, you have 2⁶, or 64, ancestors. This number grows so quickly that it hits roughly a billion ancestors by the time you get 30 generations down the line. But world population didn't reach a billion until the 1800s, and that is far fewer than 30 generations back in time.
Essentially, as you go back in time, you rapidly reach a point at which everyone who was alive at a given point is the ancestor of either everybody (if they had children, and their children had children, etc) or nobody (if their line died out).