Remember how Voldemort accidentally turned part of Harry into a Horcrux?
And how constantly being around the dark magic of a Horcrux makes you act like a mean-spirited, evil, and generally miserable person?
Well, that might be the reason behind all the Dursleys' terrible behavior: They were negatively affected in the presence of a Horcrux aka Harry Potter.
Here's the theory in full, according to ChristieIsBored (who also links to this fascinating Tumblr post):
In sum, to quote jodatoufin, "Since Harry was kind of a Horcrux that could explain why the Dursleys were so cruel to him."
Maybe it wasn't the Durselys' fault they were so awful after all.