The Labour party leader made the comments in an interview with Sky News.
He was asked: "What goes through your mind when you see a white van outside a house with an England flag?"
The Labour leader has been at the centre of a political firestorm after the former shadow attorney general Emily Thornberry was accused of being a "snob" when she posted an image of a house in Rochester with white flags on the windows and a white van in the driveway.
She resigned on Thursday evening over a tweet after fellow Labour MPs had accused her of being "derogatory" and "out of touch".
Thornberry was in the area campaigning for the party as residents voted in the Rochester and Strood by-election, triggered when former Tory MP Mark Reckless defected to UKIP.
Miliband also said that he was "angry because I thought her tweets gave a misleading impression [of the party] when she photographed the house in which the family lived".