Shelton received some criticism online for the lyrics, which compare a woman's love to riches, with many people arguing it was inconsiderate for a multimillionaire to release a song about being on minimum wage during a year of record unemployment in the US.
Plenty of people called the song "tone deaf," and said it made Shelton seem clueless to the current state of the world.
But Shelton has now responded directly to the backlash, saying that if people find it offensive, they just have to "agree to disagree."
In an interview with CMT on Thursday, Shelton said he "related" to the song because of his own struggles making ends meet in the early days of his career.
The Voice coach went on to say that people who were criticising the song "clearly hadn't heard [it] or read the lyrics," and it was likely "four or five people that probably don't know anything about country music."
"I just feel like these days, there are people out there who don't want to know the truth," he said. "They just want to hear what they want to hear, and they want to pick a fight."
When asked why it took so long for him to address the criticism, Shelton said it's because it was "absolutely ridiculous" and "didn't even deserve a response."