That cost reflects the difference between factory workers who make ~$12 an hour and sweatshop employees who make dollars a day, and in unsafe, poorly regulated conditions for unethically long hours.
The less depressing reason to buy American-made clothes is simple: supporting American manufacturing jobs. Even a small purchase, buying an American made T-shirt, supports Americans workers.
An added bonus of buying U.S. made? If you have a problem with a product, you can usually find the company and actually get in touch with someone who can help you.