“I think it came to a point where it was derogatory toward people who wore their hair natural,” TK Wonder said