When we speak to people from another country and culture, it is probably best to avoid using any kinds of "idioms". I have said things like "Will you quit it?" to my helper and she replied "Are you asking me to resign?" "No I just want you to stop what you are doing now?" "Quit it" means "Stop doing that", not "please resign". 

Sometimes a helper is suffering from homesickness and the employer makes things much worse by telling the helper that she has a "black face" or that they don't like looking at her "black face". The Chinese employer is referring to a "stormy demeanor, a sad face", but to the helper's ears this is a "racial slur", an insult. It's like calling a black person a "N*****". 

How is using a racial slur going to help someone overcome homesickness?? Of course, after hearing those words, they are going to find the work environment difficult, because they misunderstand and think their employer is a racist. 

When people respond to your words in a strange way, it is a good idea, to think about what you said, and how your words might have been misunderstood. Clear up the misunderstandings as soon as you can. With people from two different cultures, both using English as a 2nd language to communicate with each other, there are bound to be many misunderstandings, and hurt feelings. We need to patient and try to hear our words and see our action from the perspective of the other party.