What it means in America: Trousers.
What it means in Britain: KNICKERS. Knickers. Seriously, you Americans don't understand how much pants just aren't trousers to most British people (I think there might be some weirdos in the North that call trousers pants, but forget those people).
When you talk about wearing your favourite pants to work, we imagine you in the office in your underwear. For this reason, it is very hard for us to be scared of you.