If you are asked a question that feels hostile, SLOW DOWN. Do NOT defend yourself, don't go on the counter-offensive. Inquire about the background, context, motivation, bigger picture behind the question.
Involve OTHERS in the room. This is basic group dynamics 101. Sometimes the hostile questioner is also perceived as hostile in the department her/himself. So refer the hostility outwards to the group. But make sure you're talking in the THIRD PERSON SINGULAR. “Would it be possible to get some clarification from others on this issue...?”
Don't let one person single you out and get into a dogfight. Then, at best, you lose the rest of the group. And other, more wrong, stuff can easily happen.
Once upon a time I got into a tussle like this over Deleuze and Guattari. It wasn't even about my work! Suffice it to say, I didn't get a fly back.
Remember the golden rule: they ALREADY think you're smart. You are there to show you can be a GOOD AND INTERESTING COLLEAGUE. For 20-40 minutes. Use them wisely!