Will one nvarchar column affect query performance?
Without knowing more details, it's impossible to tell, but you might be suffering from implicit conversion.  There are a number of sites with information on the topic that you can use to help troubleshoot your particular query.

The short answer is that no, you should not need to convert all columns to nvarchar.  If you are trying to join an nvarchar column to a varchar column, however, you could see a major performance impediment.