Is *gaijin* a term that's only acceptable when it's a non-Japanese person using it self-referentially (similar to [n-word privileges](http://tvtropes.org/pmwiki/pmwiki.php/Main/NWordPrivileges) in TV Tropes), or is it considered ok to use in normal conversation between non-Japanese and Japanese?

For example, could one say, "Can you rewrite that address in romaji so us gaijin can understand it?", or would it be better to use a more bland term?