Is there a negative stigma associated with choosing a university for non-academic reasons?
Assuming there is a negative stigma, why would you ever need to tell anyone why you picked the university you attended? Having a great passion for Japan and Japanese should be enough for anyone assuming you do an awesome project which you can and should do no matter where you go to school. This is especially true if you're likely to want to stay in Japan after your schooling to be an academic there, and it's a great reason to go to a Japanese university if you'd like to make a life there due to your passion.