Culture clashes in authorship norms
I feel like the right answer must be to continue the conversation with the Japanese postdoc and ultimately give him the responsibility to understand and accept your position.

I understand that at first he interpreted your lack of desire for coauthorship in a certain way.  And then you explained your position, as you have written here.  How did he react to that?  If someone insists on interpreting your behavior in a way different from how you are explicitly identifying it as being meant to be interpreted, what can you really do other than ensure that your future actions are consistent with the interpretation of your present behavior that you have conveyed?

I don't see any insurmountable "cultural differences" here.  In my field (mathematics, which sounds like it is equal or close to yours) there are plenty of singly authored papers by Japanese people, so obviously the practice of supervisory coauthorship is not a fundamental tenet of Japanese culture or anything like that.  Even if it were: if the postdoc is in the US working in your group, then he has volunteered in a very strong sense to participate in your local brand of Western academic culture.  If he cannot understand that actions can have different meanings in your neck of the woods than what he was used to earlier in his career (whether by virtue of his being Japanese or for some other reason), then he's headed for big trouble.

I would say: clearly explain the situation as best you can, in several sittings if necessary.  Don't put your name on a paper that you don't feel you contributed to just because that makes the postdoc uncomfortable.  In the near future, try to go a little out of your way to show that you still want to do business with the postdoc (assuming you still do, which sounds reasonable).