Should I justify an average GRE score with a good excuse?
I don’t think you need or should mention it.

First of all, 4/6 is not a bad score, even for a PhD application. Moreover, Writing is probably the least important section of the GRE test, at least for a physics major.

Second, the professors who review your application would expect to see other more meaningful indicators of your research potential, and thus, explaining why you don’t have a higher score will make you look petty. In fact, try not to talk about your Writing score because the reviewers might not care about it at all, unless you mention it.

After all, the rule of thumb is to focus on the strength of your application instead of explaining the weaknesses.