How to get rid of all the .serverauth.$$ in the HOME directory on OS X?
You can prevent these from accumulating if you always exit the X11 server gracefully.  That means logging out gracefully and shutting down gracefully.  No way you can always do that though, so about all you can do is delete these every now and then.  You could write a script to find and remove the ones not in use and run it manually or schedule it to run routinely via cron.  Below is an example script:

#!/bin/bash
cd || {
  echo "Unable to cd to home." &gt;&amp;2
  exit 1
}
Candidates=$(echo .serverauth.*)
if [ -z "$Candidates" ];then
  exit
fi
for Candidate in $Candidates ; do
  PID=${Candidate#.serverauth.}
  kill -0 $Candidate 2&gt;&amp;-
  if [ $? != 0 ];then # The PID is not currently active
    printf "Removing orphaned xserverauth file \"$Candidate\".\n";
    rm $Candidate || {
      printf "Failed to remove file \"$Candidate\".\n" &gt;&amp;2.
    }
  fi
done