Restart Screen Sharing Service on Mac mini Server
I had the same trouble, and to a headless Mini also. This is what I found that worked…

sudo launchctl unload /System/Library/LaunchDaemons/com.apple.screensharing.plist  
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.screensharing.plist


I had the opportunity to install an SSD in my Mini, and did a clean install of Lion at that time, and the issue persisted.

I made an alias

alias fixscreen='sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.screensharing.plist &amp;&amp;  sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.screensharing.plist'