How can I tell if someone is trying to hack port 8333?
You are looking for ways to improve the client's protections against flaws? 

Here's a list of its current protections:


http://en.bitcoin.it/wiki/Weaknesses#Denial_of_Service_.28DoS.29_attacks