title: TCP connect error "No buffer space available"
body: <p>I got this error when I was testing a home-made distributed key-value store on very large scale (2048 machines), it works fine on 1024 machines. The communication pattern is n:n,which means each machine send lots of requests to every one else.
  I used C TCP socket to implement it, SO_REUSEADDR to recycle those sockets. 
  Can any one give me some tips about the possibilities that can cause this error? The code is complex, It's hard to simplify it and post here without losing necessary logic. So I just want to know all the possible causes to start to debug. </p>
  
  <p>BTW I changed settings as:
  echo 8000 > /proc/sys/kernel/threads-max</p>
  
  <p>echo 8000 > /proc/sys/net/core/somaxconn #limit of listen queue, default is 128</p>
  
  <p>echo 8000 > /proc/sys/net/core/netdev_max_backlog</p>
  
  <p>echo 10 > /proc/sys/net/ipv4/tcp_fin_timeout</p>
  
  <p>echo "10000   65000" > /proc/sys/net/ipv4/ip_local_port_range</p>
  
  <p>I did this is just for enable maximum number of connections.</p>
  
  <p>Thanks,
  -Tony</p>