title: golang get massive read tcp ip:port i/o timeout in ubuntu 14.04 LTS
body: <p>I wrote a golang program which run well in the past several months in ubuntu 12.04 LTS until I upgraded it to 14.04 LTS</p>
  
  <p>My program is focused on sending HTTP requests which send about 2-10 HTTP requests per second. The HTTP request address vary.</p>
  
  <p>When the problem occurs, first, some of the requests shows <code>read tcp [ip]:[port]: i/o timeout</code>, then after several minutes all requests show <code>read tcp [ip]:[port]: i/o timeout</code>, not any request can be sent.</p>
  
  <p><strong>I restart the program, everything become right again.</strong></p>
  
  <p><strong>All of our servers(2 server) have such problem after upgraded from 12.04 to 14.04</strong></p>
  
  <p><strong>I create new goroutine for every request</strong></p>
  
  <p><strong>the problem does not occur in the same interval, sometimes it won't occur one or two day, sometimes It occur twice in an hour</strong></p>
  
  <p>Bellow is my code requesting HTTP Address:</p>
  
  <pre><code>t := &amp;http.Transport{
      Dial:            timeoutDial(data.Timeout),
      TLSClientConfig: &amp;tls.Config{InsecureSkipVerify: true},
  }
  //req := s.ParseReq(data)
  req := data.convert2Request()
  if req == nil {
      return
  }
  
  var resp *http.Response
  if data.Redirect {
      c := &amp;http.Client{
          Transport: t,
      }
      resp, err = c.Do(req)
  } else {
      resp, err = t.RoundTrip(req)
  }
  
  data.updateTry()
  
  r := s.ParseResp(data, resp, err)
  </code></pre>
  
  <p>updateTry:</p>
  
  <pre><code>func (d *SendData) updateTry() {
      d.Try++
      d.LastSend = time.Now()
  }
  </code></pre>
  
  <p>timeoutDial:</p>
  
  <pre><code>func timeoutDial(timeout int) func(netw, addr string) (net.Conn, error) {
      if timeout &lt;= 0 {
          timeout = 10
      }
      return func(netw, addr string) (net.Conn, error) {
          deadline := time.Now().Add(time.Duration(timeout) * time.Second)
          c, err := net.DialTimeout(netw, addr, time.Second*time.Duration(timeout+5))
          if err != nil {
              return nil, err
          }
          c.SetDeadline(deadline)
          return c, nil
      }
  }
  </code></pre>
  
  <p>and My dealing with response is:</p>
  
  <pre><code>func (s *Sender) ParseResp(data SendData, resp *http.Response, err error) (r Resp) {
      r = Resp{URL: data.URL}
      if err != nil {
          r.Err = err.Error()
      } else {
          r.HttpCode = resp.StatusCode
          r.Header = resp.Header
          r.URL = resp.Request.URL.String()
          defer resp.Body.Close()
          // we just read part of response and log it.
          reader := bufio.NewReader(resp.Body)
          buf := make([]byte, bytes.MinRead) // 512 byte
          for len(r.Body) &lt; 1024 {           // max 1k
              var n int
              if n, _ = reader.Read(buf); n == 0 {
                  break
              }
              r.Body += string(buf[:n])
          }
      }
      return
  }
  </code></pre>
  
  <p>I also found setting in /etc/sysctl.conf which can make the problem happen less frequently:</p>
  
  <pre><code>net.core.somaxconn = 65535
  net.netfilter.nf_conntrack_max = 655350
  net.netfilter.nf_conntrack_tcp_timeout_established = 1200
  </code></pre>
  
  <p>I need help for solving this problem.</p>
  
  <p>It seems like this but I don't see any solution <a href="https://bugs.launchpad.net/juju-core/+bug/1307434" rel="nofollow">https://bugs.launchpad.net/juju-core/+bug/1307434</a></p>