title: C++ Socket Programming: Network Low Speed Analysis
body: <p>I have bought a VPS as a remote server and deployed shadowsocks as a proxy on this remote server.</p>
  <p><strong>Case 1:</strong></p>
  <p>When I use uTorrent to download files through shadowsocks, the speed could reach 10 MB/sec (which is 80 Mbits/sec). The diagram is:</p>
  <pre><code> ---------------------------                    --------------------------      ---------
  |    Local computer         |                  |   Remote server          |    |         |
  |              ------------ |   80 Mbits/sec   |  -----------             |    |         |
  | uTorrent&lt;--&gt;| shadowsocks||&lt;----------------&gt;| |shadowsocks|&lt;---------------&gt;|Internet |
  |             |    client  ||                  | |  server   |            |    |         |
  |              ------------ |                  |  -----------             |    |         |
   ---------------------------                    --------------------------      ---------
  </code></pre>
  <p><strong>Case 2:</strong></p>
  <p>Now, I wrote a client/server program by C++. But the network speed is extremely low, and here is my test results:</p>
  <blockquote>
  <p>Client: download speed</p>
  <p>t_lRecvTotalBytes = 1024000 Bytes</p>
  <p>t_lRecvTotalTime = 2942410 microseconds.</p>
  <p>network speed: 2.7 MBit/s</p>
  <hr />
  <p>Server: upload speed</p>
  <p>t_lRecvTotalBytes = 1024000 Bytes</p>
  <p>t_lRecvTotalTime = 1741625 microseconds.</p>
  <p>network speed: 4.5 MBit/s</p>
  </blockquote>
  <p>The diagram is:</p>
  <pre><code> ---------------------------                    -------------------------- 
  |    Local computer         |                  |   Remote server          |
  |     ---------------       |2.7~4.5 Mbits/sec |  --------------          |
  |    | My C++ client |&lt;-------------------------&gt;| My C++ server|         |
  |     ---------------       |                  |  --------------          |
  |                           |                  |                          |
   ---------------------------                    -------------------------- 
  </code></pre>
  <p><strong>Case 3:</strong></p>
  <p><code>iperf3</code> is used to test the bandwith between <code>local computer</code> and <code>Remote server</code>. And the result is:</p>
  <blockquote>
  <p>[ ID] Interval           Transfer     Bitrate</p>
  <p>[  5]   0.00-10.00  sec  7.25 MBytes  6.08 Mbits/sec</p>
  <p>[  5]  10.00-20.00  sec  11.6 MBytes  9.73 Mbits/sec</p>
  <p>[  5]  20.00-30.00  sec  12.2 MBytes  10.3 Mbits/sec</p>
  <p>[  5]  30.00-40.00  sec  11.4 MBytes  9.53 Mbits/sec</p>
  <p>[  5]  40.00-50.00  sec  12.0 MBytes  10.0 Mbits/sec</p>
  <p>[  5]  50.00-60.00  sec  8.28 MBytes  6.94 Mbits/sec</p>
  <hr />
  <p>[ ID] Interval           Transfer     Bitrate         Retr</p>
  <p>[  5]   0.00-60.00  sec  62.9 MBytes  8.79 Mbits/sec  222             sender</p>
  <p>[  5]   0.00-60.00  sec  62.7 MBytes  8.77 Mbits/sec                  receiver</p>
  </blockquote>
  <p>The diagram is:</p>
  <pre><code> ---------------------------                    -------------------------- 
  |    Local computer         |                  |   Remote server          |
  |     ---------------       |   8.8 Mbits/sec  |  --------------          |
  |    | iperf3 client |&lt;-------------------------&gt;| iperf3 server|         |
  |     ---------------       |                  |  --------------          |
  |                           |                  |                          |
   ---------------------------                    -------------------------- 
  </code></pre>
  <p><strong>MY CLIENT/SERVER CODE</strong></p>
  <p>server.cpp</p>
  <pre><code>#include &lt;iostream&gt;
  #include &lt;arpa/inet.h&gt;
  #include &lt;string&gt;
  #include &lt;chrono&gt;
  #include &lt;unistd.h&gt;
  #include &lt;stdlib.h&gt;
  #include &lt;iomanip&gt;
  #include &lt;signal.h&gt;
  #include &lt;fcntl.h&gt;
  
  #define PORT        (7549)
  #define CYCLE_TIME  (100)
  
  int g_iSockFd;
  
  void sigQuitHandler(int signal) {
      std::cout &lt;&lt; &quot;received SIGQUIT, doing graceful shutting down ...&quot;
          &lt;&lt; std::endl;
      close(g_iSockFd);
      exit(1);
  }
  
  
  void sigIntrHandler(int signal) {
      std::cout &lt;&lt; &quot;received SIGINT, doing graceful shutting down ...&quot;
          &lt;&lt; std::endl;
      close(g_iSockFd);
      exit(1);
  }
  
  /*
   * FUNCTION  :  enable or disable blocking of a file descriptor (socket)
   *   @PARAM  :  fd              file descriptor to be processed
   *   @PARAM  :  blocking        a bool flag to indicate blocking or non-blocking
   *                              true  ----&gt; blocking
   *                              false ----&gt; non-blocking
   *   @RETURN :  true on success
   *              false if there was an error
   */
  bool SetSocketBlockingEnabled(int fd, bool blocking) {
      if (fd &lt; 0) {
          return false;
      }
     int flags = fcntl(fd, F_GETFL, 0);
     if (flags == -1) {
         return false;
     }
     flags = blocking ? (flags &amp; ~O_NONBLOCK) : (flags | O_NONBLOCK);
     return (fcntl(fd, F_SETFL, flags) == 0) ? true : false;
  }
  
  
  int main(int argc, char** argv) {
      // register signal handler function
      struct sigaction t_struQuitSigAction;
      struct sigaction t_struIntrSigAction;
      t_struQuitSigAction.sa_handler = sigQuitHandler;
      t_struQuitSigAction.sa_flags = 0;
      sigemptyset(&amp;t_struQuitSigAction.sa_mask);
      sigaction(SIGQUIT, &amp;t_struQuitSigAction, 0);
      t_struIntrSigAction.sa_handler = sigIntrHandler;
      t_struIntrSigAction.sa_flags = 0;
      sigemptyset(&amp;t_struIntrSigAction.sa_mask);
      sigaction(SIGINT, &amp;t_struIntrSigAction, 0);
  
      // init socket parameter
      char t_cRecvBuffer[102400];
      struct sockaddr_in t_struSockAddr;
      memset(&amp;t_struSockAddr, 0, sizeof(t_struSockAddr));
      int t_iOpt = 1;
      // IPv4
      t_struSockAddr.sin_family = AF_INET;
      t_struSockAddr.sin_addr.s_addr = INADDR_ANY; // inet_addr(&quot;127.0.0.1&quot;);
      t_struSockAddr.sin_port = htons(PORT);
      // create socket
      int g_iSockFd = socket(AF_INET, SOCK_STREAM, 0);
      if (g_iSockFd == -1) {
          std::cout &lt;&lt; &quot;Error accurs when creating socket.&quot;
              &lt;&lt; std::endl &lt;&lt; &quot;System exits abnormally.&quot; &lt;&lt; std::endl;
          return -1;
      }
      int t_iError;
      if ((t_iError = setsockopt(g_iSockFd, SOL_SOCKET, SO_REUSEADDR,
                  &amp;t_iOpt, sizeof(t_iOpt))) &lt; 0) {
          std::cout &lt;&lt; &quot;reuseaddr setsockopt fail. error = &quot;
              &lt;&lt; t_iError &lt;&lt; std::endl;
          return -1;
      }
      if ((t_iError = setsockopt(g_iSockFd, SOL_SOCKET, SO_REUSEPORT,
                  &amp;t_iOpt, sizeof(t_iOpt))) &lt; 0) {
          std::cout &lt;&lt; &quot;reuseport setsockopt fail. error = &quot;
              &lt;&lt; t_iError &lt;&lt; std::endl;
          return -1;
      }
      SetSocketBlockingEnabled(g_iSockFd, true);
      if (bind(g_iSockFd, (struct sockaddr *)&amp;t_struSockAddr,
                  sizeof(t_struSockAddr)) &lt; 0) {
          std::cout &lt;&lt; &quot;bind fail.&quot; &lt;&lt; std::endl;
          return -1;
      }
      if (listen(g_iSockFd, 10) &lt; 0) {
          std::cout &lt;&lt; &quot;listen fail.&quot; &lt;&lt; std::endl;
          return -1;
      }
      int t_iNewSocket;
      while (1) {
          std::string t_strResponse(10240, 0);
          struct sockaddr_in t_struClientAddr;
          socklen_t t_ClientAddrLen = 0;
          memset(&amp;t_struClientAddr, 0, sizeof(t_struClientAddr));
          if ((t_iNewSocket = accept(g_iSockFd,
                          (struct sockaddr*)&amp;t_struClientAddr,
                          &amp;t_ClientAddrLen)) &lt; 0) {
              continue;
          }
          else {
              std::cout &lt;&lt; &quot;accept successfully.&quot; &lt;&lt; std::endl;
          }
  
          // recv
          long t_lRecvTotalBytes = 0;
          long t_lRecvTotalTime = 0;
          double t_dSpeed = 0.0;
          std::chrono::high_resolution_clock::time_point t_StartTime;
          std::chrono::high_resolution_clock::time_point t_EndTime;
          bool t_bHasBegin = false;
          while (1) {
              int t_iRecvBytes = recv(t_iNewSocket, t_cRecvBuffer,
                      sizeof(t_cRecvBuffer), 0);
              if (t_iRecvBytes &lt;= 0) {
                  std::cout &lt;&lt; &quot;Error accurs when receiving. &quot; &lt;&lt; std::endl;
                  break;
              }
              else {
                  if (t_bHasBegin == false) {
                      t_bHasBegin = true;
                      t_StartTime = std::chrono::high_resolution_clock::now();
                  }
                  t_lRecvTotalBytes += t_iRecvBytes;
                  std::string t_strRecv(t_cRecvBuffer, t_iRecvBytes);
                  std::cout &lt;&lt; &quot;Receiving &quot; &lt;&lt; t_iRecvBytes &lt;&lt; &quot;/&quot;
                      &lt;&lt; t_lRecvTotalBytes
                      &lt;&lt; &quot; bytes. &quot; &lt;&lt; std::endl
                      &lt;&lt; &quot;Msg ends with: &quot;
                      &lt;&lt; int(t_strRecv[t_iRecvBytes - 1])
                      &lt;&lt; std::endl;
                  if (t_cRecvBuffer[t_iRecvBytes - 1] == 1) {
                      std::cout &lt;&lt; &quot;recv finished.&quot; &lt;&lt; std::endl;
                      break;
                  }
              }
          }
          t_EndTime = std::chrono::high_resolution_clock::now();
          t_lRecvTotalTime =
              std::chrono::duration_cast&lt;std::chrono::microseconds&gt;(
                  t_EndTime - t_StartTime
              ).count();
          t_dSpeed = double(t_lRecvTotalBytes) / t_lRecvTotalTime
              * 1000000 * 8 / 1024 / 1024;
          std::cout &lt;&lt; &quot;t_lRecvTotalBytes = &quot;
              &lt;&lt; t_lRecvTotalBytes &lt;&lt; &quot; Bytes&quot; &lt;&lt; std::endl;
          std::cout &lt;&lt; &quot;t_lRecvTotalTime = &quot;
              &lt;&lt; t_lRecvTotalTime &lt;&lt; &quot; microseconds.&quot; &lt;&lt; std::endl;
          std::cout &lt;&lt; &quot;network speed: &quot;
              &lt;&lt; std::setprecision(2)
              &lt;&lt; t_dSpeed &lt;&lt; &quot; MBit/s&quot; &lt;&lt; std::endl;
  
          // response
          for (int i = 0; i &lt; CYCLE_TIME; ++i) {
              if (i == CYCLE_TIME - 1) {
                  t_strResponse[t_strResponse.size() - 1] = 1;
              }
              int t_iSendBytes = send(t_iNewSocket, t_strResponse.c_str(), \
                      t_strResponse.size(), 0);
              if (t_iSendBytes != t_strResponse.size()) {
                  std::cout &lt;&lt; &quot;send error&quot; &lt;&lt; std::endl;
                  return -1;
              }
              else {
                  std::cout &lt;&lt; &quot;Send message successfully. &quot;
                      &lt;&lt; std::endl &lt;&lt; &quot;CNT = &quot; &lt;&lt; i &lt;&lt; std::endl
                      &lt;&lt; &quot;size: &quot; &lt;&lt; t_iSendBytes &lt;&lt; &quot; Bytes.&quot; &lt;&lt; std::endl;
              }
          }
          // close
          close(t_iNewSocket);
          std::cout &lt;&lt; &quot;t_lRecvTotalBytes = &quot;
              &lt;&lt; t_lRecvTotalBytes &lt;&lt; &quot; Bytes&quot; &lt;&lt; std::endl;
          std::cout &lt;&lt; &quot;t_lRecvTotalTime = &quot;
              &lt;&lt; t_lRecvTotalTime &lt;&lt; &quot; microseconds.&quot; &lt;&lt; std::endl;
          std::cout &lt;&lt; &quot;network speed: &quot;
              &lt;&lt; std::setprecision(2)
              &lt;&lt; t_dSpeed &lt;&lt; &quot; MBit/s&quot; &lt;&lt; std::endl;
      }
      return 0;
  }
  </code></pre>
  <p>client.cpp</p>
  <pre><code>#include &lt;iostream&gt;
  #include &lt;arpa/inet.h&gt;
  #include &lt;string&gt;
  #include &lt;chrono&gt;
  #include &lt;unistd.h&gt;
  #include &lt;stdlib.h&gt;
  #include &lt;iomanip&gt;
  #include &lt;signal.h&gt;
  
  #define CYCLE_TIME      (100)
  #define PORT            (7549)
  
  #define LOCAL_MODE      (0)
  #define REMOTE_MODE     (1)
  
  int g_iSockFd;
  
  void sigQuitHandler(int signal) {
      std::cout &lt;&lt; &quot;received SIGQUIT, doing graceful shutting down ...&quot;
          &lt;&lt; std::endl;
      close(g_iSockFd);
      exit(1);
  }
  
  
  void sigIntrHandler(int signal) {
      std::cout &lt;&lt; &quot;received SIGINT, doing graceful shutting down ...&quot;
          &lt;&lt; std::endl;
      close(g_iSockFd);
      exit(1);
  }
  
  int main(int argc, char** argv) {
      int t_iWorkMode = LOCAL_MODE;
      if (argc == 2) {
          int t_iTempWorkMode = atoi(argv[1]);
          switch (t_iTempWorkMode) {
              case LOCAL_MODE:
                  t_iWorkMode = LOCAL_MODE;
                  break;
              case REMOTE_MODE:
                  t_iWorkMode = REMOTE_MODE;
                  break;
              default:
                  std::cout &lt;&lt; &quot;mode should be 1 or 2. LOCAL_MODE is used.&quot;
                      &lt;&lt; std::endl;
                  t_iWorkMode = LOCAL_MODE;
          }
      }
      // register signal handler function
      struct sigaction t_struQuitSigAction;
      struct sigaction t_struIntrSigAction;
      t_struQuitSigAction.sa_handler = sigQuitHandler;
      t_struQuitSigAction.sa_flags = 0;
      sigemptyset(&amp;t_struQuitSigAction.sa_mask);
      sigaction(SIGQUIT, &amp;t_struQuitSigAction, 0);
      t_struIntrSigAction.sa_handler = sigIntrHandler;
      t_struIntrSigAction.sa_flags = 0;
      sigemptyset(&amp;t_struIntrSigAction.sa_mask);
      sigaction(SIGINT, &amp;t_struIntrSigAction, 0);
  
      struct in_addr t_in_addr;
      memset(&amp;t_in_addr, 0, sizeof(t_in_addr));
      if (inet_aton(&quot;127.0.0.1&quot;, &amp;t_in_addr) == 0) {
          std::cout &lt;&lt; &quot;inet_aton fail.&quot; &lt;&lt; std::endl;
          return -1;
      }
  
      std::string t_strMsg;
      char t_cRecvBuffer[102400];
      // init socket parameter
      struct sockaddr_in t_struSockAddr;
      memset(&amp;t_struSockAddr, 0, sizeof(t_struSockAddr));
      // IPv4
      t_struSockAddr.sin_family = AF_INET;
      // remote IP address
      if (t_iWorkMode == REMOTE_MODE) {
          t_struSockAddr.sin_addr.s_addr = inet_addr(&quot;remote server IP address&quot;);
      }
      else {
          t_struSockAddr.sin_addr.s_addr = inet_addr(&quot;127.0.0.1&quot;);
      }
      // remote port
      t_struSockAddr.sin_port = htons(PORT);
      // create socket
      int g_iSockFd = socket(AF_INET, SOCK_STREAM, 0);
      if (g_iSockFd == -1) {
          std::cout &lt;&lt; &quot;Error accurs when creating socket.&quot;
              &lt;&lt; std::endl &lt;&lt; &quot;System exits abnormally.&quot; &lt;&lt; std::endl;
          return -1;
      }
      if (connect(g_iSockFd, (struct sockaddr *)&amp;t_struSockAddr, \
                  sizeof(struct sockaddr)) &lt; 0) {
          std::cout &lt;&lt; std::endl
              &lt;&lt; &quot;Error accurs when connecting to server.&quot;
              &lt;&lt; std::endl &lt;&lt; &quot;System exits abnormally.&quot; &lt;&lt; std::endl;
          return -1;
      }
      std::cout &lt;&lt; &quot;Connected to remote server.&quot; &lt;&lt; std::endl;
  
      int t_iSendBytes = 0;
      // send request
      t_strMsg = std::string(10240, 0);
      for (int i = 0; i &lt; CYCLE_TIME; ++i) {
          // 1 means the end of message
          if (i == CYCLE_TIME - 1) {
              t_strMsg[t_strMsg.size() - 1] = 1;
          }
          t_iSendBytes = send(g_iSockFd, t_strMsg.c_str(), \
                  t_strMsg.size(), 0);
          if (t_iSendBytes != t_strMsg.size()) {
              std::cout &lt;&lt; &quot;Request message is not &quot;
                  &lt;&lt; &quot;sent completely. System exits abnormally. &quot;
                  &lt;&lt; &quot;t_iSendBytes = &quot; &lt;&lt; t_iSendBytes &lt;&lt; &quot; Bytes. &quot; &lt;&lt; std::endl;
              return -1;
          }
          else {
              std::cout &lt;&lt; &quot;CNT = &quot; &lt;&lt; i &lt;&lt; std::endl
                  &lt;&lt; &quot;Msg ends with &quot;
                  &lt;&lt; int(t_strMsg[t_strMsg.size() - 1]) &lt;&lt; std::endl
                  &lt;&lt; &quot;Send request message. Bytes: &quot; &lt;&lt; t_iSendBytes
                  &lt;&lt; std::endl;
          }
      }
      std::cout &lt;&lt; &quot;Send finished.&quot; &lt;&lt; std::endl;
  
      std::cout &lt;&lt; &quot;Begin to receive ...&quot; &lt;&lt; std::endl;
      long t_lRecvTotalBytes = 0;
      long t_lRecvTotalTime = 0;
      double t_dSpeed = 0.0;
      std::chrono::high_resolution_clock::time_point t_StartTime;
      std::chrono::high_resolution_clock::time_point t_EndTime;
      t_StartTime = std::chrono::high_resolution_clock::now();
      memset(t_cRecvBuffer, 0, sizeof(t_cRecvBuffer));
      while (1) {
          int t_iRecvBytes = recv(g_iSockFd, t_cRecvBuffer, sizeof(t_cRecvBuffer), 0);
          if (t_iRecvBytes == -1) {
              std::cout &lt;&lt; &quot;Error accurs when receiving. &quot;
                  &lt;&lt; std::endl &lt;&lt; &quot;System exits abnormally.&quot;;
              return -1;
          }
          else if (t_iRecvBytes &gt; 0) {
              t_lRecvTotalBytes += t_iRecvBytes;
              std::string t_strRecv(t_cRecvBuffer, t_iRecvBytes);
              // std::cout &lt;&lt; t_strRecv;
              std::cout &lt;&lt; &quot;Receiving &quot; &lt;&lt; t_iRecvBytes &lt;&lt; &quot;/&quot;
                  &lt;&lt; t_lRecvTotalBytes &lt;&lt; &quot; bytes.&quot; &lt;&lt; std::endl;
              // 1 means the end of message.
              if (t_cRecvBuffer[t_iRecvBytes - 1] == 1) {
                  std::cout &lt;&lt; &quot;recv finished.&quot; &lt;&lt; std::endl;
                  break;
              }
          }
      }
      t_EndTime = std::chrono::high_resolution_clock::now();
      t_lRecvTotalTime =
          std::chrono::duration_cast&lt;std::chrono::microseconds&gt;(
              t_EndTime - t_StartTime
          ).count();
      t_dSpeed = double(t_lRecvTotalBytes) / t_lRecvTotalTime
          * 1000000 * 8 / 1024 / 1024;
      std::cout &lt;&lt; &quot;t_lRecvTotalBytes = &quot;
          &lt;&lt; t_lRecvTotalBytes &lt;&lt; &quot; Bytes&quot; &lt;&lt; std::endl;
      std::cout &lt;&lt; &quot;t_lRecvTotalTime = &quot;
          &lt;&lt; t_lRecvTotalTime &lt;&lt; &quot; microseconds.&quot; &lt;&lt; std::endl;
      std::cout &lt;&lt; &quot;network speed: &quot;
          &lt;&lt; std::setprecision(2)
          &lt;&lt; t_dSpeed &lt;&lt; &quot; MBit/s&quot; &lt;&lt; std::endl;
      // close
      close(g_iSockFd);
      return 0;
  }
  </code></pre>
  <p><strong>GOAL</strong></p>
  <p>The goal is that the network speed between my client and server program could reach about 80 Mbits/sec. But I cannot figure out a way to solve the problem. And the network speed tested by utorrent and iperf make me confused.</p>
  <p>I will appreciate it if anyone can help me. Thank you!</p>