title: Netty ServerBootStrap option or childOption and its buffer size impact on speed
body: <p>Netty seems to decrease the upload/download speed when the buffer size is increased beyond the default value and/or incorrect location for the override is chosen(<code>option</code> or <code>childOption</code>) in ServerBootStrap object. This becomes even more visible when the connection has a bigger latency(~300ms)</p>
  
  <p><strong>The setup</strong>:</p>
  
  <p>Netty client is on MacOS and has <strong>default values only</strong>. Using "Network Link Conditioner" with 300ms DNS latency. Default values: SendBuffer/ReceiveBuffer/LowWaterMark/HighWaterMark - 128KB/128KB/32KB/64KB.</p>
  
  <p>Netty server is on Windows 8.1 and has default values SendBuffer/ReceiveBuffer/LowWaterMark/HighWaterMark - 64KB/64KB/32KB/64KB.</p>
  
  <p>Netty version 4.1.6 Final.</p>
  
  <p>Measuring speed with wireshark using IO graphs with settings: Y Axis -> SUM(Y field), Y field -> tcp.len</p>
  
  <p>Devices are on local network.</p>
  
  <p><strong>Result(speed values)</strong>:</p>
  
  <p>Server->Client transfer(setting the SO_SNDBUF and watermark low/high values to 0.5*SO_SNDBUF/SO_SNDBUF):</p>
  
  <pre><code>Setting location\SO_SNDBUF   | 64KB   | 128KB  | 1024KB
  -----------------------------------------------------------------
  option                       | 3.6MB/s| 3.6MB/s| 3.6MB/s
  childOption                  | 0.2MB/s| 0.5MB/s| 3.6MB/s
  </code></pre>
  
  <p>Client->Server transfer(setting the SO_RCVBUF):</p>
  
  <pre><code>Setting location\SO_RCVBUF   | 64KB   | 128KB  | 1024KB
  -----------------------------------------------------------------
  option                       | 0.2MB/s| 0.5MB/s| 3.6MB/s
  childOption                  | 3.6MB/s| 0.4MB/s| 3.6MB/s
  </code></pre>
  
  <p><strong>Server Code:</strong></p>
  
  <p>"args" values:</p>
  
  <p>"sendToClient"/"sendToServer"(implicit) for transfer direction. 
  "option"/"childOption" for override type.
  "1"/"2"/"3" for buffer values.</p>
  
  <p>ObjectEchoServer.java</p>
  
  <pre><code>import io.netty.bootstrap.ServerBootstrap;
  import io.netty.channel.*;
  import io.netty.channel.nio.NioEventLoopGroup;
  import io.netty.channel.socket.SocketChannel;
  import io.netty.channel.socket.nio.NioServerSocketChannel;
  import io.netty.handler.codec.serialization.ClassResolvers;
  import io.netty.handler.codec.serialization.ObjectDecoder;
  import io.netty.handler.codec.serialization.ObjectEncoder;
  import io.netty.handler.logging.LogLevel;
  import io.netty.handler.logging.LoggingHandler;
  import io.netty.handler.ssl.SslContext;
  import io.netty.handler.ssl.SslContextBuilder;
  import io.netty.handler.ssl.util.SelfSignedCertificate;
  
  public final class ObjectEchoServer {
  
      static final int PORT = 8007;
  
      public static void main(String[] args) throws Exception {
          int rcvBuf, sndBuf, lowWaterMark, highWaterMark;
          rcvBuf = sndBuf = lowWaterMark = highWaterMark = 0;
          switch (args[2]){
              case "1":
                  rcvBuf = 64;
                  sndBuf = 64;
                  lowWaterMark = 32;
                  highWaterMark = 64;
                  break;
              case "2":
                  rcvBuf = 128;
                  sndBuf = 128;
                  lowWaterMark = 64;
                  highWaterMark = 128;
                  break;
              case "3":
                  rcvBuf = 1024;
                  sndBuf = 1024;
                  lowWaterMark = 512;
                  highWaterMark = 1024;
                  break;
  
          }
  
          EventLoopGroup bossGroup = new NioEventLoopGroup(1);
          EventLoopGroup workerGroup = new NioEventLoopGroup();
          try {
              ServerBootstrap b = new ServerBootstrap();
              b.group(bossGroup, workerGroup)
               .channel(NioServerSocketChannel.class)
               .handler(new LoggingHandler(LogLevel.INFO))
               .childHandler(new ChannelInitializer&lt;SocketChannel&gt;() {
                  @Override
                  public void initChannel(SocketChannel ch) throws Exception {
                      ChannelPipeline p = ch.pipeline();
                      p.addLast(
                              new ObjectEncoder(),
                              new ObjectDecoder(ClassResolvers.cacheDisabled(null)),
                              new ObjectEchoServerHandler(args[0]));
                  }
               });
              if(args[1].equalsIgnoreCase("childOption")) {
                  b.childOption(ChannelOption.SO_RCVBUF, rcvBuf * 1024);
                  b.childOption(ChannelOption.SO_SNDBUF, sndBuf * 1024);
                  b.childOption(ChannelOption.WRITE_BUFFER_WATER_MARK, new WriteBufferWaterMark(lowWaterMark * 1024, highWaterMark * 1024));
              } else if(args[1].equalsIgnoreCase("option")){
                  b.option(ChannelOption.SO_RCVBUF, rcvBuf * 1024);
                  b.option(ChannelOption.SO_SNDBUF, sndBuf * 1024);
                  b.option(ChannelOption.WRITE_BUFFER_WATER_MARK, new WriteBufferWaterMark(lowWaterMark * 1024, highWaterMark * 1024));
              }
              // Bind and start to accept incoming connections.
              b.bind(PORT).sync().channel().closeFuture().sync();
          } finally {
              bossGroup.shutdownGracefully();
              workerGroup.shutdownGracefully();
          }
      }
  }
  </code></pre>
  
  <p>ObjectEchoServerHandler.java</p>
  
  <pre><code>import io.netty.channel.ChannelFuture;
  import io.netty.channel.ChannelFutureListener;
  import io.netty.channel.ChannelHandlerContext;
  import io.netty.channel.ChannelInboundHandlerAdapter;
  
  public class ObjectEchoServerHandler extends ChannelInboundHandlerAdapter {
      private Object msg;
      ChannelHandlerContext ctx;
      String sendToClient;
  
      public ObjectEchoServerHandler(String sendToClient){
          this.sendToClient = sendToClient;
  
      }
  
      @Override
      public void channelActive(ChannelHandlerContext ctx){
          this.ctx = ctx;
      }
  
      @Override
      public void channelRead(ChannelHandlerContext ctx, Object msg) {
          if(sendToClient.equalsIgnoreCase("sendToClient")) {//send a data stream to server
              this.msg = msg;
              ctx.writeAndFlush(msg).addListener(trafficGenerator);
          } //else receive a data stream from client
      }
  
      @Override
      public void channelReadComplete(ChannelHandlerContext ctx) {
          ctx.flush();
      }
  
      @Override
      public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
          cause.printStackTrace();
          ctx.close();
      }
  
      private final ChannelFutureListener trafficGenerator = new ChannelFutureListener() {
          @Override
          public void operationComplete(ChannelFuture future) {
              if (future.isSuccess()) {
                  ctx.writeAndFlush(msg).addListener(trafficGenerator);
              } else {
                  future.cause().printStackTrace();
                  future.channel().close();
              }
          }
      };
  }
  </code></pre>
  
  <p><strong>Client code:</strong></p>
  
  <p>"args" values:</p>
  
  <p>"sendToClient"/"sendToServer"(implicit) for transfer direction.</p>
  
  <p>ObjectEchoClient.java</p>
  
  <pre><code>import io.netty.bootstrap.Bootstrap;
  import io.netty.channel.*;
  import io.netty.channel.nio.NioEventLoopGroup;
  import io.netty.channel.socket.SocketChannel;
  import io.netty.channel.socket.nio.NioSocketChannel;
  import io.netty.handler.codec.serialization.ClassResolvers;
  import io.netty.handler.codec.serialization.ObjectDecoder;
  import io.netty.handler.codec.serialization.ObjectEncoder;
  
  public final class ObjectEchoClient {
  
      static final String HOST = System.getProperty("host", "127.0.0.1");
      static final int PORT = 8007;
      static final int SIZE = 256;
  
      public static void main(String[] args) throws Exception {
  
          EventLoopGroup group = new NioEventLoopGroup();
          try {
              Bootstrap b = new Bootstrap();
              b.group(group)
               .channel(NioSocketChannel.class)
               .handler(new ChannelInitializer&lt;SocketChannel&gt;() {
                  @Override
                  public void initChannel(SocketChannel ch) throws Exception {
                      ChannelPipeline p = ch.pipeline();
                      p.addLast(
                              new ObjectEncoder(),
                              new ObjectDecoder(ClassResolvers.cacheDisabled(null)),
                              new ObjectEchoClientHandler(args[0]));
                      System.out.println("senbuf:"+ ch.config().getSendBufferSize());
                      System.out.println("waterhigh:"+ ch.config().getWriteBufferWaterMark().high());
                      System.out.println("waterlow:"+ ch.config().getWriteBufferWaterMark().low());
                      System.out.println("recbuf:"+ ch.config().getReceiveBufferSize());
  
                  }
               });
              // Start the connection attempt.
              b.connect(HOST, PORT).sync().channel().closeFuture().sync();
          } finally {
              group.shutdownGracefully();
          }
      }
  }
  </code></pre>
  
  <p>ObjectEchoClientHandler.java</p>
  
  <pre><code>import io.netty.channel.ChannelFuture;
  import io.netty.channel.ChannelFutureListener;
  import io.netty.channel.ChannelHandlerContext;
  import io.netty.channel.ChannelInboundHandlerAdapter;
  
  import java.util.ArrayList;
  import java.util.List;
  
  public class ObjectEchoClientHandler extends ChannelInboundHandlerAdapter {
  
      private final List&lt;String&gt; firstMessage;
      private ChannelHandlerContext ctx;
      private String sendToClient;
  
      ObjectEchoClientHandler(String sendToClient) {
          this.sendToClient = sendToClient;
  
          firstMessage = new ArrayList&lt;&gt;(ObjectEchoClient.SIZE);
          for (int i = 0; i &lt; ObjectEchoClient.SIZE; i++) {
              firstMessage.add(Integer.toString(i));
          }
      }
  
      @Override
      public void channelActive(ChannelHandlerContext ctx) {
          this.ctx = ctx;
        if(sendToClient.equalsIgnoreCase("sendToClient")) {//get a data stream from server
            this.ctx.writeAndFlush(firstMessage);
        } else {//send a stream of to server
            this.ctx.writeAndFlush(firstMessage).addListener(trafficGenerator);
        }
      }
  
      @Override
      public void channelRead(ChannelHandlerContext ctx, Object msg) {
          ctx.flush();
      }
  
      @Override
      public void channelReadComplete(ChannelHandlerContext ctx) {
          ctx.flush();
      }
  
      @Override
      public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
          cause.printStackTrace();
          ctx.close();
      }
  
      private final ChannelFutureListener trafficGenerator = new ChannelFutureListener() {
          @Override
          public void operationComplete(ChannelFuture future) {
              if (future.isSuccess()) {
                  ctx.writeAndFlush(firstMessage).addListener(trafficGenerator);
              } else {
                  future.cause().printStackTrace();
                  future.channel().close();
              }
          }
      };
  }
  </code></pre>
  
  <p><strong>The big question:</strong></p>
  
  <p>What is the correct way/place to set the buffers? The info I found(mostly code examples) is all over the place. </p>
  
  <p>One thread(<a href="https://stackoverflow.com/questions/35496345/what-is-the-difference-between-serverbootstrap-option-and-serverbootstrap-chil/35496582">What is the difference between ServerBootstrap.option() and ServerBootstrap.childOption() in netty 4.x</a>) says that the <code>ServerBootStrap.childOption</code> should be used for per client info, so I would reason that both buffers should be in the <code>ServerBootStrap.childOption</code>.</p>