title: JavaMail (SMTPS) + Grizzly, handshake failure
body: <p><br/>
  I'm currently trying to test my own implementation of SMTP server which i made on top of Grizzly framework and now struggling with handshake problem during checking SMTPS support with help of JavaMail.<br/>
  I found <a href="https://stackoverflow.com/questions/22724862/grizzly-with-tls-handshaking-issue"><strong>similar problem</strong></a> with handshake but hopefully (or not for me) i don't use client mode for Grizlly SSLEngineConfigurator, so i guess that root of the problem should be different.<br/>
  My code for Grizzly SSL configuration initialization (in integration test) looks following:</p>
  
  <pre><code>try (InputStream keystoreStream = TransportEncryptionIT.class.getResourceAsStream("server_keystore.jks")) {
       SSLContextConfigurator sslCon = new SSLContextConfigurator();
       sslCon.setSecurityProtocol("TLSv1.2");
       sslCon.setKeyStoreType("JKS");
       sslCon.setKeyStoreBytes(IOUtils.toByteArray(keystoreStream));
       sslCon.setKeyStorePass("");
       sslConf = new SSLEngineConfigurator(sslCon, false, false, false);
  }
  </code></pre>
  
  <p>I also install SSLFilter to FilterChain (with server SSL config) immidiatly after TransportFilter (which is the first filter in chain).</p>
  
  <pre><code>filterChainBuilder.add(new SSLFilter(sslConf , null));
  </code></pre>
  
  <p>And configure javamail for SMTPS in following way:</p>
  
  <pre><code>Properties properties = new Properties();
  properties.setProperty("mail.transport.protocol", "smtps");
  properties.setProperty("mail.smtps.host", ${host});
  properties.setProperty("mail.smtps.port", ${port});
  //need this since i'm using self signed certificate which is not added to truststore
  properties.setProperty("mail.smtps.ssl.trust", "*"); 
  </code></pre>
  
  <p>Results of handshake (SSL debug server):</p>
  
  <pre><code>[Raw read]: length = 163
  0000: 01 00 00 9F 03 01 54 7C   D5 30 98 30 CA 59 39 D7  ......T..0.0.Y9.
  0010: B0 4A B6 FC 8F 8D E9 BD   4B 88 D6 BF EE E8 F9 FF  .J......K.......
  0020: 8A EB 28 CF 98 AB 00 00   38 C0 0A C0 14 00 35 C0  ..(.....8.....5.
  0030: 05 C0 0F 00 39 00 38 C0   09 C0 13 00 2F C0 04 C0  ....9.8...../...
  0040: 0E 00 33 00 32 C0 07 C0   11 00 05 C0 02 C0 0C C0  ..3.2...........
  0050: 08 C0 12 00 0A C0 03 C0   0D 00 16 00 13 00 04 00  ................
  0060: FF 01 00 00 3E 00 0A 00   34 00 32 00 17 00 01 00  ....&gt;...4.2.....
  0070: 03 00 13 00 15 00 06 00   07 00 09 00 0A 00 18 00  ................
  0080: 0B 00 0C 00 19 00 0D 00   0E 00 0F 00 10 00 11 00  ................
  0090: 02 00 12 00 04 00 05 00   14 00 08 00 16 00 0B 00  ................
  00A0: 02 01 00                                           ...
  Grizzly-worker(1), READ: TLSv1 Handshake, length = 163
  Grizzly-worker(2), fatal error: 80: problem unwrapping net record
  javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 1
  Grizzly-worker(2), SEND TLSv1.2 ALERT:  fatal, description = internal_error
  Grizzly-worker(2), WRITE: TLSv1.2 Alert, length = 2
  </code></pre>
  
  <p>Results of handshake (SSL debug client):</p>
  
  <pre><code>[Raw read]: length = 249
  0000: 01 00 00 F5 03 03 54 7C   D5 30 6E 1C E7 B3 36 DE  ......T..0n...6.
  0010: A6 26 73 78 2B 66 D1 D1   E8 C4 94 CA 63 34 22 BF  .&amp;sx+f......c4".
  0020: 60 9D 13 03 59 1F 00 00   70 C0 24 C0 28 00 3D C0  `...Y...p.$.(.=.
  0030: 26 C0 2A 00 6B 00 6A C0   0A C0 14 00 35 C0 05 C0  &amp;.*.k.j.....5...
  0040: 0F 00 39 00 38 C0 23 C0   27 00 3C C0 25 C0 29 00  ..9.8.#.'.&lt;.%.).
  0050: 67 00 40 C0 09 C0 13 00   2F C0 04 C0 0E 00 33 00  g.@...../.....3.
  0060: 32 C0 07 C0 11 00 05 C0   02 C0 0C C0 2C C0 2B C0  2...........,.+.
  0070: 30 00 9D C0 2E C0 32 00   9F 00 A3 C0 2F 00 9C C0  0.....2...../...
  0080: 2D C0 31 00 9E 00 A2 C0   08 C0 12 00 0A C0 03 C0  -.1.............
  0090: 0D 00 16 00 13 00 04 00   FF 01 00 00 5C 00 0A 00  ............\...
  00A0: 34 00 32 00 17 00 01 00   03 00 13 00 15 00 06 00  4.2.............
  00B0: 07 00 09 00 0A 00 18 00   0B 00 0C 00 19 00 0D 00  ................
  00C0: 0E 00 0F 00 10 00 11 00   02 00 12 00 04 00 05 00  ................
  00D0: 14 00 08 00 16 00 0B 00   02 01 00 00 0D 00 1A 00  ................
  00E0: 18 06 03 06 01 05 03 05   01 04 03 04 01 03 03 03  ................
  00F0: 01 02 03 02 01 02 02 01   01                       .........
  main, READ: TLSv1.2 Handshake, length = 249
  main, handling exception: javax.net.ssl.SSLProtocolException: Handshake message sequence     violation, 1
  main, SEND TLSv1.2 ALERT:  fatal, description = unexpected_message
  main, WRITE: TLSv1.2 Alert, length = 2
  [Raw write]: length = 7
  0000: 15 03 03 00 02 02 0A                               .......
  main, called closeSocket()
  </code></pre>
  
  <hr>
  
  <p>Strange thing for me is that SMTP + StartTLS works perfectly with my server, handshake was done successfully, i checked this via ssl debug.<br/></p>
  
  <p>Sole thing which I'm not sure is that I'm using self created certificate which is not added to truststore. Could it be that java mail option "mail.${protocol}.ssl.trust" works for SMTP + StartTLS but somehow does not work for SMTPS ?<br/>
  I also tried to use <a href="http://www.oracle.com/technetwork/java/javamail145sslnotes-1562622.html" rel="nofollow noreferrer"><strong>MailSSLSocketFactory</strong></a> to force javamail to trust server certificate but it seems to be even not invoked (checked with debuger).</p>
  
  <p>JavaMail version 1.4.7<br/>
  Grizzly 2.3.17</p>
  
  <hr>
  
  <p><strong>Update</strong></p>
  
  <p>Finally i found difference in source code, it seems to be nothing about javamail but my usage of Grizzly framework and installation of SSLFilter to FilterChain.
  When I start server as SMTPS server, filter is installed in following way:</p>
  
  <pre><code>    filterChainBuilder.add(new TransportFilter());
  
      if (configuration.getSslConfig() != null) {
          sslFilter = new SSLFilter(configuration.getSslConfig(), null);
          if (configuration.isSmtpsEnabled()) {
              filterChainBuilder.add(sslFilter);
          }
      }
  
      smtpLineCodecFilter = new SmtpLineCodecFilter();
      filterChainBuilder.add(smtpLineCodecFilter);
  </code></pre>
  
  <p>But for StartTLS (since client might ask to raise transport encryption) it's done in another way:</p>
  
  <pre><code>    FilterChain securedFilterChain = new DefaultFilterChain(defaultChain);
      int transportFilterindex = defaultChain.indexOfType(TransportFilter.class);
      // Add connection security layer to the chain
      securedFilterChain.add(transportFilterindex + 1, sslFilter);
  </code></pre>
  
  <p>This difference leads to different filters to be installed, so filter chain looks like this:<br/> 
  In first case - SSLBaseFilter.SSLTransportFilterWrapper -> SSLFilter<br/>
  In second case - TransportFilter -> SSLFilter</p>
  
  <p>Now I don't know for sure which is the proper way, but at least second case works for me.<br/> 
  I would highly appreciate Grizzly Guru answers :)</p>