title: IBM WebSphere Liberty working with IHS Server Problems with Certificate(s)
body: <p>I have a liberty server and an IHS server (both on different hosts). This is on rhel7.</p>
  
  <p>I've followed a number of guides that I found that walk thru the setup of keystores and sharing of certs but still run into issues. For reference, I tried the methods detailed <a href="https://jazz.net/wiki/bin/view/Deployment/CreateIHSPLUGINFORLIBERTYPROFILE" rel="nofollow noreferrer">https://jazz.net/wiki/bin/view/Deployment/CreateIHSPLUGINFORLIBERTYPROFILE</a> and <a href="https://www.ibm.com/support/knowledgecenter/en/SSEQTJ_9.0.5/com.ibm.websphere.ihs.doc/ihs/tihs_install_config_liberty.html" rel="nofollow noreferrer">https://www.ibm.com/support/knowledgecenter/en/SSEQTJ_9.0.5/com.ibm.websphere.ihs.doc/ihs/tihs_install_config_liberty.html</a>.</p>
  
  <p>A quick summary of what I have done this past attempt... On the liberty server I added the following to my server.xml to add a plugin include (to keep things easier to read):</p>
  
  <pre><code>&lt;include location="${server.config.dir}/plugin-join-include.xml" /&gt;
  </code></pre>
  
  <p>And the plugin-join-include.xml:</p>
  
  <pre><code>        &lt;featureManager&gt;
                  &lt;feature&gt;ssl-1.0&lt;/feature&gt;
          &lt;/featureManager&gt;
  
          &lt;keyStore id="libertyKey" password="xxxx"
                location="${server.config.dir}/resources/security/libkey.jks"/&gt;
  
          &lt;pluginConfiguration webserverPort="10500" webserverSecurePort="10447"
                  pluginInstallRoot="/opt/IBM/wasadmin/Plugins"
                  sslKeyRingLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.kdb"
                  sslStashFileLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.sth"
                  sslCertlabel="libertyKey"/&gt;
  </code></pre>
  
  <p>On my IHS server the path for the root Plugin directory is as indicated above in the file at /opt/IBM/wasadmin/Plugins and the plugin-key.kdb and plugin-key.sth are located in the locations listed above.</p>
  
  <p>I reboot the liberty server and it generated the plugin-cfg.xml in the logs/state directory:</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt;&lt;!--HTTP server plugin config file for app generated on 2020.03.04 at 12:32:02 UTC--&gt;
  &lt;Config ASDisableNagle="false" AcceptAllContent="false" AppServerPortPreference="HostHeader" ChunkedResponse="false" ConfigHash="1077723051" FIPSEnable="false" IISDisableNagle="false" IISPluginPriority="High" IgnoreDNSFailures="false" RefreshInterval="60" ResponseChunkSize="64" SSLConsolidate="false" TrustedProxyEnable="false" VHostMatchingCompat="false"&gt;
     &lt;Log LogLevel="Error" Name="/opt/IBM/wasadmin/Plugins/logs/webserver1/http_plugin.log"/&gt;
     &lt;Property Name="ESIEnable" Value="true"/&gt;
     &lt;Property Name="ESIMaxCacheSize" Value="1024"/&gt;
     &lt;Property Name="ESIInvalidationMonitor" Value="false"/&gt;
     &lt;Property Name="ESIEnableToPassCookies" Value="false"/&gt;
     &lt;Property Name="PluginInstallRoot" Value="/opt/IBM/wasadmin/Plugins"/&gt;
  &lt;!-- Configuration generated using httpEndpointRef=defaultHttpEndpoint--&gt;
  &lt;!-- The default_host contained only aliases for endpoint defaultHttpEndpoint.
           The generated VirtualHostGroup will contain only configured web server ports:
                  webserverPort=10500
                  webserverSecurePort=10447 --&gt;
     &lt;VirtualHostGroup Name="default_host"&gt;
        &lt;VirtualHost Name="*:10500"/&gt;
        &lt;VirtualHost Name="*:10447"/&gt;
     &lt;/VirtualHostGroup&gt;
     &lt;ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="true" LoadBalance="Round Robin" Name="affms_default_node_Cluster" PostBufferSize="0" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60" ServerIOTimeoutRetry="-1"&gt;
        &lt;Server CloneID="412b3187-16c4-41b0-86e8-1e327c1c6b1b" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="default_node_affms" ServerIOTimeout="900" WaitForContinue="false"&gt;
           &lt;Transport Hostname="libertyhost" Port="10500" Protocol="http"/&gt;
           &lt;Transport Hostname="libertyhost" Port="10447" Protocol="https"&gt;
              &lt;Property Name="keyring" Value="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.kdb"/&gt;
              &lt;Property Name="stashfile" Value="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.sth"/&gt;
              &lt;Property Name="certLabel" Value="libertyKey"/&gt;
           &lt;/Transport&gt;
        &lt;/Server&gt;
        &lt;PrimaryServers&gt;
           &lt;Server Name="default_node_app"/&gt;
        &lt;/PrimaryServers&gt;
     &lt;/ServerCluster&gt;
     &lt;UriGroup Name="default_host_app_default_node_Cluster_URIs"&gt;
        &lt;Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/app/ui/*"/&gt;
        &lt;Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/adminCenter/*"/&gt;
     &lt;/UriGroup&gt;
     &lt;Route ServerCluster="app_default_node_Cluster" UriGroup="default_host_app_default_node_Cluster_URIs" VirtualHostGroup="default_host"/&gt;
  </code></pre>
  
  <p>I copy the plugin-cfg.xml over to the IHS server at /opt/IBM/wasadmin/Plugins/config/webserver1/plugin-cfg.xml</p>
  
  <p>The plugin stuff works only the SSL portion between IHS and the app server give me issues.</p>
  
  <p>On the IHS server I have created the plugin-key.kdb keystore with stash plugin-key.sth and imported the cert I extracted from the liberty server /opt/IBM/wlp/usr/servers/app/resources/security/key.p12 the default personal cert and copy it to the IHS server and add the default liberty cert to the plugin-key.kdb keystore.</p>
  
  <p>At the bottom of the httpd.conf is:</p>
  
  <pre><code>LoadModule was_ap22_module /opt/IBM/wasadmin/Plugins/bin/64bits/mod_was_ap22_http.so
  WebSpherePluginConfig /opt/IBM/wasadmin/Plugins/config/webserver1/plugin-cfg.xml
  </code></pre>
  
  <p>I bounce the apachectl and go to the url for the application on port 8443 <a href="https://hostname:8443/app/ui" rel="nofollow noreferrer">https://hostname:8443/app/ui</a> and get a "500 Internal Server Error".</p>
  
  <p>The http_plugin.log plugin logs show:</p>
  
  <pre><code>[04/Mar/2020:18:23:48.31652] 00002f26 3cff9700 - ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414) PARTNER CERTIFICATE DN=CN=hostname,OU=app,O=ibm,C=us, Serial=xxxxxxxxxxx
  [04/Mar/2020:18:23:48.31655] 00002f26 3cff9700 - ERROR: Ensure correct certificate is marked as default certificate in plugin-key.kdb.                           Consult documentation regarding Administering application security and Securing communications for more information.
  [04/Mar/2020:18:23:48.31657] 00002f26 3cff9700 - ERROR: Last validation error [575010]: GSKVAL_ERROR_NO_CHAIN_BUILT
  [04/Mar/2020:18:23:48.31658] 00002f26 3cff9700 - ERROR: Subject [[Class=]GSKVALMethod::PKIX[Issuer=]OU=memberRoot,O=xxxx-xxx-xxx-xxx-xxxxxxxx,DC=com.ibm.ws.collective[#=]082d6a83e5ec[Subject=]CN=hostname,OU=app,O=ibm,C=us] failed certificate validation
  [04/Mar/2020:18:23:48.31659] 00002f26 3cff9700 - ERROR: X509 Certificate validation log: [[Class=]GSKVALMethod::PKIX[Time=]2020:3:4:18:23:48.316[buildChain=][Error=]GSKVAL_ERR_NO_CHAIN_BUILT[Info=]OU=memberRoot,O=xxx-xxx-xxx-xxx-xxx,DC=com.ibm.ws.collective[Cert=][Issuer=]OU=memberRoot,O=xxx-xxx-xxx-xxx-xxxx,DC=com.ibm.ws.collective[#=]082d6a83e5ec[Subject=]CN=hostname,OU=app,O=ibm,C=us[=Cert][=buildChain]^M
  ]
  [04/Mar/2020:18:23:48.31664] 00002f26 3cff9700 - ERROR: ws_common: websphereGetStream: Could not open stream
  [04/Mar/2020:18:23:48.31667] 00002f26 3cff9700 - ERROR: ws_common: websphereExecute: Failed to create the stream
  [04/Mar/2020:18:23:48.31668] 00002f26 3cff9700 - ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to 'default_node_app' on host 'hostname:10447'; will try another one
  [04/Mar/2020:18:23:48.31669] 00002f26 3cff9700 - ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find an app server to handle this request
  [04/Mar/2020:18:23:48.31670] 00002f26 3cff9700 - ERROR: ESI: getResponse: failed to get response: rc = 2
  [04/Mar/2020:18:23:48.31671] 00002f26 3cff9700 - ERROR: [xxx.xx.xx.17://app/ui/] ws_common: websphereHandleRequest: Failed to handle request rc=2
  
  </code></pre>
  
  <p>Thank you!</p>