title: How to solve 101500 error message performing an HTTPS call from WSO2 API Manager Store? Have I to set a certificate?
body: <p>I have the following problem with WSO2 API Manager and I suspect that it could be a problem related to certificate of the final HTTPS endpoint that it was registered.</p>
  
  <p>I try to explain my situation in details:</p>
  
  <p>First thing: I changed this section of the <strong>repository/conf/axis2/axis2.xml</strong> file in order to contact the <strong>443</strong> port instead the default <strong>8243</strong> port to call the registered endpoint over <strong>HTTPS</strong> (at the moment I can't change the registered end point port and I can't install a reverse proxy, but I have to test if the system works as expected, basically I need to call the final endpoint on the <strong>443</strong> port and I have to obtain the JSON response).</p>
  
  <p>The original section that I changed is:</p>
  
  <pre><code>&lt;transportReceiver name="https" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLListener"&gt;
      &lt;parameter name="port" locked="false"&gt;8243&lt;/parameter&gt;
      &lt;parameter name="non-blocking" locked="false"&gt;true&lt;/parameter&gt;
      &lt;!--parameter name="bind-address" locked="false"&gt;hostname or IP address&lt;/parameter--&gt;
      &lt;!--parameter name="WSDLEPRPrefix" locked="false"&gt;https://apachehost:port/somepath&lt;/parameter--&gt;
     &lt;parameter name="httpGetProcessor" locked="false"&gt;org.wso2.carbon.mediation.transport.handlers.PassThroughNHttpGetProcessor&lt;/parameter&gt;
      &lt;parameter name="keystore" locked="false"&gt;
          &lt;KeyStore&gt;
              &lt;Location&gt;repository/resources/security/wso2carbon.jks&lt;/Location&gt;
              &lt;Type&gt;JKS&lt;/Type&gt;
              &lt;Password&gt;wso2carbon&lt;/Password&gt;
              &lt;KeyPassword&gt;wso2carbon&lt;/KeyPassword&gt;
          &lt;/KeyStore&gt;
      &lt;/parameter&gt;
      &lt;parameter name="truststore" locked="false"&gt;
          &lt;TrustStore&gt;
              &lt;Location&gt;repository/resources/security/client-truststore.jks&lt;/Location&gt;
              &lt;Type&gt;JKS&lt;/Type&gt;
              &lt;Password&gt;wso2carbon&lt;/Password&gt;
          &lt;/TrustStore&gt;
      &lt;/parameter&gt;
      &lt;!--&lt;parameter name="SSLVerifyClient"&gt;require&lt;/parameter&gt;
          supports optional|require or defaults to none --&gt;
  &lt;/transportReceiver&gt;
  </code></pre>
  
  <p>I changed it in this way:</p>
  
  <pre><code>&lt;transportReceiver name="https" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLListener"&gt;
      &lt;parameter name="port" locked="false"&gt;443&lt;/parameter&gt;
      &lt;parameter name="non-blocking" locked="false"&gt;true&lt;/parameter&gt;
      &lt;!--parameter name="bind-address" locked="false"&gt;hostname or IP address&lt;/parameter--&gt;
      &lt;!--parameter name="WSDLEPRPrefix" locked="false"&gt;https://apachehost:port/somepath&lt;/parameter--&gt;
     &lt;parameter name="httpGetProcessor" locked="false"&gt;org.wso2.carbon.mediation.transport.handlers.PassThroughNHttpGetProcessor&lt;/parameter&gt;
      &lt;parameter name="keystore" locked="false"&gt;
          &lt;KeyStore&gt;
              &lt;Location&gt;repository/resources/security/wso2carbon.jks&lt;/Location&gt;
              &lt;Type&gt;JKS&lt;/Type&gt;
              &lt;Password&gt;wso2carbon&lt;/Password&gt;
              &lt;KeyPassword&gt;wso2carbon&lt;/KeyPassword&gt;
          &lt;/KeyStore&gt;
      &lt;/parameter&gt;
      &lt;parameter name="truststore" locked="false"&gt;
          &lt;TrustStore&gt;
              &lt;Location&gt;repository/resources/security/client-truststore.jks&lt;/Location&gt;
              &lt;Type&gt;JKS&lt;/Type&gt;
              &lt;Password&gt;wso2carbon&lt;/Password&gt;
          &lt;/TrustStore&gt;
      &lt;/parameter&gt;
      &lt;!--&lt;parameter name="SSLVerifyClient"&gt;require&lt;/parameter&gt;
          supports optional|require or defaults to none --&gt;
  &lt;/transportReceiver&gt;
  </code></pre>
  
  <p>Basically I only changed the <strong>8243</strong> default port with <strong>443</strong> standard HTTPS port used to expose the final endpoint.</p>
  
  <p>Now, executing the API from the <strong>Store</strong> portal I obtain a <strong>cURL</strong> command that works on the expected <strong>443</strong> port:</p>
  
  <pre><code>curl -k -X POST "https://ENDPOINT_IP_ADDRESS:443/puntualitest/v1.0.0/puntuali" -H "accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer XXXXX-YYYY-ZZZZ-KKKK-WWWW" -d "{ \"header\": { \"msgUid\": \"a36bea3f-6dc6-49d7-9376-f31692930ba9\", \"timestamp\": 1567060509108, \"metadata\": { \"TRACKER_BIZID_REV_CODICE\": \"7175\", \"TRACKER_BIZID_REV_NUMERO\": \"545/2019\" }, \"codApplication\": null, \"codEnte\": null, \"invocationContext\": null, \"caller\": \"SRW\", \"user\": null, \"service\": \"crediti.invioPosizioneCreditoria\" }, \"body\": { \"@dto\": \"binary\", \"content\": \"PD94bWwgdmVyc2..............................+\" }}"
  </code></pre>
  
  <p>This seems correct, trying to perform the previous cURL command from the bash shell of the machine on which WSO2 API Manager is installed I obtain a JSON response from the API, this:</p>
  
  <pre><code>{"timestamp":"2020-02-29T12:13:54.630+0000","status":404,"error":"Not Found","message":"No message available","path":"/puntualitest/v1.0.0/puntuali"}
  </code></pre>
  
  <p>It contains an error message but I think that it is cause by a "wrong" payload, anyway it seems that the final registered API endpoint received my request, elaboreted it and return me a JSON message (is it this reasoning correct)?</p>
  
  <p>The problem is that trying to perform the request directly from the inside of the <strong>Store</strong> portal of WSO2 API Manager I am obtaining the following error message:</p>
  
  <pre><code>&lt;am:fault xmlns:am="http://wso2.org/apimanager"&gt;
    &lt;am:code&gt;101500&lt;/am:code&gt;
    &lt;am:type&gt;Status report&lt;/am:type&gt;
    &lt;am:message&gt;Runtime Error&lt;/am:message&gt;
    &lt;am:description&gt;Error in Sender&lt;/am:description&gt;
  &lt;/am:fault&gt;
  </code></pre>
  
  <p>Reading on the documantation it seems to me that the error having code <strong>101500</strong> could be related to a <strong>certificate problem</strong>: </p>
  
  <p><a href="https://stackoverflow.com/questions/39227993/wso2-esb-4-9-0-what-means-error-101500">WSO2 ESB 4.9.0: what means error 101500</a></p>
  
  <p>The previous link referer to ESB product and not API Manager but I suspect that the problem could be the same. I suspect it also because into my log file (<strong>/usr/lib/wso2/wso2am/2.6.0/repository/logs/wso2carbon.log</strong>) when I perform the previous request from the Store portal I obtain the following error message:</p>
  
  <pre><code>TID: [-1] [] [2020-02-29 13:34:58,686] ERROR {org.apache.synapse.transport.passthru.SourceHandler} -  I/O error: Received fatal alert: certificate_unknown {org.apache.synapse.transport.passthru.SourceHandler}
  javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
          at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
          at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647)
          at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1615)
          at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1781)
          at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1070)
          at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896)
          at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766)
          at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
          at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:245)
          at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:280)
          at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:410)
          at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:119)
          at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
          at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
          at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
          at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
          at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
          at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
          at java.lang.Thread.run(Thread.java:748)
  TID: [-1234] [] [2020-02-29 13:34:58,827]  WARN {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl} -  Invalid session id for thrift authenticator. {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl}
  TID: [-1234] [] [2020-02-29 13:34:58,829] ERROR {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl} -  Error in invoking validate key via thrift.. {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl}
  TID: [-1234] [] [2020-02-29 13:34:58,830]  WARN {org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClient} -  Login failed.. Authenticating again.. {org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClient}
  TID: [-1234] [] [2020-02-29 13:34:58,846]  INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} -  'admin@carbon.super [-1234]' logged in at [2020-02-29 13:34:58,845+0100] from IP address  {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
  TID: [-1] [] [2020-02-29 13:34:58,941] ERROR {org.apache.synapse.transport.passthru.TargetHandler} -  I/O error: General SSLEngine problem {org.apache.synapse.transport.passthru.TargetHandler}
  javax.net.ssl.SSLHandshakeException: General SSLEngine problem
          at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521)
          at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528)
          at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1197)
          at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1165)
          at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
          at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:237)
          at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:271)
          at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:410)
          at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:119)
          at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
          at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
          at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
          at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
          at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
          at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
          at java.lang.Thread.run(Thread.java:748)
  Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
          at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
          at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709)
          at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318)
          at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
          at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
          at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
          at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
          at sun.security.ssl.Handshaker$1.run(Handshaker.java:970)
          at sun.security.ssl.Handshaker$1.run(Handshaker.java:967)
          at java.security.AccessController.doPrivileged(Native Method)
          at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459)
          at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:255)
          at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:293)
          ... 9 more
  Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
          at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
          at sun.security.validator.Validator.validate(Validator.java:262)
          at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
          at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
          at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
          at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626)
          ... 17 more
  Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
          at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
          at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
          ... 23 more
  TID: [-1234] [] [2020-02-29 13:34:58,948]  WARN {org.apache.synapse.endpoints.EndpointContext} -  Endpoint : SirPuntuali--vv1.0.0_APIproductionEndpoint with address https://ENDPOINT_IP_ADDRESS/cmd/j/ will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
  TID: [-1234] [] [2020-02-29 13:34:58,948]  WARN {org.apache.synapse.endpoints.EndpointContext} -  Suspending endpoint : SirPuntuali--vv1.0.0_APIproductionEndpoint with address https://ENDPOINT_IP_ADDRESS/cmd/j/ - last suspend duration was : 30000ms and current suspend duration is : 30000ms - Next retry after : Sat Feb 29 13:35:28 CET 2020 {org.apache.synapse.endpoints.EndpointContext}
  TID: [-1234] [] [2020-02-29 13:34:58,949]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  STATUS = Executing default 'fault' sequence, ERROR_CODE = 101500, ERROR_MESSAGE = Error in Sender {org.apache.synapse.mediators.builtin.LogMediator}
  TID: [-1] [] [2020-02-29 13:34:58,979]  INFO {org.wso2.carbon.databridge.core.DataBridge} -  user admin connected {org.wso2.carbon.databridge.core.DataBridge}
  </code></pre>
  
  <p>So it seems that WSO2 API Manager is trying to send the request to the correct endpoint but there is a certificate problem. Is it this reasoning correct?</p>
  
  <p>If this could be the problem now I have some doubts about what I have to do to solve my problem:</p>
  
  <ol>
  <li><p>Have I to obtain a certificate generated on the server hosting the final endpoint and have I to set it on my WSO2 API Manager or, on the contrary, have I to generate a certificate on the WSO2 API Manager machine and I have to provide it to the machine hosting the final API? 
  Reading on the documentation it seems to me that I have to obtain a certificate from the API hosting machine and I have to upload this certificate into WSO2 API Manager application (as shown here: <a href="https://apim.docs.wso2.com/en/latest/learn/design-api/endpoints/certificates/" rel="nofollow noreferrer">https://apim.docs.wso2.com/en/latest/learn/design-api/endpoints/certificates/</a>). But I am not sure of this assumption.</p></li>
  <li><p>A self signed certificate is ok? In case what is the procedure to generate it and what kind of certificate I need to obtain (I have to provide precise information to the guys working on the final API machine)</p></li>
  <li><p>Probably a trivial question: the Store portal is generating a <strong>cURL</strong> request using the <strong>-k</strong> option that is used to ignore certificate (infact performing it directly in the shell it seems to work fine). Why sending the request from the Store portal is not working? I suspect that it generates a <strong>cURL</strong> request for test pourpose but that under the hood the API Manager is not performing a simple cURL request.</p></li>
  </ol>