title: how to find root cause of stuck thread from thread dump report?
body: <p>I have a live application deployed on a #Weblogic (12c) application server that has 4 stuck threads right now. it occurred after more than one month of uptime.<br>
  <code>extra info:</code>
  as the thread dump shows the problem occurs in my "postSMS" method that uses okhttp library to call another RESTFull service:<br>
  here is my method body:</p>
  
  <pre><code>@POST
  @Path("postSMS")
  @Consumes({MediaType.APPLICATION_JSON + ";charset=utf-8"})
  @Produces({MediaType.TEXT_PLAIN + ";charset=utf-8"})
  public void postSMS(SMSObjMMC objMMC, @Suspended AsyncResponse response, @Context final HttpServletRequest request) {
      if (objMMC != null &amp;&amp; objMMC.getOpCode() == null) {
          objMMC.setOpCode(EnumSpcOpcode.FROM_MMC.getOpcode());
      }
      try {
          RestConstant.setTimeoutToRequest(AppConstants.HOST_SPEC.getMMC().getClientTimeOut(), response);
          String remoteAddr = request.getRemoteAddr() + "-" + request.getRemotePort();
          String remoteHost = request.getRemoteHost();
          HttpHeaderRequest httpHeader = HttpHelper.extractHttpHeaderFromRequest(request);//Save Header
          SmsOut smsOut = new SmsOut(ParseDataType.tryParsLong(objMMC.getMobileNo()), objMMC.getBody(), objMMC.getOpCode());
          smsOut.setHttpHeader(httpHeader);
          smsOut.setSmsProvider(AppConstants.getSmsProviderSendPost());
          String messageId = smsHelper.sendSms(smsOut); \\this object is injected
          if (ParseDataType.tryParsLong(messageId) &gt; 1000)
              LOGGER_SUMMARY.info(objMMC.getMobileNo() + "-&gt;" + MESSAGE_SMS_SENT_SUCCESS + ": " + messageId +
                      " from line: " + AppConstants.getSmsProviderSendPost().getSmsNumber() + " OpCode: " + objMMC.getOpCode());
          else
              LOGGER_SUMMARY.error(objMMC.getMobileNo() + "-&gt;" + MESSAGE_SMS_SENT_FAILED + ": " +
                      EnumSMSErrorCodes.getDescriptionForCode((int) ParseDataType.tryParsInt(messageId)) + " OpCode: " +
                      objMMC.getOpCode() + " from line: " + AppConstants.getSmsProviderSendPost().getSmsNumber());
          LOGGER_POSTED_REQUEST.info(objMMC.getMobileNo() + "-&gt; messageId: " + messageId + ", form MMC(RemoteAddr:" + remoteAddr +
                  " ,remoteHost:" + remoteHost + "): MMC Message Body[" +
                  objMMC.getBody().replaceAll("\n", " ") + "]");
          response.resume(messageId);
      } catch (Exception ex) {
          response.resume("-1");
          LOGGER_SUMMARY.error("EXCEPTION IN POST SmsGatewayJaxrsConfig FROM MMC: " + objMMC.getMobileNo() + " OpCode: " +
                  objMMC.getOpCode() + " ,message is:" + ex.getMessage());
          LOGGER_STACK_TRACE.error("EXCEPTION IN POST SmsGatewayJaxrsConfig FROM MMC: " + objMMC.getMobileNo(), ex);
      }
  }
  </code></pre>
  
  <p>I need help to find the root cause of this thread stuck.
  here is the thread dump report in following:</p>
  
  <pre><code>[STUCK] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'
  priority:1 - threadId:0x000000002b5ad800 - nativeId:0x152c - nativeId (decimal):5420 - state:RUNNABLE
  stackTrace:
  java.lang.Thread.State: RUNNABLE
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  at java.net.SocketInputStream.read(SocketInputStream.java:170)
  at java.net.SocketInputStream.read(SocketInputStream.java:141)
  at okio.Okio$2.read(Okio.java:140)
  at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
  at okio.RealBufferedSource.indexOf(RealBufferedSource.java:355)
  at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:227)
  at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215)
  at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
  at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
  at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
  at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
  at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
  at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
  at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
  at okhttp3.RealCall.execute(RealCall.java:77)
  at client.okhttp.CommonOkHttpClient.doGetRequest(CommonOkHttpClient.java:66)
  at client.okhttp.ProviderClient.doSendSmsRequest(ProviderClient.java:107)
  at client.okhttp.ProviderClient.doSendSmsRequest(ProviderClient.java:95)
  at sun.reflect.GeneratedMethodAccessor240.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
  at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
  at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
  at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101)
  at ir.bki.util.produces.loggers.interceptors.LoggingInterceptor.intercept(LoggingInterceptor.java:64)
  at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:94)
  at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
  at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101)
  at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:101)
  at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
  at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101)
  at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
  at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
  at sun.reflect.GeneratedMethodAccessor160.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:94)
  at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
  at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
  at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
  at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
  at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
  at com.sun.proxy.$Proxy245.doSendSmsRequest(Unknown Source)
  at client.okhttp.ProviderClient_siljso_NoIntfViewImpl.__WL_invoke(Unknown Source)
  at weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:33)
  at client.okhttp.ProviderClient_siljso_NoIntfViewImpl.doSendSmsRequest(Unknown Source)
  at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:436)
  at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:127)
  at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
  at org.jboss.weld.bean.proxy.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:67)
  at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
  at client.okhttp.ProviderClient$Proxy$_$$_Weld$EnterpriseProxy$.doSendSmsRequest(Unknown Source)
  at client.SmsHelper.sendSms(SmsHelper.java:53)
  at service.sms.exchange.SmsSend.postSMS(SmsSend.java:87)
  at service.sms.exchange.SmsSend$Proxy$_$$_WeldSubclass.postSMS$$super(Unknown Source)
  at sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49)
  at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77)
  at ir.bki.util.produces.loggers.interceptors.LoggableCasheInterceptor.interceptDatabaseMethod(LoggableCasheInterceptor.java:63)
  at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:74)
  at org.jboss.weld.interceptor.proxy.NonTerminalAroundInvokeInvocationContext.proceedInternal(NonTerminalAroundInvokeInvocationContext.java:64)
  at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77)
  at ir.bki.util.produces.loggers.interceptors.LoggingInterceptor.intercept(LoggingInterceptor.java:64)
  at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:74)
  at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84)
  at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72)
  at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56)
  at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
  at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
  at service.sms.exchange.SmsSend$Proxy$_$$_WeldSubclass.postSMS(Unknown Source)
  at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
  at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:143)
  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
  at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
  at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
  at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
  at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
  at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
  at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
  at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
  at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
  at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:471)
  at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:425)
  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:383)
  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:336)
  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:223)
  at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)
  at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
  at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
  at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)
  at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
  at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
  at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:32)
  at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
  at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3683)
  at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3649)
  at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:326)
  at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
  at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
  at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
  at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2433)
  at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2281)
  at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2259)
  at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1686)
  at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1646)
  at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:270)
  at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:348)
  at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333)
  at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:54)
  at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
  at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:640)
  at weblogic.work.ExecuteThread.execute(ExecuteThread.java:406)
  at weblogic.work.ExecuteThread.run(ExecuteThread.java:346)
  </code></pre>