title: How to resolve deadlock issue in ColdFusion 9: coldfusion.util.AbstractCache$Lock
body: <p>I have been attempting to resolve an issue where the execution of certain scripts causes a deadlock, putting all subsequent requests into limbo, using up 99.9% of the CPU, and eventually effectively crashing the server.</p>
  
  <p>Here is an example stack trace for one of the requests which has been put in limbo (waiting eternally):</p>
  
  <pre><code>Thread Stack Trace
  Trace Time: 21:00:44.463 06-Jun-2012
  Request ID: 6131
  Script Name: http://www.example.com/allreviews.cfm
  Started: 21:00:21.225 06-Jun-2012
  Exec Time: 23238ms
  Memory Used: (24%)230,667KB
  Memory Free: 701,428KB
  Thread ID: 0x191e (6430)
  Thread Name: jrpp-494
  Priority: 5
  Hashcode: 1081611879
  State: WAITING
  
  "jrpp-494" prio=5 in Object.wait()
  
  java.lang.Object.wait(Object.java:???)[Native Method]
  - waiting on &lt;0x9253305&gt; (a coldfusion.util.AbstractCache$Lock)
  java.lang.Object.wait(Object.java:485)
  coldfusion.util.AbstractCache.fetch(AbstractCache.java:46)
  coldfusion.util.SoftCache.get_statsOff(SoftCache.java:133)
  coldfusion.util.SoftCache.get(SoftCache.java:81)
  coldfusion.runtime.TemplateClassLoader.findClass(TemplateClassLoader.java:609)
  coldfusion.runtime.RuntimeServiceImpl.getFile(RuntimeServiceImpl.java:785)
  coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:766)
  coldfusion.tagext.lang.CustomTag.setName(CustomTag.java:21)
  cfApplication2ecfm456206189._factor0(/srv/www/htdocs/www.example.com/www/Application.cfm:28)
  cfApplication2ecfm456206189.runPage(/srv/www/htdocs/www.example.com/www/Application.cfm:1)
  coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
  coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
  coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
  coldfusion.filter.CfincludeFilter.include(CfincludeFilter.java:33)
  coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:279)
  coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
  coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
  coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
  coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
  coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
  coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
  coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
  coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
  coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
  coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
  coldfusion.CfmServlet.service(CfmServlet.java:200)
  coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
  jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
  com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doHttpServletRequest(FusionReactorCoreFilter.java:503)
  com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFusionRequest(FusionReactorCoreFilter.java:337)
  com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:246)
  com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter(FusionReactorFilter.java:121)
  jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
  coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
  coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
  jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
  jrun.servlet.FilterChain.service(FilterChain.java:101)
  jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
  jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
  jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
  jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
  jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
  jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
  jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
  jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
  jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
  </code></pre>
  
  <p>If you're interested, you can see the <a href="https://gist.github.com/2884737" rel="nofollow noreferrer">full stack trace</a> with what I'll call 'the locking script' at the top and all the others waiting on it.</p>
  
  <p>When I first encountered this issue, I didn't have the stack traces. I posted the question, "<a href="https://stackoverflow.com/questions/10902098/when-coldfusion-is-maxing-out-the-cpu-how-do-i-find-out-what-its-chewing-choki">When ColdFusion is maxing out the CPU, how do I find out what it's chewing/choking on?</a>". I received many helpful responses and by looking at the <a href="https://gist.github.com/2884737" rel="nofollow noreferrer">stack traces</a> I was able to determine that it was the same three scripts that were causing this deadlock issue over and over again.</p>
  
  <p>In each case the top line in 'the locking script' reads:</p>
  
  <pre><code>coldfusion.compiler.ClassReader.skipFully(ClassReader.java:79)
  </code></pre>
  
  <p>And <strong>all</strong> the other requests are clogged up behind it have the following line in their respective stack traces:</p>
  
  <pre><code>- waiting on &lt;0x9253305&gt; (a coldfusion.util.AbstractCache$Lock)
  </code></pre>
  
  <p>One thing that was bothering me, was <a href="https://stackoverflow.com/questions/10508534/why-doesnt-a-request-with-an-infinite-loop-get-killed-by-coldfusion-request-tim/10517019#10517019">why my request timeout was not being respected</a>; these scripts would just hang forever and never die. WTF, right? So I had to do it myself. So then when I kill 'the locking script', the others are released from limbo. At that point, if they are below the request timeout, they finish processing, and if they are over it (which most all of them usually are) then they simply proceed to timeout. But they won't timeout on their own and the requests just pile up until the active threads are used and the thread queue is full and everything seizes up. </p>
  
  <p>Manually killing these each time they're requested is obviously not a solution, so, as my wife is always reminding me, "debug, debug, debug". Using a conditional <code>&lt;cfabort&gt;</code> I stepped through and found that it was making it all the way through the Application.cfm, through my header.cfm, and right up to the <code>&lt;cfinclude&gt;</code> of the problem script. If I put the <code>&lt;cfabort&gt;</code> <em>inside</em> the problem script (even at the very top), it <strong>would not abort</strong> and the deadlock issue would occur. If I put it just before the include, the request <strong>would abort</strong> and the deadlock issue would be avoided. Bizarre.</p>
  
  <p>There's no code between these two places, right? Just before the include and just inside the include ought to be functionally equivalent, no? Probably not, because clearly <em>something</em> is going on in there.</p>
  
  <p>I'm not using any <code>&lt;cflock&gt;</code> tags. The locking that is happening is appears to be happening at the template cache level. This same behavior is observed regardless of whether the 'Trusted Cache', 'Cache Template In Request', or "Component Cache' options are checked in admin (in any combination of checked/unchecked). I've cleared the template cache and the component cache each more than once. I restarted the CF server over and over...all to no avail.</p>
  
  <p>During troubleshooting, I read this <a href="http://helpx.adobe.com/coldfusion/kb/deadlock-caused-compiler-cache-lock.html" rel="nofollow noreferrer">article describing a similar issue with a compiler cache lock in CF8</a> (8.0.1) along with instructions for applying a patch to fix it. But that isn't CF9...so obviously I can't apply their patch.</p>
  
  <p>What to do? Has anybody else encountered this issue? ...And have a solution?</p>