title: Does WebSphere 7 HTTPSession implementation contravene J2EE spec?
body: <p>Saw a problem recently where all 200 web container threads became hung, meaning none were available to service incoming requests and so the application froze.</p>
  
  <p>Here is a simple web app and JMeter test that I think demonstrates the cause of this problem. The web app consists of two classes, the following servlet:</p>
  
  <pre><code>public class SessionTestServlet extends HttpServlet {
  
      protected static final String SESSION_KEY = "session_key";
  
      protected void doGet(HttpServletRequest request, HttpServletResponse response)
              throws ServletException, IOException {
          // set data on session so the listener is invoked
          String sessionData = new String("Session data");
          request.getSession().setAttribute(SESSION_KEY, sessionData);
          PrintWriter writer = response.getWriter();                           
          writer.println("&lt;html&gt;&lt;body&gt;OK&lt;/body&gt;&lt;/html&gt;");                      
          writer.flush();
          writer.close();
      }
  }
  </code></pre>
  
  <p>and the following implementation of HttpSessionListener and HTTPSessionAttributeListener:</p>
  
  <pre><code>public class SessionTestListener implements 
          HttpSessionListener, HttpSessionAttributeListener {
  
      private static final ConcurrentMap&lt;String, HttpSession&gt; allSessions 
          = new ConcurrentHashMap&lt;String, HttpSession&gt;(); 
  
      public void attributeRemoved(HttpSessionBindingEvent hsbe) {}
  
      public void attributeAdded(HttpSessionBindingEvent hsbe) {
          System.out.println("Attribute added, " + hsbe.getName() 
              + "=" + hsbe.getValue());
  
          int count = 0;
          for (HttpSession session : allSessions.values()) {
              if (session.getAttribute(SessionTestServlet.SESSION_KEY) != null) {
                  count++;
              }
          }
          System.out.println(count + " of " + allSessions.size() 
              + " sessions have attribute set.");
      }
  
      public void attributeReplaced(HttpSessionBindingEvent hsbe) {}
  
      public void sessionCreated(HttpSessionEvent hse) {
          allSessions.put(hse.getSession().getId(), session);                              
      }
  
      public void sessionDestroyed(HttpSessionEvent hse) {
          allSessions.remove(hse.getSession().getId());
      }                
  }
  </code></pre>
  
  <p>The JMeter test has 100 requests hit the servlet every second:</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt;
  &lt;jmeterTestPlan version="1.2" properties="2.1"&gt;
    &lt;hashTree&gt;
      &lt;TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true"&gt;
        &lt;stringProp name="TestPlan.comments"&gt;&lt;/stringProp&gt;
        &lt;boolProp name="TestPlan.functional_mode"&gt;false&lt;/boolProp&gt;
        &lt;boolProp name="TestPlan.serialize_threadgroups"&gt;false&lt;/boolProp&gt;
        &lt;elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true"&gt;
          &lt;collectionProp name="Arguments.arguments"/&gt;
        &lt;/elementProp&gt;
        &lt;stringProp name="TestPlan.user_define_classpath"&gt;&lt;/stringProp&gt;
      &lt;/TestPlan&gt;
      &lt;hashTree&gt;
        &lt;ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true"&gt;
          &lt;stringProp name="ThreadGroup.on_sample_error"&gt;continue&lt;/stringProp&gt;
          &lt;elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true"&gt;
            &lt;boolProp name="LoopController.continue_forever"&gt;false&lt;/boolProp&gt;
            &lt;intProp name="LoopController.loops"&gt;-1&lt;/intProp&gt;
          &lt;/elementProp&gt;
          &lt;stringProp name="ThreadGroup.num_threads"&gt;100&lt;/stringProp&gt;
          &lt;stringProp name="ThreadGroup.ramp_time"&gt;1&lt;/stringProp&gt;
          &lt;longProp name="ThreadGroup.start_time"&gt;1327193422000&lt;/longProp&gt;
          &lt;longProp name="ThreadGroup.end_time"&gt;1327193422000&lt;/longProp&gt;
          &lt;boolProp name="ThreadGroup.scheduler"&gt;false&lt;/boolProp&gt;
          &lt;stringProp name="ThreadGroup.duration"&gt;&lt;/stringProp&gt;
          &lt;stringProp name="ThreadGroup.delay"&gt;&lt;/stringProp&gt;
        &lt;/ThreadGroup&gt;
        &lt;hashTree&gt;
          &lt;HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="HTTP Request" enabled="true"&gt;
            &lt;elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true"&gt;
              &lt;collectionProp name="Arguments.arguments"/&gt;
            &lt;/elementProp&gt;
            &lt;stringProp name="HTTPSampler.domain"&gt;localhost&lt;/stringProp&gt;
            &lt;stringProp name="HTTPSampler.port"&gt;9080&lt;/stringProp&gt;
            &lt;stringProp name="HTTPSampler.connect_timeout"&gt;&lt;/stringProp&gt;
            &lt;stringProp name="HTTPSampler.response_timeout"&gt;&lt;/stringProp&gt;
            &lt;stringProp name="HTTPSampler.protocol"&gt;http&lt;/stringProp&gt;
            &lt;stringProp name="HTTPSampler.contentEncoding"&gt;&lt;/stringProp&gt;
            &lt;stringProp name="HTTPSampler.path"&gt;/SESSION_TESTWeb/SessionTestServlet&lt;/stringProp&gt;
            &lt;stringProp name="HTTPSampler.method"&gt;GET&lt;/stringProp&gt;
            &lt;boolProp name="HTTPSampler.follow_redirects"&gt;true&lt;/boolProp&gt;
            &lt;boolProp name="HTTPSampler.auto_redirects"&gt;false&lt;/boolProp&gt;
            &lt;boolProp name="HTTPSampler.use_keepalive"&gt;true&lt;/boolProp&gt;
            &lt;boolProp name="HTTPSampler.DO_MULTIPART_POST"&gt;false&lt;/boolProp&gt;
            &lt;boolProp name="HTTPSampler.monitor"&gt;false&lt;/boolProp&gt;
            &lt;stringProp name="HTTPSampler.embedded_url_re"&gt;&lt;/stringProp&gt;
          &lt;/HTTPSamplerProxy&gt;
          &lt;hashTree&gt;
            &lt;ConstantTimer guiclass="ConstantTimerGui" testclass="ConstantTimer" testname="Constant Timer" enabled="true"&gt;
              &lt;stringProp name="ConstantTimer.delay"&gt;1000&lt;/stringProp&gt;
            &lt;/ConstantTimer&gt;
            &lt;hashTree/&gt;
          &lt;/hashTree&gt;
        &lt;/hashTree&gt;
        &lt;ResultCollector guiclass="ViewResultsFullVisualizer" testclass="ResultCollector" testname="View Results Tree" enabled="true"&gt;
          &lt;boolProp name="ResultCollector.error_logging"&gt;false&lt;/boolProp&gt;
          &lt;objProp&gt;
            &lt;name&gt;saveConfig&lt;/name&gt;
            &lt;value class="SampleSaveConfiguration"&gt;
              &lt;time&gt;true&lt;/time&gt;
              &lt;latency&gt;true&lt;/latency&gt;
              &lt;timestamp&gt;true&lt;/timestamp&gt;
              &lt;success&gt;true&lt;/success&gt;
              &lt;label&gt;true&lt;/label&gt;
              &lt;code&gt;true&lt;/code&gt;
              &lt;message&gt;true&lt;/message&gt;
              &lt;threadName&gt;true&lt;/threadName&gt;
              &lt;dataType&gt;true&lt;/dataType&gt;
              &lt;encoding&gt;false&lt;/encoding&gt;
              &lt;assertions&gt;true&lt;/assertions&gt;
              &lt;subresults&gt;true&lt;/subresults&gt;
              &lt;responseData&gt;false&lt;/responseData&gt;
              &lt;samplerData&gt;false&lt;/samplerData&gt;
              &lt;xml&gt;true&lt;/xml&gt;
              &lt;fieldNames&gt;false&lt;/fieldNames&gt;
              &lt;responseHeaders&gt;false&lt;/responseHeaders&gt;
              &lt;requestHeaders&gt;false&lt;/requestHeaders&gt;
              &lt;responseDataOnError&gt;false&lt;/responseDataOnError&gt;
              &lt;saveAssertionResultsFailureMessage&gt;false&lt;/saveAssertionResultsFailureMessage&gt;
              &lt;assertionsResultsToSave&gt;0&lt;/assertionsResultsToSave&gt;
              &lt;bytes&gt;true&lt;/bytes&gt;
            &lt;/value&gt;
          &lt;/objProp&gt;
          &lt;stringProp name="filename"&gt;&lt;/stringProp&gt;
        &lt;/ResultCollector&gt;
        &lt;hashTree/&gt;
        &lt;ResultCollector guiclass="SummaryReport" testclass="ResultCollector" testname="Summary Report" enabled="true"&gt;
          &lt;boolProp name="ResultCollector.error_logging"&gt;false&lt;/boolProp&gt;
          &lt;objProp&gt;
            &lt;name&gt;saveConfig&lt;/name&gt;
            &lt;value class="SampleSaveConfiguration"&gt;
              &lt;time&gt;true&lt;/time&gt;
              &lt;latency&gt;true&lt;/latency&gt;
              &lt;timestamp&gt;true&lt;/timestamp&gt;
              &lt;success&gt;true&lt;/success&gt;
              &lt;label&gt;true&lt;/label&gt;
              &lt;code&gt;true&lt;/code&gt;
              &lt;message&gt;true&lt;/message&gt;
              &lt;threadName&gt;true&lt;/threadName&gt;
              &lt;dataType&gt;true&lt;/dataType&gt;
              &lt;encoding&gt;false&lt;/encoding&gt;
              &lt;assertions&gt;true&lt;/assertions&gt;
              &lt;subresults&gt;true&lt;/subresults&gt;
              &lt;responseData&gt;false&lt;/responseData&gt;
              &lt;samplerData&gt;false&lt;/samplerData&gt;
              &lt;xml&gt;true&lt;/xml&gt;
              &lt;fieldNames&gt;false&lt;/fieldNames&gt;
              &lt;responseHeaders&gt;false&lt;/responseHeaders&gt;
              &lt;requestHeaders&gt;false&lt;/requestHeaders&gt;
              &lt;responseDataOnError&gt;false&lt;/responseDataOnError&gt;
              &lt;saveAssertionResultsFailureMessage&gt;false&lt;/saveAssertionResultsFailureMessage&gt;
              &lt;assertionsResultsToSave&gt;0&lt;/assertionsResultsToSave&gt;
              &lt;bytes&gt;true&lt;/bytes&gt;
            &lt;/value&gt;
          &lt;/objProp&gt;
          &lt;stringProp name="filename"&gt;&lt;/stringProp&gt;
        &lt;/ResultCollector&gt;
        &lt;hashTree/&gt;
      &lt;/hashTree&gt;
    &lt;/hashTree&gt;
  &lt;/jmeterTestPlan&gt;
  </code></pre>
  
  <p>When this test is run against the test web app deployed on WebSphere 7, the application quickly stops responding and a core dump shows this:</p>
  
  <pre><code>1LKDEADLOCK    Deadlock detected !!!
  NULL           ---------------------
  NULL           
  2LKDEADLOCKTHR  Thread "WebContainer : 2" (0x000000000225C600)
  3LKDEADLOCKWTR    is waiting for:
  4LKDEADLOCKMON      sys_mon_t:0x00000000151938C0 infl_mon_t: 0x0000000015193930:
  4LKDEADLOCKOBJ      com/ibm/ws/session/store/memory/MemorySession@00000000A38EA0C8/00000000A38EA0D4: 
  3LKDEADLOCKOWN    which is owned by:
  2LKDEADLOCKTHR  Thread "WebContainer : 1" (0x00000000021FB500)
  3LKDEADLOCKWTR    which is waiting for:
  4LKDEADLOCKMON      sys_mon_t:0x0000000015193820 infl_mon_t: 0x0000000015193890:
  4LKDEADLOCKOBJ      com/ibm/ws/session/store/memory/MemorySession@00000000A14E22C0/00000000A14E22CC: 
  3LKDEADLOCKOWN    which is owned by:
  2LKDEADLOCKTHR  Thread "WebContainer : 2" (0x000000000225C600)
  NULL 
  </code></pre>
  
  <p>It appears that when a thread (T1) executing the servlet's doGet() method calls setAttribute() on the instance of HttpSession implementation (S1), it locks on the monitor of S1. While holding that lock it goes into the iteration of allSessions inside the listener's attributeAdded() method and calls getAttribute(). It looks like inside getAttribute(), WebSphere locks on that instance's monitor (possibly because it is setting a lastUpdateTime field?). So, T1 will in turn lock the monitors of S1, S2, S3, S4, S5... all the while holding the lock on S1 from the setAttribute() call in the servlet.</p>
  
  <p>So if at the same time another thread (T2) is locking on another session (S2)'s monitor in the servlet and then goes into the loop in addAttribute(), the threads deadlock on the S1 and S2 monitors.</p>
  
  <p>I have been unable to find anything explicit in the J2EE specs about this but this part of the Servlet 2.4 spec implies that the container should not be synchronizing on instances of HttpSession implementations:</p>
  
  <blockquote>
    <p><strong>SRV.7.7.1 Threading Issues</strong></p>
    
    <p>Multiple servlets executing request threads may have active access to
    a single session object at the same time. The Developer has the
    responsibility for synchronizing access to session resources as
    appropriate.</p>
  </blockquote>
  
  <p>JBoss does not show any deadlocks when we run the test against it. So my questions are:</p>
  
  <ul>
  <li>Is my understanding correct?</li>
  <li>If so, is this a bug or contravention of the J2EE spec in WebSphere?</li>
  <li>If not, and it is valid behaviour that the developer should know about and code around, is this behaviour documented anywhere?</li>
  </ul>
  
  <p>Thanks</p>