title: Glassfish 3.12 goes to 100% CPU usage with two users
body: <p>Our Glassfish 3.12 went to 100% last night, which locked the users out of the system (there are only two). I did a dump of the Glassfish log (below) and there are ton of locks, although none of them used more than 1 second of CPU. 100% CPU usage seems to be a common problem with Glassfish 3.1.2, but I haven't found a clear solution yet. Any ideas welcome. </p>
  
  <pre><code>./asadmin generate-jvm-report --type=thread --user admin
  
  
  Thread Synchronization Statistics:
  
  -----------------------
  Number of times this thread was blocked (to enter/reenter a Monitor): 12
  Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0
  Total CPU time for this thread: 0 seconds 79,691,017 nanoseconds.
  User-level CPU time for this thread: 0 seconds 79,691,017 nanoseconds.
  Object Monitors currently held or requested by this thread: []
  Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
  --------------------------------------------------------------------------------
  Thread Execution Information:
  -----------------------
  Thread "Thread-12" thread-id: 24 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.atomic.AtomicInteger@4cc6afaa
           at: java.lang.Object.wait(Native Method)
           at: com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
           at: com.sun.grizzly.Controller.start(Controller.java:869)
           at: com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
           at: com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
           at: com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
           at: com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142)
           at: com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
  Thread Synchronization Statistics:
  -----------------------
  Number of times this thread was blocked (to enter/reenter a Monitor): 0
  Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 946
  Total CPU time for this thread: 0 seconds 17,073,572 nanoseconds.
  User-level CPU time for this thread: 0 seconds 17,073,572 nanoseconds.
  Object Monitors currently held or requested by this thread: []
  Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
  --------------------------------------------------------------------------------
  Thread Execution Information:
  -----------------------
  Thread "Grizzly-kernel-thread(1)" thread-id: 22 thread-state: RUNNABLE Running in native
           at: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
           at: sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
           at: sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
           at: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
           at: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
           at: com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513)
           at: com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190)
           at: com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
           at: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
           at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
           at: java.lang.Thread.run(Thread.java:745)
  Thread Synchronization Statistics:
  -----------------------
  Number of times this thread was blocked (to enter/reenter a Monitor): 1
  Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0
  Total CPU time for this thread: 0 seconds 55,218,828 nanoseconds.
  User-level CPU time for this thread: 0 seconds 55,218,828 nanoseconds.
  Object Monitors currently held or requested by this thread: []
  Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
  --------------------------------------------------------------------------------
  Thread Execution Information:
  -----------------------
  Thread "Grizzly-kernel-thread(1)" thread-id: 20 thread-state: RUNNABLE Running in native
           at: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
           at: sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
           at: sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
           at: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
           at: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
           at: com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513)
           at: com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190)
           at: com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
           at: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
           at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
           at: java.lang.Thread.run(Thread.java:745)
  Thread Synchronization Statistics:
  -----------------------
  Number of times this thread was blocked (to enter/reenter a Monitor): 3
  Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0
  Total CPU time for this thread: 0 seconds 60,298,518 nanoseconds.
  User-level CPU time for this thread: 0 seconds 60,298,518 nanoseconds.
  Object Monitors currently held or requested by this thread: []
  Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
  --------------------------------------------------------------------------------
  Thread Execution Information:
  -----------------------
  Thread "Thread-7" thread-id: 19 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.atomic.AtomicInteger@836e947
           at: java.lang.Object.wait(Native Method)
           at: com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
           at: com.sun.grizzly.Controller.start(Controller.java:869)
           at: com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
           at: com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
           at: com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
           at: com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142)
           at: com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
  Thread Synchronization Statistics:
  -----------------------
  Number of times this thread was blocked (to enter/reenter a Monitor): 0
  Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 946
  Total CPU time for this thread: 0 seconds 21,716,796 nanoseconds.
  User-level CPU time for this thread: 0 seconds 21,716,796 nanoseconds.
  Object Monitors currently held or requested by this thread: []
  Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
  --------------------------------------------------------------------------------
  Thread Execution Information:
  -----------------------
  Thread "Thread-4" thread-id: 16 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.atomic.AtomicInteger@4d0d3887
           at: java.lang.Object.wait(Native Method)
           at: com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
           at: com.sun.grizzly.Controller.start(Controller.java:869)
           at: com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
           at: com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
           at: com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
           at: com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142)
           at: com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
  Thread Synchronization Statistics:
  -----------------------
  Number of times this thread was blocked (to enter/reenter a Monitor): 2
  Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 946
  Total CPU time for this thread: 0 seconds 58,339,427 nanoseconds.
  User-level CPU time for this thread: 0 seconds 58,339,427 nanoseconds.
  Object Monitors currently held or requested by this thread: []
  Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
  --------------------------------------------------------------------------------
  Thread Execution Information:
  -----------------------
  Thread "Thread-2" thread-id: 14 thread-state: WAITING Waiting on lock: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1efa4e96
           at: sun.misc.Unsafe.park(Native Method)
           at: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
           at: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
           at: java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
           at: com.sun.enterprise.server.logging.GFFileHandler.log(GFFileHandler.java:665)
           at: com.sun.enterprise.server.logging.GFFileHandler$1.run(GFFileHandler.java:179)
  Thread Synchronization Statistics:
  -----------------------
  Number of times this thread was blocked (to enter/reenter a Monitor): 0
  Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 78
  Total CPU time for this thread: 0 seconds 68,044,526 nanoseconds.
  User-level CPU time for this thread: 0 seconds 68,044,526 nanoseconds.
  Object Monitors currently held or requested by this thread: []
  Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
  --------------------------------------------------------------------------------
  Thread Execution Information:
  -----------------------
  Thread "pool-1-thread-1" thread-id: 13 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@433d69bd
           at: sun.misc.Unsafe.park(Native Method)
           at: java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
           at: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
           at: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
           at: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
           at: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
           at: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
           at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
           at: java.lang.Thread.run(Thread.java:745)
  Thread Synchronization Statistics:
  -----------------------
  Number of times this thread was blocked (to enter/reenter a Monitor): 0
  Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 1,892
  Total CPU time for this thread: 0 seconds 210,072,724 nanoseconds.
  User-level CPU time for this thread: 0 seconds 210,072,724 nanoseconds.
  Object Monitors currently held or requested by this thread: []
  Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
  --------------------------------------------------------------------------------
  Thread Execution Information:
  -----------------------
  Thread "FelixStartLevel" thread-id: 12 thread-state: WAITING Waiting on lock: java.util.ArrayList@74e919c7
           at: java.lang.Object.wait(Native Method)
           at: java.lang.Object.wait(Object.java:503)
           at: org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:272)
           at: java.lang.Thread.run(Thread.java:745)
  Thread Synchronization Statistics:
  -----------------------
  Number of times this thread was blocked (to enter/reenter a Monitor): 42
  Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 42
  Total CPU time for this thread: 0 seconds 688,900,829 nanoseconds.
  User-level CPU time for this thread: 0 seconds 688,900,829 nanoseconds.
  Object Monitors currently held or requested by this thread: []
  Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
  --------------------------------------------------------------------------------
  Thread Execution Information:
  -----------------------
  Thread "FelixDispatchQueue" thread-id: 11 thread-state: WAITING Waiting on lock: java.util.ArrayList@739b93c7
           at: java.lang.Object.wait(Native Method)
           at: java.lang.Object.wait(Object.java:503)
           at: org.apache.felix.framework.util.EventDispatcher.run(EventDispatcher.java:1063)
           at: org.apache.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:54)
           at: org.apache.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:101)
           at: java.lang.Thread.run(Thread.java:745)
  Thread Synchronization Statistics:
  -----------------------
  Number of times this thread was blocked (to enter/reenter a Monitor): 232
  Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 217
  Total CPU time for this thread: 0 seconds 117,033,893 nanoseconds.
  User-level CPU time for this thread: 0 seconds 117,033,893 nanoseconds.
  Object Monitors currently held or requested by this thread: []
  Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
  --------------------------------------------------------------------------------
  Thread Execution Information:
  -----------------------
  Thread "Signal Dispatcher" thread-id: 4 thread-state: RUNNABLE
  Thread Synchronization Statistics:
  -----------------------
  Number of times this thread was blocked (to enter/reenter a Monitor): 0
  Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0
  Total CPU time for this thread: 0 seconds 70,212 nanoseconds.
  User-level CPU time for this thread: 0 seconds 70,212 nanoseconds.
  Object Monitors currently held or requested by this thread: []
  Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
  --------------------------------------------------------------------------------
  Thread Execution Information:
  -----------------------
  Thread "Finalizer" thread-id: 3 thread-state: WAITING Waiting on lock: java.lang.ref.ReferenceQueue$Lock@6b03291c
           at: java.lang.Object.wait(Native Method)
           at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
           at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
           at: java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
  Thread Synchronization Statistics:
  -----------------------
  Number of times this thread was blocked (to enter/reenter a Monitor): 20
  Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 19
  Total CPU time for this thread: 0 seconds 52,757,964 nanoseconds.
  User-level CPU time for this thread: 0 seconds 52,757,964 nanoseconds.
  Object Monitors currently held or requested by this thread: []
  Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
  --------------------------------------------------------------------------------
  Thread Execution Information:
  -----------------------
  Thread "Reference Handler" thread-id: 2 thread-state: WAITING Waiting on lock: java.lang.ref.Reference$Lock@1c802427
           at: java.lang.Object.wait(Native Method)
           at: java.lang.Object.wait(Object.java:503)
           at: java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
  Thread Synchronization Statistics:
  -----------------------
  Number of times this thread was blocked (to enter/reenter a Monitor): 18
  Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 18
  Total CPU time for this thread: 0 seconds 39,254,957 nanoseconds.
  User-level CPU time for this thread: 0 seconds 39,254,957 nanoseconds.
  Object Monitors currently held or requested by this thread: []
  Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
  No deadlock found
  </code></pre>