text:
Jon Masamitsu's Weblog Garbage Collection - Let the VM do it. https://blogs.oracle.com/jonthecollector/feed/entries/atom 2013-06-28T22:23:35+00:00 Apache Roller https://blogs.oracle.com/jonthecollector/entry/the_unspoken_the_why_of The Unspoken - The Why of GC Ergonomics jonthecollector 2013-06-28T22:23:35+00:00 2013-06-28T22:23:35+00:00 Do you use GC ergonomics, -XX:+UseAdaptiveSizePolicy, with the UseParallelGC collector? The jist of GC ergonomics for that collector is that it tries to grow or shrink the heap to meet a specified goal. The goals that you can choose are maximum pause time and/or throughput. Don't get too excited there. I'm speaking about UseParallelGC (the throughput collector) so there are definite limits to what pause goals can be achieved. When you say out loud "I don't care about pause times, give me the best throughput I can get" and then say to yourself "Well, maybe 10 seconds really is too long", then think about a pause time goal. By default there is no pause time goal and the throughput goal is high (98% of the time doing application work and 2% of the time doing GC work). You can get more details on this in my very first blog.
  GC ergonomics
 The UseG1GC has its own version of GC ergonomics, but I'll be talking only about the UseParallelGC version.
 If you use this option and wanted to know what it (GC ergonomics) was thinking, try
 -XX:AdaptiveSizePolicyOutputInterval=1
 This will print out information every i-th GC (above i is 1) about what the GC ergonomics to trying to do. For example,
 UseAdaptiveSizePolicy actions to meet *** throughput goal *** GC overhead (%) Young generation: 16.10 (attempted to grow) Tenured generation: 4.67 (attempted to grow) Tenuring threshold: (attempted to decrease to balance GC costs) = 1

 GC ergonomics tries to meet (in order) <nl>
 Pause time goal
 Throughput goal
 Minimum footprint </nl>
 The first line says that it's trying to meet the throughput goal.
 UseAdaptiveSizePolicy actions to meet *** throughput goal ***

 This run has the default pause time goal (i.e., no pause time goal) so it is trying to reach a 98% throughput. The lines
 Young generation: 16.10 (attempted to grow) Tenured generation: 4.67 (attempted to grow)

 say that we're currently spending about 16% of the time doing young GC's and about 5% of the time doing full GC's. These percentages are a decaying, weighted average (earlier contributions to the average are given less weight). The source code is available as part of the OpenJDK so you can take a look at it if you want the exact definition. GC ergonomics is trying to increase the throughput by growing the heap (so says the "attempted to grow").
 The last line
 Tenuring threshold: (attempted to decrease to balance GC costs) = 1

 says that the ergonomics is trying to balance the GC times between young GC's and full GC's by decreasing the tenuring threshold. During a young collection the younger objects are copied to the survivor spaces while the older objects are copied to the tenured generation. Younger and older are defined by the tenuring threshold. If the tenuring threshold hold is 4, an object that has survived fewer than 4 young collections (and has remained in the young generation by being copied to the part of the young generation called a survivor space) it is younger and copied again to a survivor space. If it has survived 4 or more young collections, it is older and gets copied to the tenured generation. A lower tenuring threshold moves objects more eagerly to the tenured generation and, conversely a higher tenuring threshold keeps copying objects between survivor spaces longer. The tenuring threshold varies dynamically with the UseParallelGC collector. That is different than our other collectors which have a static tenuring threshold. GC ergonomics tries to balance the amount of work done by the young GC's and the full GC's by varying the tenuring threshold. Want more work done in the young GC's? Keep objects longer in the survivor spaces by increasing the tenuring threshold.
 This is an example of the output when GC ergonomics is trying to achieve a pause time goal
 UseAdaptiveSizePolicy actions to meet *** pause time goal *** GC overhead (%) Young generation: 20.74 (no change) Tenured generation: 31.70 (attempted to shrink)

 The pause goal was set at 50 millisecs and the last GC was
 0.415: [Full GC (Ergonomics) [PSYoungGen: 2048K->0K(26624K)] [ParOldGen: 26095K->9711K(28992K)] 28143K->9711K(55616K), [Metaspace: 1719K->1719K(2473K/6528K)], 0.0758940 secs] [Times: user=0.28 sys=0.00, real=0.08 secs]

 The full collection took about 76 millisecs so GC ergonomics wants to shrink the tenured generation to reduce that pause time. The previous young GC was
 0.346: [GC (Allocation Failure) [PSYoungGen: 26624K->2048K(26624K)] 40547K->22223K(56768K), 0.0136501 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]

 so the pause time there was about 14 millisecs so no changes are needed.
 If trying to meet a pause time goal, the generations are typically shrunk. With a pause time goal in play, watch the GC overhead numbers and you will usually see the cost of setting a pause time goal (i.e., throughput goes down). If the pause goal is too low, you won't achieve your pause time goal and you will spend all your time doing GC.
 GC ergonomics is meant to be simple because it is meant to be used by anyone. It was not meant to be mysterious and so this output was added. If you don't like what GC ergonomics is doing, you can turn it off with -XX:-UseAdaptiveSizePolicy, but be pre-warned that you have to manage the size of the generations explicitly. If UseAdaptiveSizePolicy is turned off, the heap does not grow. The size of the heap (and the generations) at the start of execution is always the size of the heap. I don't like that and tried to fix it once (with some help from an OpenJDK contributor) but it unfortunately never made it out the door. I still have hope though.
 Just a side note. With the default throughput goal of 98% the heap often grows to it's maximum value and stays there. Definitely reduce the throughput goal if footprint is important. Start with -XX:GCTimeRatio=4 for a more modest throughput goal (%20 of the time spent in GC). A higher value means a smaller amount of time in GC (as the throughput goal). https://blogs.oracle.com/jonthecollector/entry/the_unspoken_gc_times The Unspoken - Application Times jonthecollector 2013-04-18T20:22:33+00:00 2013-04-18T20:22:33+00:00 Sometimes it is interesting to know how long your application runs between garbage collections. That can be calculated from the GC logs but a convenient way to see that information is through the command line flags -XX:+PrintGCApplicationStoppedTime and -XX:+PrintGCApplicationConcurrentTime. Adding these to your command line produces output such as this.

 Application time: 0.2863875 seconds Total time for which application threads were stopped: 0.0225087 seconds Application time: 0.1476791 seconds Total time for which application threads were stopped: 0.0255697 seconds

 The application ran (reported in the first line) for about 287 milliseconds and then was stopped for about 22 milliseconds (reported in the second line).
 The flags can be used separately or together.
 Add -XX:+PrintGCTimeStamps -XX:+PrintGCDetails and you get

 Application time: 0.1325032 seconds 20.149: [GC (Allocation Failure) 20.149: [ParNew: 78656K->8704K(78656K), 0.02215 98 secs] 225454K->158894K(253440K), 0.0222106 secs] [Times: user=0.13 sys=0.00, real=0.03 secs] Total time for which application threads were stopped: 0.0224188 seconds

 When the flags was first implemented the "stopped" time really was the GC stopped time but it was later changed to include the complete stopped time. There is other work that may be done during a "safepoint" (period during which the application is halted and the VM can do some work without the application changing). All that is now included in the "stopped" time. In the example above I would say that GC was the only thing happening during the safepoint (GC time being 0.0222106 secs and the complete time being nearly the same at 0.0224188 secs). I can't speak authoritatively about what else can happen because I just don't know. But to give one example I've heard about is compiled code deoptimization. When the code for a JIT'ed method needs to be thrown away (typically because some assumption made during the compilation has been violated), the VM has to switch from the compiled code to interpreting that method again. That switch is done at a safepoint (but don't quote me on that because its not my area). https://blogs.oracle.com/jonthecollector/entry/the_unspoken_cms_and_printgcdetails The Unspoken - CMS and PrintGCDetails jonthecollector 2013-02-13T06:07:57+00:00 2013-02-13T06:07:57+00:00 What follows is an example of the GC logging output with CMS (UseConcMarkSweepGC) and PrintGCDetails plus some explanation of the output.
 The "CMS-initial-mark" indicates the start of a CMS concurrent collection.
 [GC [1 CMS-initial-mark: 463236K(515960K)] 464178K(522488K), 0.0018216 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
 463236K above is the space occupied by objects in the old (CMS) generation at the start of the collection. Not all those objects are necessarily alive.
 515960K is the total size of the old (CMS) generation. This value changes if the generation grows or shrinks.
 464178K is the sum of the space occupied by objects in the young generation and the old (CMS) generation.
 522488K is the total size of the heap (young generation plus old (CMS) generation)
 0.0018216 secs is the duration of the initial mark pause. The initial mark is a stop-the-world phase.
 After the initial mark completes the CMS concurrent mark starts. The concurrent mark phase is a concurrent phase and can be interrupted by young generation collections. In this case the ParNew (UseParNewGC) is being used to collect the young generation. When a ParNew collection is ready to start, it raises a flag and the CMS collector yields execution to ParNew and waits for ParNew to finish before resuming.
 [GC[ParNew: 6528K->702K(6528K), 0.0130227 secs] 469764K->465500K(522488K), 0.0130578 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
 6528K is the space in the young generation occupied by objects at the start of the ParNew collection. Not all those objects are necessarily alive.
 702K is the space occupied by live objects at the end of the ParNew collection.
 6528K is the total space in the young generation.
 0.0130227 is the pause duration for the ParNew collection.
 469764K is the space occupied by objects in the young generation and the old (CMS) generation before the collection starts.
 465500K is the space occupied by live objects in the young generation and all objects in the old (CMS) generation. For a ParNew collection, only the liveness of the objects in the young generation is known so the objects in the old (CMS) generation may be live or dead.
 522488K is the total space in the heap.
 [Times: user=0.05 sys=0.00, real=0.01 secs] is like the output of time(1) command. The ratio user / real give you an approximation for the speed up you're getting from the parallel execution of the ParNew collection. The sys time can be an indicator of system activity that is slowing down the collection. For example if paging is occurring, sys will be high.
 [GC[ParNew: 6526K->702K(6528K), 0.0136447 secs] 471324K->467077K(522488K), 0.0136804 secs] [Times: user=0.04 sys=0.01, real=0.01 secs]
 [GC[ParNew: 6526K->702K(6528K), 0.0161873 secs] 472901K->468830K(522488K), 0.0162411 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
 [GC[ParNew: 6526K->702K(6528K), 0.0152107 secs] 474654K->470569K(522488K), 0.0152543 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
 ...
 [GC[ParNew: 6526K->702K(6528K), 0.0144212 secs] 481073K->476809K(522488K), 0.0144719 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
 This is the completion of the concurrent marking phase. After this point the precleaning starts.
 [CMS-concurrent-mark: 1.039/1.154 secs] [Times: user=2.32 sys=0.02, real=1.15 secs]
 The 1.039 is the elapsed time for the concurrent marking. The 1.154 is the wall clock time.
 The "Times" output is less meaningful because it is measured from the start of the concurrent marking and includes more than just the work done for the concurrent marking.
 This is the end of the precleaning phase.
 [CMS-concurrent-preclean: 0.006/0.007 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
 The format of the precleaning phase output is analogous to that of the concurrent marking phase.
 [GC[ParNew: 6526K->702K(6528K), 0.0141896 secs] 482633K->478368K(522488K), 0.0142292 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
 [GC[ParNew: 6526K->702K(6528K), 0.0162142 secs] 484192K->480082K(522488K), 0.0162509 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
 The remark phase is scheduled so that it does not occur back-to-back with a ParNew so as not to appear to be a pause that is the sum of the ParNew and the remark pause. A second precleaning phase is started and is aborted when the remark phase is ready to start. Aborting this second precleaning phase is the expected behavior. That it was aborted is not an indication of an error. Since the remark phase is waiting, why not preclean but don't delay the remark for the sake of precleaning.
 [CMS-concurrent-abortable-preclean: 0.022/0.175 secs] [Times: user=0.36 sys=0.00, real=0.17 secs]
 This is the remark phase.
 [GC[YG occupancy: 820 K (6528 K)][Rescan (parallel) , 0.0024157 secs][weak refs processing, 0.0000143 secs][scrub string table, 0.0000258 secs] [1 CMS-remark: 479379K(515960K)] 480200K(522488K), 0.0025249 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
 [YG occupancy: 820 K (6528 K)] shows that at the start of the remark the occupancy (sum of the total of the allocated objects in the young generation is 820K out of a total of 6528K. The length of the remark pause depends to some degree on the occupancy of the young generation so we print it out.
 The "Rescan" completes the marking of live objects while the application is stopped. In this case the rescan was done in parallel and took 0.0024157 secs.
 "weak refs processing" and "scrub string table" are tasks done during the remark. Those tasks took 0.0000143 secs and 0.0000258 secs, respectively. If those numbers dominate the remark pause time, they can explain unexpectedly large pauses. Not that they cannot legitimately be large. Just that generally they are not and when they are, take note. If the weak refs processing is dominate, you might be able to cut that time down by using parallel reference processing (-XX:+ParallelRefProcEnabled). No comment on the case when scrub string table is dominant. I've never had to deal with it.
 The concurrent sweeping phase starts at the end of the remark.
 [GC[ParNew: 6526K->702K(6528K), 0.0133250 secs] 441217K->437145K(522488K), 0.0133739 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
 [GC[ParNew: 6526K->702K(6528K), 0.0125530 secs] 407061K->402841K(522488K), 0.0125880 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
 ...
 [GC[ParNew: 6526K->702K(6528K), 0.0121435 secs] 330503K->326239K(522488K), 0.0121996 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
 The sweep phase ends here.
 [CMS-concurrent-sweep: 0.756/0.833 secs] [Times: user=1.68 sys=0.01, real=0.83 secs]
 The format above is analogous to that of the concurrent marking.
 And lastly the reset phase.
 [CMS-concurrent-reset: 0.009/0.009 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
 It is not expected that another concurrent CMS collection start before several ParNew collections have been done. If another CMS collection starts immediately, check how full the old (CMS) generation is. If the old (CMS) generation is close to being full immediately after the end of a collection, the heap might be too small.
 I took this log with an early build of JDK8
 java version "1.8.0-ea" Java(TM) SE Runtime Environment (build 1.8.0-ea-b73) Java HotSpot(TM) Server VM (build 25.0-b14, mixed mode)
 and used the flags
 -server -XX:+UseConcMarkSweepGC -XX:NewRatio=8 -XX:-PrintGCCause -XX:ParallelGCThreads=4 -Xmx1g -XX:+PrintGCDetails
 I usually also use -XX+PrintTimeStampsps to get time stamps in the logs and use -XX:+PrintGCDateStamps if I want to correlate the GC output to application events. https://blogs.oracle.com/jonthecollector/entry/hey_joe_phases_of_cms The Unspoken - Phases of CMS jonthecollector 2013-02-11T22:35:40+00:00 2013-02-11T22:35:40+00:00 A CMS (UseConcMarkSweepGC) garbage collection has several phase. As explain here some of those phases are stop-the-world and some are concurrent. CMS (-XX:+UseConcMarkSweepGC) or the concurrent mark sweep GC could have been called the mostly concurrent mark sweep GC and here's why.
 These are the phases of a CMS concurrent collection.

 1. Initial mark. This is the the start of CMS collection. This is a stop-the-world phase. All the applications threads are stopped and CMS scans the application threads (e.g. the thread stacks) for object references.
 2. Concurrent mark. This is a concurrent phase. CMS restarts all the applications threads and then starting from the objects found in 1), it finds all the other live objects. Almost (see the remark phase).
 3. Precleaning phase. This is a concurrent phase. This phase is an optimization and you don't really need to understand what it does so feel free to skip to 4. While CMS is doing concurrent marking (2.), the application threads are running and they can be changing the objects they are using. CMS needs to find any of these changes and ultimately does that during the remark phase. However, CMS would like to discovery this concurrently and so has the precleaning phase. CMS has a way of recording when an application thread makes a change to the objects that it is using. Precleaning looks at the records of these changes and marks live objects as live. For example if thread AA has a reference to an object XX and passes that reference to thread BB, then CMS needs to understand that BB is keeping XX alive now even if AA no longer is.
 4. The remark phase. The remark phase is a stop-the-world. CMS cannot correctly determine which objects are alive (mark them live), if the application is running concurrently and keeps changing what is live. So CMS stops all the application threads during the remark phase so that it can catch up with the changes the application has been making.
 5. The sweep phase. This is a concurrent phase. CMS looks at all the objects and adds newly dead objects to its freelists. CMS does freelist allocation and it is during the sweep phase that those freelists get repopulated.
 6. The reset phase. This is a concurrent phase. CMS is cleaning up its state so that it is ready for the next collection.
 That's the important part of this blog. I'm going to ramble now.
 I recently talked to some users who, of course, knew how their application worked but who also didn't know so much about how garbage collection worked. Why should they. That's my job. During our chat I came to appreciate much more the fact that there were things that I habitually left unspoken. So, here's one of the unspoken truths that I forget to explain: The phases of CMS. https://blogs.oracle.com/jonthecollector/entry/really_icms_really Really? iCMS? Really? jonthecollector 2011-11-01T10:23:27+00:00 2011-11-01T10:35:57+00:00 When I use the term iCMS, I'm referring to the incremental mode of CMS (-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode). This is a mode of CMS where the concurrent phases (concurrent marking and concurrent sweeping) are run in short increments (does some work, yields and sleeps for a while, then runs again). It was implemented in CMS primarily for use on platforms with a single hardware thread. The concurrent phases are typically long (think seconds and not milliseconds). If CMS hogged the single hardware thread for several seconds, the application would not execute during those several seconds and would in effect experience a stop-the-world pause. With iCMS the application makes progress during concurrent phases. That of course is good. The down side is that there is more overhead to iCMS (e.g., all that stopping and starting), more floating garbage (objects that die during a CMS collection that we can't tell have died until the next collection) and more flags to tune (just what you wanted, right). Also it's just more complex (more things to go wrong). We put quite a bit of effort into teaching iCMS how to do the work in pieces and still finish the collection before the heap fill up, but it's not perfect. Are you using iCMS? Does your platform only have one hardware thread? Ok, maybe two hardware threads is reason enough if you don't want half your cycles eaten by CMS during a long concurrent phases. But, otherwise, I'd suggest you try without the incremental mode. There are exceptions to every rule, but I think you'll be happier. Really. https://blogs.oracle.com/jonthecollector/entry/our_collectors Our Collectors jonthecollector 2008-02-01T15:51:15+00:00 2008-02-04T19:48:29+00:00 I drew this diagram on a white board for some customers recently. They seemed to like it (or were just being very polite) so I thought I redraw it for your amusement.

 Each blue box represents a collector that is used to collect a generation. The young generation is collected by the blue boxes in the yellow region and the tenured generation is collected by the blue boxes in the gray region.
 <nl>
 "Serial" is a stop-the-world, copying collector which uses a single GC thread.
 "ParNew" is a stop-the-world, copying collector which uses multiple GC threads. It differs from "Parallel Scavenge" in that it has enhancements that make it usable with CMS. For example, "ParNew" does the synchronization needed so that it can run during the concurrent phases of CMS.
 "Parallel Scavenge" is a stop-the-world, copying collector which uses multiple GC threads.
 "Serial Old" is a stop-the-world, mark-sweep-compact collector that uses a single GC thread.
 "CMS" is a mostly concurrent, low-pause collector.
 "Parallel Old" is a compacting collector that uses multiple GC threads. <nl/>
 Using the -XX flags for our collectors for jdk6,
 <nl>
 UseSerialGC is "Serial" + "Serial Old"
 UseParNewGC is "ParNew" + "Serial Old"
 UseConcMarkSweepGC is "ParNew" + "CMS" + "Serial Old". "CMS" is used most of the time to collect the tenured generation. "Serial Old" is used when a concurrent mode failure occurs.
 UseParallelGC is "Parallel Scavenge" + "Serial Old"
 UseParallelOldGC is "Parallel Scavenge" + "Parallel Old" <nl/>
 FAQ
 1) UseParNew and UseParallelGC both collect the young generation using multiple GC threads. Which is faster?
 There's no one correct answer for this questions. Mostly they perform equally well, but I've seen one do better than the other in different situations. If you want to use GC ergonomics, it is only supported by UseParallelGC (and UseParallelOldGC) so that's what you'll have to use.
 2) Why doesn't "ParNew" and "Parallel Old" work together?
 "ParNew" is written in a style where each generation being collected offers certain interfaces for its collection. For example, "ParNew" (and "Serial") implements space_iterate() which will apply an operation to every object in the young generation. When collecting the tenured generation with either "CMS" or "Serial Old", the GC can use space_iterate() to do some work on the objects in the young generation. This makes the mix-and-match of collectors work but adds some burden to the maintenance of the collectors and to the addition of new collectors. And the burden seems to be quadratic in the number of collectors. Alternatively, "Parallel Scavenge" (at least with its initial implementation before "Parallel Old") always knew how the tenured generation was being collected and could call directly into the code in the "Serial Old" collector. "Parallel Old" is not written in the "ParNew" style so matching it with "ParNew" doesn't just happen without significant work. By the way, we would like to match "Parallel Scavenge" only with "Parallel Old" eventually and clean up any of the ad hoc code needed for "Parallel Scavenge" to work with both.
 Please don't think too much about the examples I used above. They are admittedly contrived and not worth your time.
 3) How do I use "CMS" with "Serial"?
 -XX:+UseConcMarkSweepGC -XX:-UseParNewGC. Don't use -XX:+UseConcMarkSweepGC and -XX:+UseSerialGC. Although that's seems like a logical combination, it will result in a message saying something about conflicting collector combinations and the JVM won't start. Sorry about that. Our bad.
 4) Is the blue box with the "?" a typo?
 That box represents the new garbage collector that we're currently developing called Garbage First or G1 for short. G1 will provide <nl>
 More predictable GC pauses
 Better GC ergonomics
 Low pauses without fragmentation
 Parallelism and concurrency in collections
 Better heap utilization <nl/>
 G1 straddles the young generation - tenured generation boundary because it is a generational collector only in the logical sense. G1 divides the heap into regions and during a GC can collect a subset of the regions. It is logically generational because it dynamically selects a set of regions to act as a young generation which will then be collected at the next GC (as the young generation would be).
 The user can specify a goal for the pauses and G1 will do an estimate (based on past collections) of how many regions can be collected in that time (the pause goal). That set of regions is called a collection set and G1 will collect it during the next GC.
 G1 can choose the regions with the most garbage to collect first (Garbage First, get it?) so gets the biggest bang for the collection buck.
 G1 compacts so fragmentation is much less a problem. Why is it a problem at all? There can be internal fragmentation due to partially filled regions.
 The heap is not statically divided into a young generation and a tenured generation so the problem of an imbalance in their sizes is not there.
 Along with a pause time goal the user can specify a goal on the fraction of time that can be spent on GC during some period (e.g., during the next 100 seconds don't spend more than 10 seconds collecting). For such goals (10 seconds of GC in a 100 second period) G1 can choose a collection set that it expects it can collect in 10 seconds and schedules the collection 90 seconds (or more) from the previous collection. You can see how an evil user could specify 0 collection time in the next century so again, this is just a goal, not a promise.
 If G1 works out as we expect, it will become our low-pause collector in place of "ParNew" + "CMS". And if you're about to ask when will it be ready, please don't be offended by my dead silence. It's the highest priority project for our team, but it is software development so there are the usual unknowns. It will be out by JDK7. The sooner the better as far as we're concerned.
 Updated February 4.  Yes, I can edit an already posted blog. Here's a reference to the G1 paper if you have ACM portal access.
 http://portal.acm.org/citation.cfm?id=1029879 https://blogs.oracle.com/jonthecollector/entry/gc_errata_2 GC Errata 2 jonthecollector 2008-01-10T10:28:07+00:00 2008-01-10T18:28:07+00:00 I know that none of you are still making the transition from jdk1.4.2 to jdk5 (aka jdk1.5.0, yah, we changed the numbering) but if you were, I'd want you to know that you might need a larger permanent generation. In jdk5 class data sharing was implemented (the -Xshare option, see the "java -X" output) when the client VM (-client) is being used. The implementation of class data sharing split some of the information about classes into two parts (the part that can be shared and the part that is specific to an execution of the JVM). The space in the permanent generation for classes that can be shared increased by 10-20%. That doesn't mean that you need a permanent generation that is 10-20% larger! Only some of the classes are affected and only some of the data structures for those classes are affected. But if you're running a tight ship in terms of the size of permanent generation, you might be affected by this. Of course, none of you are still making the transition to jdk5.
 Some users have reported occasional long pauses which occur at the time of a GC but cannot be accounted for by GC times. Such long pauses may be a symptom of a known latency in the use of (Linux's) mprotect() which is used by the JVM while steering application threads to so-called safepoints. A known workaround at the JVM level to step around the Linux mprotect() issue is the use of JVM option -XX:+UseMembar.
 The real fix, unfortunately, lies in the Linux kernel; see http://bugzilla.kernel.org/show_bug.cgi?id=5493. You might wish to pursue this with your Linux support engineer and see if a patch is available for the problem.
 To avoid running into this bug, make sure there is plenty of physical memory free in your system so that the Linux pager does not start evicting pages for any process from memory to swap.
 If you're seeing large variations in your minor GC pauses and are using UseParNewGC (which is on by default with the low pause collector), you might be running into 6433335. A large variation is a factor of a 100. This is a bug that has to do with large objects (or large free blocks) in the heap. If you turn off UseParNewGC and see larger pauses (by a multiplier that is on the order of the number of cpu's) but more regular pauses, then 6433335 is a likely candidate. It's been fixed in jdk 1.4.2 update 14, jdk 5 update 10 and jdk 6 update 1.
 I've heard of more than one report recently of crashes with the low-pause collector that were caused by bug
 6558100 - CMS crash following parallel work queue overflow
 This bug is fixed in 1.4.2u18, 5.0u14 and 6u4. You can workaround the bug with the flag -XX:-ParallelRemarkEnabled. You could also run into this bug if you explicitly enable ParallelRefProcEnabled, so if you include -XX:+ParallelRefProcEnabled, remove it. ParallelRefProcEnabled is off by default so if you don't explicitly turn it on, don't worry about it.
 Starting in jdk6 the biased locking optimization is "on" by default (command line option UseBiasedLocking). This optimization reduces the cost of uncontended locks by treating the thread that owns the lock preferentially. It's a nice optimization but could increase the memory usage of the JVM.
 If you move from jdk5 to jdk6 and start seeing messages such as
 java.lang.OutOfMemoryError: requested <NNN> bytes for GrET in C
 you could try turning off biased locking and see if it helps. Improvements in biased locking were made in 6u1 to make this much less of a problem. https://blogs.oracle.com/jonthecollector/entry/did_you_know Did You Know ... jonthecollector 2007-10-26T11:45:21+00:00 2007-10-26T18:45:21+00:00 These are a few esoteric factoids that I never expected users to need, but which have actually come up recently. Most of the text is just background information. If you already recognize the command line flags that I've bold'ed, you probably already know more than is good for you.
  ParallelCMSThreads
 The low-pause collector (UseConcMarkSweepGC) does parts of the collection of the tenured generation concurrently with the execution of the application (i.e., not during a stop-the-world). There are principally two concurrent phases of the collection: the concurrent marking phase and the concurrent sweeping phase. In JDK 6 the concurrent marking phase can use more than 1 GC threads (uses parallelism as well as concurrency). This use of the parallelism is controlled by the command line flag CMSConcurrentMTEnabled. The number of threads used during a concurrent marking phase is ParallelCMSThreads. If it is not set on the command line it is calculated as
 (ParallelGCThreads + 3)/4)
 where ParallelGCThreads is the command line flag for setting the number of GC threads to be used in a stop-the-world parallel collection.
 Where did this number come from? We added parallelism to the concurrent marking phase because we observed that a single GC thread doing concurrent marking could be overwhelmed by the allocations of many applications threads (i.e., while the concurrent marking was happening, lots of applications threads doing allocations could exhaust the heap before the concurrent marking finished). We could see this with a fewer application threads allocating at a furious rate or many application threads allocating at a more modest rate, but whatever the application we would often seen the concurrent marking thread overwhelmed on platforms with 8 or more processors.
 The above policy provides a second concurrent marking threads at ParallelGCThreads=5 and approaches a fourth of ParallelGCThread at the higher processor numbers. Because we still do have the added overheard of parallelism 2 concurrent marking threads provide only a small boost in concurrent marking over a single concurrent marking thread. We expect that to still be adequate up to ParallelGCThreads=8. At ParallelGCThreads=9 we get a third concurrent marking thread and that's when we expect to need it.
  CMSMaxAbortablePrecleanTime
 Our low-pause collector (UseConcMarkSweepGC) which we are usually careful to call our mostly concurrent collector has several phases, two of which are stop-the-world (STW) phases.
 <nl>
 STW initial mark
 Concurrent marking
 Concurrent precleaning
 STW remark
 Concurrent sweeping
 Concurrent reset </nl>
 The first STW pause is used to find all the references to objects in the application (i.e., object references on thread stacks and in registers). After this first STW pause is the concurrent marking phase during which the application threads runs while GC is doing additional marking to determine the liveness of objects. After the concurrent marking phase there is a concurrent preclean phase (described more below) and then the second STW pause which is called the remark phase. The remark phase is a catch-up phase in which the GC figures out all the changes that the application threads have made during the previous concurrent phases. The remark phase is the longer of these two pauses. It is also typically the longest of any of the STW pauses (including the minor collection pauses). Because it is typically the longest pause we like to use parallelism where ever we can in the remark phase.
 Part of the work in the remark phase involves rescanning objects that have been changed by an application thread (i.e., looking at the object A to see if A has been changed by the application thread so that A now references another object B and B was not previously marked as live). This includes objects in the young generation and here we come to the point of these ramblings. Rescanning the young generation in parallel requires that we divide the young generation into chunks so that we can give chunks out to the parallel GC threads doing the rescanning. A chunk needs to begin on the start of an object and in general we don't have a fast way to find the starts of objects in the young generation.
 Given an arbitrary location in the young generation we are likely in the middle of an object, don't know what kind of object it is, and don't know how far we are from the start of the object. We know that the first object in the young generation starts at the beginning of the young generation and so we could start at the beginning and walk from object to object to do the chunking but that would be expensive. Instead we piggy-back the chunking of the young generation on another concurrent phase, the precleaning phase.
 During the concurrent marking phase the applications threads are running and changing objects so that we don't have an exact picture of what's alive and what's not. We ultimately fix this up in the remark phase as described above (the object-A-gets-changed-to-point-to-object-B example). But we would like to do as much of the collection as we can concurrently so we have the concurrent precleaning phase. The precleaning phase does work similar to parts of the remark phase but does it concurrently. The details are not needed for this story so let me just say that there is a concurrent precleaning phase. During the latter part of the concurrent precleaning phase the the young generation "top" (the next location to be allocated in the young generation and so at an object start) is sampled at likely intervals and is saved as the start of a chunk. "Likely intervals" just means that we want to create chunks that are not too small and not too large so as to get good load balancing during the parallel remark.
 Ok, so here's the punch line for all this. When we're doing the precleaning we do the sampling of the young generation top for a fixed amount of time before starting the remark. That fixed amount of time is CMSMaxAbortablePrecleanTime and its default value is 5 seconds. The best situation is to have a minor collection happen during the sampling. When that happens the sampling is done over the entire region in the young generation from its start to its final top. If a minor collection is not done during that 5 seconds then the region below the first sample is 1 chunk and it might be the majority of the young generation. Such a chunking doesn't spread the work out evenly to the GC threads so reduces the effective parallelism.
 If the time between your minor collections is greater than 5 seconds and you're using parallel remark with the low-pause collector (which you are by default), you might not be getting parallel remarking after all. A symptom of this problem is significant variations in your remark pauses. This is not the only cause of variation in remark pauses but take a look at the times between your minor collections and if they are, say, greater than 3-4 seconds, you might need to up CMSMaxAbortablePrecleanTime so that you get a minor collection during the sampling.
 And finally, why not just have the remark phase wait for a minor collection so that we get effective chunking? Waiting is often a bad thing to do. While waiting the application is running and changing objects and allocating new objects. The former makes more work for the remark phase when it happens and the latter could cause an out-of-memory before the GC can finish the collection. There is an option CMSScavengeBeforeRemark which is off by default. If turned on, it will cause a minor collection to occur just before the remark. That's good because it will reduce the remark pause. That's bad because there is a minor collection pause followed immediately by the remark pause which looks like 1 big fat pause.l
  DontCompileHugeMethods
 We got a complaint recently from a user who said that all his GC pauses were too long. I, of course, take such a statement with a grain of salt, but I still try to go forward with an open mind. And this time the user was right, his GC pauses were way too long. So we started asking the usual questions about anything unusual about the application's allocation pattern. Mostly that boils down to asking about very large objects or large arrays of objects. I'm talking GB size objects here. But, no, there were nothing like that. The user was very helpful in terms of trying experiments with his application, but we weren't getting anywhere until the user came back and said that he had commented out part of his code and the GC's got much smaller. Hmmm. Curiouser and curiouser. Not only that, but the code that was commented out was not being executed. At this point the strain on my brain began to be too much and I lost consciousness. Fortunately, another guy in the group persevered and with some further experiments determined that the code that was being commented out was in a method that was not always being JIT'ed.
 Methods larger than a certain size will not be JIT'ed in hotspot. Commenting out some code would bring the size of the method below the JIT size limit and the method would get compiled. How did that affect GC you might ask? When a method is compiled, the compilers generate and save information on where object references live (e.g., where on the stack or in which registers). We refer to these as oop maps and oop maps are generated to speed up GC. If the method has not been JIT'ed, the GC has to generate the oop maps itself during the GC. We do that by a very laborious means that we call abstract interpretation. Basically, we simulate the execution of the method with regard to where reference are stored. Large methods mean large abstract interpretation times to generate the oop maps. We do save the oop maps for the next GC, but oop maps are different at different locations in the method. If we generate an oop map for PC=200 this time but stop for a GC at PC=300 next time, we have to generate the oop map for PC=300. Anyway, the method in which code was being commented in and out, was too large to be JIT'ed with the code commented in and that led to the long GC's.
 If you have some huge methods and GC's are taking a very long time, you could try -XX:-DontCompileHugeMethods. This will tell the JIT to ignore its size limit on compilation. I'm told by the compiler guy in my carpool that it's not a good idea to use that flag in general. Refactor your methods down to a less than huge size instead. By the way, the huge method was something like 2500 lines so it was what I would call huge.
 https://blogs.oracle.com/jonthecollector/entry/size_matters Size Matters jonthecollector 2007-06-22T10:58:53+00:00 2007-06-22T17:58:53+00:00 I recently had an interesting exchange with a customer about concurrent mode failures with the low-pause collector. My initial response was with my litany of whys and wherefores of concurrent mode failures. But it got more interesting, because it turned out to be an example of where object sizes and the young generation size made an unexpected difference.
 Before I get to that let's talk a little bit about the JVM's behavior with regard to allocating large objects. Large, of course, is a relative term and what I mean here is large relative to the size of the young generation. My comments here apply to JDK 5 update 10 and later and JDK 1.4.2 update 13 and later. Earlier versions of those JVM's had a bug in them (6369448) that made their behavior different. All versions of JDK 6 are also free from this bug. By the way the customer was on JDK 5 update 6 so was subject to 6369448. That made it a little more interesting for me but may not be of interest to you if you are running on the later releases. And also the policy I describe below does not apply to the throughput collector (UseParallelGC) which has its own variation on this policy.
 Objects are normally allocated out of the young generation and get moved to the tenured generation as they age. But what happens when a object is larger than the young generation (actually larger than the eden space in the young generation)? If the application tries to allocate object AAA and there is not enough free space in eden for the allocation, a garbage collection usually occurs. The exception to this is if AAA is larger than eden. An outline of the policy for allocation and collections follow.

 Allocate out of the young generation
 Possibly allocate out of the tenured generation
 Collect the young generation
 Possibly collect the tenured generation
 Heroics

 If the allocation in 1. fails, before starting a GC the JVM considers allocating AAA out of the tenured generation at 2. The JVM compares the size of AAA to the current capacity of eden. By capacity I mean the total size available in an empty eden. If the capacity of eden is too small to hold AAA, then collecting the young generation will not help with the allocation of AAA. So the JVM tries to allocate AAA directly out of the tenured generation. If the allocation out of the tenured generation succeeds, no young generation GC is done and the JVM just continues. If AAA is smaller than eden, then the JVM proceeds to the collection of the young generation at 3. After the the young generation collection, a check is done at 4. to see if enough space has been freed to allocate AAA in eden. If AAA still cannot be allocated out of the young generation, then the tenured generation is collected at 5. and an allocation attempt is made out of the tenured generation. If AAA cannot be allocated in the tenured generation, some additional heroics are attempted (e.g., clearing all SoftReferences). Failing those heroics an out-of-memory is thrown.
 A few things to note. The initial allocation at 1. is what we refer to as fast-path allocation. Fast-path allocation does not call into the JVM to allocate an object. The JIT compilers know how to allocate out of the young generation and code for an allocation is generated in-line for object allocation. The interpreter also knows how to do the allocation without making a call to the VM. And yes, both know about thread local allocation buffers (TLAB's) and use them. The allocation out of the tenured generation at 2. could be attempted unconditionally for any sized object. That could avoid a collection of the young generation, but would also defeat the purpose of having the young generation. We want new objects allocated in the young generation because that's where they will probably die. We only want the longed-lived objects to make it into the tenured generation. Also an allocation at 2. is not a fast-path allocation so the execution of the application could be significantly affected by too much allocation at 2. At 4. a check is made to see if enough space has been freed in the young generation for AAA and if not the tenured generation is collected. Why not attempt the allocation of AAA out of the tenured generation before doing the collection? Attempting the allocation first would probably work lots of the time, but there are pathological cases where allocations would repeatedly be done at 4. If we called allocation at 2. slow-path allocation, we'd have to call allocation at 4. slow-slow-path allocation. At 4. we've already stopped the world and done a young generation collection. The pathological case would execute very slowly. We know. We've tried variations that did the slow-slow-path allocation first and the reports that we usually got was the the JVM was hung! The JVM was not hung. Going slow-slow-path just made it seem so. Doing the collection of the tenured generation actually also collects the young generation and doing that avoids the pathological case in practice.
 The final thing to note is that you don't want to be stuck using slow-path allocation (allocation at 2.) It's really slow compared to fast-path allocation. So if you have large objects (and only you would know), try to make your young generation large enough to hold plenty of them (10's would be good, 100's would be better). Or that you have very few of them.
 The policy for the throughput collector differs from the above at 2. and 5. At 2. the throughput collector will allocate an object out of the tenured generation if it is larger than half of eden. That variation tries to not fill up the young generation with large objects. Is this better? I just don't know. The heroics at 5. are also different.
 The interesting exchange with the customer that got me thinking about this blog started with this snippet of a GC log. Again this was with JDK 5 update 6 so the bug 6369448 was in play.

 963236.282: [GC 963236.282: [ParNew: 15343K->0K(21184K), 0.1025429 secs] 8289743K->8274557K(12287936K), 0.1029416 secs] 963236.479: [GC 963236.479: [ParNew: 10666K->0K(21184K), 0.1072986 secs] 963236.587: [CMS (concurrent mode failure): 8285141K->7092744K(12266752K), 91.3603721 secs] 8285224K->7092744K(12287936K), 91.4763098 secs] 963328.194: [GC 963328.194: [ParNew: 21120K->0K(21184K), 0.1455909 secs] 7135030K->7114033K(12287936K), 0.1459930 secs] 963328.434: [GC 963328.435: [ParNew: 7442K->0K(21184K), 0.0745429 secs] 963328.509: [CMS (concurrent mode failure): 7114084K->7073781K(12266752K), 78.1535121 secs] 7121475K->7073781K(12287936K), 78.2286852 secs] 963408.503: [GC 963408.503: [ParNew : 21120K->0K(21184K), 0.0651745 secs] 7116067K->7097487K(12287936K), 0.0656080 secs]

 When I first looked at this log I jumped right to the first concurrent mode failure. The first thing to notice was that the available free space in the tenured generation was large (about 4g). The total space in the tenured generation was 12266752K and the occupancy (amount of space currently being used) of the tenured generation was only 8285141K (see the part of the log immediately following the first "concurrent mode failure" message). That's about 4g of space. With that much free space the only thing I could think of was a really bad case of fragmentation. Now a concurrent mode failure is bad for the low-pause collector. If it is due to fragmentation at least the resulting collection is a compacting collection and you should not have to worry about fragmentation for a while. So why is there another concurrent mode failure after just one minor collection? And there is even more free space in the tenured generation when this second one occurred (about 5g). So now I'm sweating. I fortunately started noticing some other peculiarities in the log. For example the first minor collection starts when the young generation had only allocated 15343K out of a total of 21184K. That's not that unusual because the survivor space can be sizable and mostly empty. But the third collection is also a minor collection and the young generation has allocated 21120K by the time the minor collection starts (as does the fifth collection). Hmmm. Looking at the occupancies of the young generation at the point of the two concurrent mode failures shows the pattern: the young generation has a good chunk of free space but is still being collected. So I asked the customer if very large objects are being allocated, objects that are approaching the size of the young generation. To my relief the answer was yes. What follows now is my interpretation of the logs. I didn't have the application to run and really debug the behavior, but I think the pieces fit together.
 The first collection (a minor collection) starts seemingly early because a large object is being allocated. The object is larger than the avalable space in eden but is not larger than all of eden so the minor collection does free up enough space to allocate that large object. The next collection is started when an even larger object allocation is attempted. This is where bug 6369448 comes into play. If not for 6369448 the larger object would have been allocated out of the tenured generation at 2. (the corrected case that I discussed above). There was an inconsistency in the test at 2. and the check at 4. before the collection of the tenured generation. A young generation is divided into an eden and two survivor spaces. The test at 2. compared the object size to the total size of the young generation. The check at 4. only checked the size (as is correct) against eden. So the short circuit at 2. was not taken, the collection at 3. doesn't free up enough space, and a tenured generation collection (concurrent mode failure) is the result. The third collection is an uninteresting minor collection. The fourth collection again is a collection prompted by the allocation of an object too big to fit into eden so another concurrent mode failure happens. The fifth collection is also uninteresting as are the many minor collections that follow.
 My story for the two concurrent mode failures does depend on the large objects fitting in that gap in size between eden and the total young generation, but I'm willing to assume that there was a mix of object sizes and occasionally there was something in the gap. The two concurrent mode failure pattern above didn't happen often in the entire log and it was a long log. And the customer increased the size of the young generation and these types of concurrent mode failures did not reoccur.
 I've always expected the occurrence of slow-path allocation to be a rare event. I think it is rare, but that's not the same as never so I'm thinking about adding a counter to be printed into the GC log to indicate how many slow-path allocations have occurred since the last GC. https://blogs.oracle.com/jonthecollector/entry/get_what_you_need Get What You Need jonthecollector 2007-04-23T14:09:22+00:00 2007-04-23T21:09:22+00:00 As we all know by now GC ergonomics exists to automatically adjust the sizes of the generations to achieve a pause time goal and/or a throughput goal while using a minimum heap size. But sometimes that's not what you really want. I was talking to a user recently who was using java.lang.Runtime.totalMemory() and java.lang.Runtime.freeMemory() to monitor how full the heap was. The intent was that the user's application be able to anticipate when a GC was coming and to throttle back on the number of requests being serviced so as to avoid dropping requests due to the reduced throughput when the GC pause did occur. The problem was that the capacity of the heap (the amount returned by totalMemory()) was varying so much that anticipating when a GC was coming was not very reliable. First the pretty picture.

 Looking at the memory used plot first, I inferred from the graph that there was some processing during initialization that required a lesser amount of memory. After that there was activity that used and released memory until about 6k seconds. At that point the activity died down until about 7.5k seconds when there was an up tick in activity and then a quiet period again until about 11k seconds. After that there was pretty regular allocations being done.
 Looking at the total memory I see what I'm expecting to see. The initial total heap size at zero time is larger than is needed during the first phase and the total memory drops. Then as allocations picked up the size of the heap grew and, on average, stayed at a higher level until the first quiet period. During the first quiet period the size total heap size decayed to a lower value that corresponds to the lesser demand for allocations. Then there is the short burst of activity with an corresponding growth in the heap. The toal heap again drops during the second quiet period and then grows again as the activity picks up.
 I like the behavior I see, but it wasn't what the user wanted.
 The amount of variation in the total heap size during the active periods is high. This is varying too quickly for this user. The reason for the variations is that the GC ergonomics is trying to react to variations in the behavior of the applications and so is making changes to the heap as soon as a variation in the application's behavior is recognized. There are two ways to look at this:

 GC ergonomics was designed for the case where a steady state behavior has been reached, and as exhibited by the variations in the amount of memory used, this application doesn't reach a steady state (on the time scale of interest) and GC ergonomics is doing the best that it can under the circumstances, and
 the user's application is what it is and what should he do?

 The first bullet is the excuse I use for almost everything. Let's consider the second.
 First a word about what GC ergonomics does by default that provides some damping of variations.
 GC ergonomics calculates the desired size of a generation (in terms of meeting the goals specified by the user) and then steps toward that size. There is a miminum size for a step and if the distance to the desired size is less than that minimum step, then the step is not made. The minimum step size is the page size on the platform. This eliminates changes as we get close to the goal and removes some of the jitter.
 When deciding whether a goal is being met, GC ergonomics uses weighted averages for quantities. For example, a weighted average for each pause time is used. The weighted averages typically change more slowly than the instantaneous values of the quantities so their use also tends to damp out variations.
 But if you really want a more stable heap size, here's what you can try. These suggestions limit the range of changes that can be made by GC ergonomics. The more minor limitations are given first followed by the more serious ones, finally ending with turning off GC ergonomics. You can just try the suggestions (in bold) and skip over the explanations as you wish.
 Reduce the jitter caused by the different sizes of the survivor spaces (and the flip-flopping of the roles of the survivor spaces) by reducing the size of the survivor spaces. Try NNN = 8 as a starter.
 -XX:MinSurvivorRatio=NNN
 -XX:InitialSurvivorRatio=NNN
 The total size of the young generation is the size of eden plus the size of from-space. Only from-space is counted because the space in the young generation that contain objects allocated by the application is only eden and from-space. The other survivor space (to-space) can be thought of as scratch space for the copying collection (the type of collection that is used for the young generation). Each survivor space alternately plays the role of from-space (i.e. during a collection survivor space A is from-space and in the next collection survivor space B is from-space). Since the two survivor spaces can be of different sizes, just the swapping can change the total size of the young generation. In a steady state situation the survivor spaces tend to be the same size but in situations where the application is changing behavior and GC ergonomics is trying to adjust the sizes of the survivor spaces to get better performance, the sizes of the survivor spaces are often different temporarily.
 The default value for MinSurvivorRatio is 3 and the default value for InitialSurvivorRatio is 8. Pick something in between. A smaller value puts more space into the survivor spaces. That space might go unused. A larger value limits the the size of the survivor spaces and could result in objects being promoted to the tenured generation prematurely. Note that the survivor space sizes are still adjusted by GC ergonomics. This change only puts a limit on how large they can get.
 Reduce the variation in the young generation size by setting a minimum and a maximum with -XX:NewSize=NNN and -XX:MaxNewSize=MMM.
 GC ergonomics will continue to adjust the size of the young generation within the range specified. If you want to make the minimum and maximum limits of the young generation the same you can use -XX:NewSize=NNN and -XX:MaxNewSize=NNN or the flag -XmnNNN will also do that.
 Reduce the range over which the generations can change by specifying the minimum and maximum heap size with -Xms and -Xmx, respectively.
 If you've already explicitly specified the size of the young generation, then specifying the limit on the entire heap will in effect specify the limits of the tenured generation. You don't have to make the minimum and maximum the same but making them the same will have the maximum effect in terms of reducing the variations.
 Turn off GC ergonomics with the flag -XX:-UseAdaptiveSizePolicy
 The size of the generations, the sizes of the survivor spaces in the young generation and the tenuring threshold stay at their starting value throughout the execution of the VM. You can exercise this level of control but the you're back to tuning the GC yourself. But sometimes that really is the best solution. The user I talked to liked the more predictable heap size better than the automatic tuning and so turned off ergonomics. https://blogs.oracle.com/jonthecollector/entry/gc_errata GC Errata 1 jonthecollector 2007-03-05T11:55:56+00:00 2007-03-05T19:55:56+00:00 Just some tid bits of information to fill some of the holes in the documentation.
 GC ergonomics is used by default on server class machines. Please see
 http://java.sun.com/docs/hotspot/gc5.0/ergo5.html
 for a description of GC ergonomics and server class machine. The flag "-server" causes the server VM to be used. "-server" does not cause the platform on which the VM is running to be considered a server class machine.
 Prior to JDK6 the UseParallelGC collector (with GC ergonomics turned on) used only the flags InitialSurvivorRatio and MinSurvivorRatio to set the initial survivor ratio and the minimum survivor ratio, respectively. GC ergonomics dynamically resizes the survivor spaces, but these flags allow the user to specify initial and minimum sizes for the survivor spaces. The flag SurvivorRatio is used by the other collectors (that don't dynamically resize the survivor spaces) to set the size of the survivor spaces. SurvivorRatio was ignored by the UseParallelGC collector. Beginning with JDK6 if the SurvivorRatio is set on the command line for the UseParallelGC collector and InitialSurvivorRatio and MinSurvivorRatio are not explicitly set on the command line, then InitialSurvivorRatio and MinSurvivorRatio are set to SurvivorRatio + 2.
 If UseAdaptiveSizePolicy is turned off when using the UseParallelGC collector, the tenured generation and the young generation stay at their initial sizes throughout the execution of the VM.
 Starting with JDK5 update 06 the maximum tenuring threshold is limited to 15. The GC logging output does not correctly reflect this change. This is a bug that will be fixed under 6521376.
 If a minimum heap size is set with -Xms and NewSize is not explicitly set, the minimum young generation size is calculated using NewRatio. The UseParallelGC collector is not correctly using the minimum value calculated from -Xms and NewRatio but is using the default value of NewSize. Explicitly setting NewSize does correctly set the minimum size for the generation. This is being fixed under bug 6524727. https://blogs.oracle.com/jonthecollector/entry/how_does_your_garden_grow How Does Your Garden Grow? jonthecollector 2007-02-12T14:40:07+00:00 2007-02-12T22:40:07+00:00 Actually, the title should be "How Does Your Generation Grow" but then the literary reference would have been totally lost.
 I've written about how the generations grow if you are using GC ergonomics.
  http://blogs.sun.com/jonthecollector/entry/it_s_not_magic
 What if you're not using GC ergonomics? There is a different policy for growing and shrinking the heap that is used by the low pause collector and the serial collector. If you're interested in that policy, this is you're lucky blog.
 It's actually pretty straightforward. At the end of the collection of a generation, we check the amount of free space in the generation. If the amount of free space is below a certain percentage (specified by MinHeapFreeRatio) of the total size of the generation, then the generation is expanded. Basically, we ask if we have enough free space in the generation so that the VM can run a while before we have to collect again. At the other end we don't want to have an excessive amount of free space so if the amount of free space is above a certain percentage (specified by MaxHeapFreeRatio) the generation is shrunk. That's it.
 If you decide to play with the free ratio parameters, leave enough distance between MinHeapFreeRatio and MaxHeapFreeRatio so that the generations are not constantly adjusting by small amounts to get to the "perfect" ratio. Also our experience is that even if a generation does not need the extra free space right now, it will shortly, so don't be too aggressive with MaxHeapFreeRatio.
 Will this policy eventually be replaced by GC ergonomics? Actually, I think not. I recently talked to a customer who told me that he was more concerned with bounds on the heap footprint than with achieving some specific throughput. The way he put it was something like "I don't want to be doing GC's all the time so there should be a good amount of free space in the heap but not too much". This policy is one approximation of what he wants.
 https://blogs.oracle.com/jonthecollector/entry/jdk6_gc_release_notes JDK6 GC Release notes jonthecollector 2007-02-06T15:17:13+00:00 2007-05-16T15:22:32+00:00 Here's a couple of pointers to the GC section of the release notes for JDK6.
  Parallel Compaction
  Concurrent Mark Sweep Collector Enhancements
 Edit done May 16, 2007.
 Ryan and Tobais, sorry but I just saw your comments. I could not figure out how to reply with a comment (comment period being closed and I didn't want to mess with the settings too much) so I'm editting the blog to respond.
 Ryan,
 The line "CMS-concurrent-abortable-preclean" does not indicate a problem. The "abortable" only indicates that the current phase (precleaning) can be interrupted in order to start the next phase.
 Tobais,
 Yes, the UseParallelOldGC does imply UseParallelGC. https://blogs.oracle.com/jonthecollector/entry/when_you_re_at_your When You're at Your Limit jonthecollector 2007-01-11T08:55:46+00:00 2007-01-11T16:55:46+00:00 It's often the case that we advise users of large applications to use large young generations. The rational is that large applications (very often multithreaded) have large allocation rates and a large young generations is good because it allows more time between minor collections. One case where that is not a good idea is when you're using the largest heap you can (e.g. close to 4G on a 32 bit system) and the live data is bigger than will fit in the tenured generation.
 When the young generation fills up, we want to do a minor collection. Minor collections are in general more productive (the young generation tends to have more garbage) and faster (the time to collect depends largely on the amount of live data in the young generation and not on the about of garbage). If, however, when we're contemplating a minor collection and the tenured generation is full, the garbage collector will think that it's time to do a major collection and collect both the tenured generation and the young generation. If the tenured generation is actually full (or very nearly full) of live data, then the tenured generation may still be full at the end of the major collection. The collection worked, but the next time the young generation fills up, a major collection will again be done.
 If you find yourself in this situation, try reducing the size of the young generation and increasing the size of the tenured generation accordingly. The goal is to have enough free space available in the tenured generation after a major collection so that the next collection can be a minor collection. Even if you can only manage enough free space in the tenured generation to do 1 minor collection, it's a win. Sizing the generations so you can get a few minor collections between major collections would be sweeter, but that all depends on how close you are to your limit.
 https://blogs.oracle.com/jonthecollector/entry/presenting_the_permanent_generation Presenting the Permanent Generation jonthecollector 2006-11-28T12:01:59+00:00 2006-12-05T16:48:55+00:00 Have you ever wondered how the permanent generation fits into our generational system? Ever been curious about what's in the permanent generation. Are objects ever promoted into it? Ever promoted out? We'll you're not alone. Here are some of the answers.
 Java objects are instantiations of Java classes. Our JVM has an internal representation of those Java objects and those internal representations are stored in the heap (in the young generation or the tenured generation). Our JVM also has an internal representation of the Java classes and those are stored in the permanent generation. That relationship is shown in the figure below.

 The internal representation of a Java object and an internal representation of a Java class are very similar. From this point on let me just call them Java objects and Java classes and you'll understand that I'm referring to their internal representation. The Java objects and Java classes are similar to the extent that during a garbage collection both are viewed just as objects and are collected in exactly the same way. So why store the Java objects in a separate permanent generation? Why not just store the Java classes in the heap along with the Java objects?
 Well, there is a philosophical reason and a technical reason. The philosophical reason is that the classes are part of our JVM implementation and we should not fill up the Java heap with our data structures. The application writer has a hard enough time understanding the amount of live data the application needs and we shouldn't confuse the issue with the JVM's needs.
 The technical reason comes in parts. Firstly the origins of the permanent generation predate my joining the team so I had to do some code archaeology to get the story straight (thanks Steffen for the history lesson).
 Originally there was no permanent generation. Objects and classes were just stored together.
 Back in those days classes were mostly static. Custom class loaders were not widely used and so it was observed that not much class unloading occurred. As a performance optimization the permanent generation was created and classes were put into it. The performance improvement was significant back then. With the amount of class unloading that occur with some applications, it's not clear that it's always a win today.
 It might be a nice simplification to not have a permanent generation, but the recent implementation of the parallel collector for the tenured generation (aka parallel old collector) has made a separate permanent generation again desirable. The issue with the parallel old collector has to do with the order in which objects and classes are moved. If you're interested, I describe this at the end.
 So the Java classes are stored in the permanent generation. What all does that entail? Besides the basic fields of a Java class there are

 Methods of a class (including the bytecodes)
 Names of the classes (in the form of an object that points to a string also in the permanent generation)
 Constant pool information (data read from the class file, see chapter 4 of the JVM specification for all the details).
 Object arrays and type arrays associated with a class (e.g., an object array containing references to methods).
 Internal objects created by the JVM (java/lang/Object or java/lang/exception for instance)
 Information used for optimization by the compilers (JITs)

 That's it for the most part. There are a few other bits of information that end up in the permanent generation but nothing of consequence in terms of size. All these are allocated in the permanent generation and stay in the permanent generation. So now you know.
 This last part is really, really extra credit. During a collection the garbage collector needs to have a description of a Java object (i.e., how big is it and what does it contain). Say I have an object X and X has a class K. I get to X in the collection and I need K to tell me what X looks like. Where's K? Has it been moved already? With a permanent generation during a collection we move the permanent generation first so we know that all the K's are in their new location by the time we're looking at any X's.
 How do the classes in the permanent generation get collected while the classes are moving? Classes also have classes that describe their content. To distinguish these classes from those classes we spell the former klasses. The classes of klasses we spell klassKlasses. Yes, conversations around the office can be confusing. Klasses are instantiation of klassKlasses so the klassKlass KZ of klass Z has already been allocated before Z can be allocated. Garbage collections in the permanent generation visit objects in allocation order and that allocation order is always maintained during the collection. That is, if A is allocated before B then A always comes before B in the generation. Therefore if a Z is being moved it's always the case that KZ has already been moved.
 And why not use the same knowledge about allocation order to eliminate the permanent generations even in the parallel old collector case? The parallel old collector does maintain allocation order of objects, but objects are moved in parallel. When the collection gets to X, we no longer know if K has been moved. It might be in its new location (which is known) or it might be in its old location (which is also known) or part of it might have been moved (but not all of it). It is possible to keep track of where K is exactly, but it would complicate the collector and the extra work of keeping track of K might make it a performance loser. So we take advantage of the fact that classes are kept in the permanent generation by collecting the permanent generation before collecting the tenured generation. And the permanent generation is currently collected serially. https://blogs.oracle.com/jonthecollector/entry/why_now Why Now? jonthecollector 2006-10-26T09:42:39+00:00 2006-10-27T15:01:09+00:00 Are you ever surprised by the occurrence of a full collection (aka major collection)? Generally a full collection only happens when an allocation is attempted and there is not enough space available in any of the generations. But there are exceptions to that rule and here are some of them. Actually, these are all of them that I could think of. I thought I would have a longer list so to fill out this blog I've included some bugs that relate to full collections that might interest you. And we're thinking about a policy change for the low pause collector that would affect full collections.
 The simplest reason for an unexpected full collection is that you asked for it. System.gc() can be called from most anywhere in the program. Try using -XX:+DisableExplicitGC to have the garbage collector ignore such requests.
 The permanent generation is collected only by a full collection. It's very seldom the cause for a full collection but don't overlook the possibility. If you turn on PrintGCDetails, you can see information about the permanent generation collection.
 0.167: [Full GC [PSYoungGen: 504K->0K(10368K)] [ParOldGen: 27780K->28136K(41408K)] 28284K->28136K(51776K) [PSPermGen: 1615K->1615K(16384K)], 0.4558222 secs]
 This output is from the throughput collector. The "PSPermGen" denotes the permanent generation. The size of the permanent generation currently is 16384K. It's occupancy before the collection is only 1615K and is probably the same after the collection. If the permanent generation needed collecting, the occupancy before the collection would have been closer to the 16384K.
 The garbage collector tracks the average of the rate of promotion of live objects from the young generation into the tenured generation. If the average exceeds the free space in the tenured generation, the next time the young generation fills up, a full collection is done instead of a minor collection. There is a pathological situation where, after the full collection, the free space in the tenured generation is still not large enough to take all the live objects expected to be promoted from the next minor collection. This will result in another full collection the next time the young generation fills up. The bad part about this is that the average value for the amount promoted only changes when a minor collection is done and if no minor collections are done, then the average does not change. Increasing the size of the heap usually avoids this problem.
 Sometimes it's not really a full collection. There was a bug affecting JDK5 (6432427) which print "Full" when the collection was not a full collection. When JNI critical sections are in use, GC can be locked out. When the JNI critical sections are exited, if a GC had been requested during the lock out, a GC is done. That GC most likely would be a minor collection but was mistakenly labeled a full collection. A second place where "Full" was also printed erroneously was when -XX:+CMSScavengeBeforeRemark is used. The same bug report explains and fixes that error.
 An attempt to allocate an object A larger than the young generation can cause a minor collection. That minor collection will fail to free enough space in the young generation to satisfy the allocation of A. A full collection is then attempted and A is then allocated out of the tenured generation. This behavior was a bug (6369448). It has been fixed in JDK's 1.4.2_13, 5.0_10, and 6. The correct behavior is for the GC to recognize when an object is too large to be allocated out of the young generation and to attempt the allocation out of the tenured generation before doing a collection.
 When an allocation cannot be satisfied by the tenured generation a full collection is done. That is the correct behavior but that behavior has an adverse affect on CMS (the low pause collector). Early in a run before the CMS (tenured) generation has expanded to a working size, concurrent mode failures (resulting in full collections) happen. These concurrent mode failures perhaps can be avoided. We are thinking about a change to the policy such that CMS expands the generation to satisfy the allocation and then starts a concurrent collection. We're cautious about this approach because similar policies in the past has led to inappropriate expansion of the CMS generation to its maximum size. To avoid these full collections try specifying a larger starting size for the CMS generation.
 So just because these are all the unexpected "Full" collections that I could think of, that doesn't mean that these are all there are. If you have a mysterious "Full" collection happening, submit a description to
  http://java.sun.com/docs/forms/gc-sendusmail.html
 I'll post any additional cases of unexpected "Full" collections that I get.
 Late addition (I'm allowed to edit my blog even after they've been posted). If you ask about unexpected "Full" collections and send in a gc log, please add the command line flags
 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps
 The additional output (of which there will be plenty) may help me see what's happening. https://blogs.oracle.com/jonthecollector/entry/the_second_most_important_gc The Second Most Important GC Tuning Knob jonthecollector 2006-09-11T15:54:49+00:00 2006-09-11T22:54:49+00:00 is NewRatio or it's equivalent. Number 1 would be the total heap size. And just so the guys down the hall don't start throwing rocks at me, the 0-th most important GC tuning knob is the choice of the type of garbage collector itself. But let's talk about number 2.
 NewRatio is a flag that specifies the amount of the total heap that will be partitioned into the young generation. It's the tenured-generation-size / young-generation-size. A NewRatio value of 2 means that the tenured generation size is 2/3 of the total heap and the young generation size is 1/3. There are other ways of specifying that split. MaxNewSize will specify the maximum size of the young generation. The flag -XmnNNN is equivalent to -XX:NewSize=NNN and -XX:MaxNewSize=NNN. NewSize is the initial size of the young generation.
 Say you've chosen the right size for your overall heap. But does the value of NewRatio split the space between the young generation and the tenured generation optimally. If your application has a large proportion of long lived data, the value of NewRatio may not put enough of the space into the tenured generation. I recently was looking at the performance of an application where too many major collections where being done. The problem was that the long lived data overflowed the tenured generation. When a collection was needed the tenured generation was basically full of live data. Much of the young generation was also filled with long lived data. The result was that a minor collection could not be done successfully (there wasn't enough room in the tenured generation for the anticipated promotions out of the young generation) so a major collection was done. The major collection worked fine, but the results again was that the tenured generation was full of long lived data and there was long lived data in the young generation. There was also free space in the young generation for more allocations, but the next collection was again destined to be a major collection. By decreasing the space in the young generation and putting that space into the tenured generation (a value of NewRatio larger than the default value was chosen), there was enough room in the tenured generation to hold all the long lived data and also space to support minor collections. This particular application used lots of short lived objects so after the fix mostly minor collections were done.
 Why not just make NewRatio conservatively large to avoid this situation? You want your young generation to be large enough so that objects have a chance to die between collections. Applications with many threads may be allocating at a high rate so a large young generation might be appropriate.
 The point here is that if you choose your overall heap size carefully, also think about the size of the young generation. This is particularly true if the maximum size of your heap is constrained for some reason (e.g., by a limited amount of physical memory).
 Something to add here is that the default value of NewRatio is platform dependent and runtime compiler (JIT) dependent. Below are the values as we get ready for the JDK6 release.
 -server on amd64 2 -server on ia32 8 -server on sparc 2 -client on ia32 12 -client on sparc 8
 These reflect the expectation that the server compiler will be used on applications with more threads and so have higher allocation rates.
 And, finally, if you're about to ask why there are different ways to set the size of the young generations:
 NewRatio gives you a way to scale the young generation size with the total heap size.
 NewSize and MaxNewSize give you precise control.
 -Xmn is a convenience
 https://blogs.oracle.com/jonthecollector/entry/more_of_a_good_thing More of a Good Thing jonthecollector 2006-08-30T12:17:22+00:00 2006-08-30T19:17:22+00:00 The throughput collector is getting more parallel.
 In our J2SE 6 release there is an option in the throughput collector to collect the tenured generation using multiple GC threads. Yes, the UseParallelGC collector wasn't all that we wanted it to be in terms of parallelism. Without this recent work only the young generation was being collected in parallel.
 The option to turn on the parallel collection of the tenured generation is -XX:+UseParallelOldGC. That's the punch line for this blog. Oh, also it's off by default.
 Below are a few details on the new collector. Actually, first I'll describe the serial collector used for the tenured generation collection and some of the things we learned from it. Then I'll describe the new collector and how we used what we had learned.
 The J2SE 5 (and earlier) collector for the tenured generation was a serial mark-sweep-compact collector. There were four phases of that collector.
 1. Mark all the live objects
 2. Calculate the new locations of the objects
 3. Adjust all object references to point to the new locations
 4. Move the objects to their new locations
 Phase 1 found and marked the live objects.
 In phase 2 the collector kept a pointer to the first location in the tenured generation to which we could move a live object during the collection. Let's call that location T. T started at the bottom of the generation. The collector scanned forward from the bottom of the generation looking for marked (i.e., live) objects. When a live object A was found, the collector saved T in A (i.e., saved the new location of A in A). It then calculated the next location T as T + sizeof(A) and restarted the scan looking for the next live object to put at the new T.
 Phase 3 started at the roots, references to objects known to the application (for the most part), and changed all the references in the roots and all the reference found from the roots to point to the new location of the objects (to the T's saved in the A's).
 Phase 4 again started at the bottom of the tenured generation and scanned forward to find live objects. When a live object was found, it was moved to it's new location (the T's).
 The end result was that all the live objects were moved to the lower end of the tenured generation and all the free space was in one contiguous region at the higher end of the tenured generation. It simplifies allocations from the tenured generation to have all the free space in a single region. Also there are no issues of fragmentation as would be the case if the collection resulted in multiple regions of free space separated by live objects. This single region of free space seemed an important property to keep when thinking about how to do a collection of the tenured generation using multiple GC threads.
 Another feature of our serial mark-sweep-compact collector that we liked is the flexibility to leave some deadwood at the lower end of the tenured generation. We use the term "deadwood" to refer to garbage at the lower end of the generation that we're not going to collect. We don't collect it to reduce the number of objects that we have to move. If we have objects
 A B C D
 and if B is garbage and we collect it, then we have to move C and D. If we treat B as deadwood, then C and D can stay where they are. We make a judgment call to waste a certain amount of space (that occupied by the deadwood) in order to move fewer objects.
 You may have noticed that each of the four phases of our mark-sweep-compaction walked over at least all the live objects. In some cases it actually walked over all the objects (live and dead). When we're scanning the heap (such as in phases 2), we look at a dead object just to find its size so that we can step over it. Any part of the collection that walks over the objects is costly so with the new parallel collector for the tenured generation we wanted to have fewer walks over the objects.
 Keeping all this in mind resulted in a parallel collector that
 \* Marked the live objects using multiple GC threads and maintained some summary information (explained below).
 \* Used the summary information to determine where each live object would go. Also determined the amount of deadwood to keep.
 \* Moved live objects so there was one contiguous block of free space in the generation and updated references to live objects using multiple GC threads.
 I'll refer to these three phases as the marking phase, summary phase and move-and-update phase. In some of the other documentation on this new collector the last phase is referred to as the compaction phase. I prefer move-and-update because that's the type of name used in the code. Just in case you're ever looking at the code.
 Marking phase
 Doing the marking in parallel was straight forward. The parallel young generation collector does this and a similar technique was used. The additional twist to the marking was creating the summary information. The heap is logically divided into chunks. Each object starts within a chunk. As we're doing the marking, when a live object was found, its size was added to the chunk containing the start of the object. At the end this gives us the size the of data for objects that begin in each chunk.
 Summary phase
 The summary phase first walks over the chunks looking for the extent of the heap that contains a good amount of deadwood. We call that part of the heap the dense prefix. Live objects that are moved will be placed in the heap starting immediately after the dense prefix. Given that we know the amount of live data for each chunk, we can calculate the new location of any live object using the summary data. Basically adding up the sizes of the live data in all the chunks before chunk A tells you where the live data in chunk A is going to land. The summary phase is currently done by one thread but can be done in parallel if it turns out to be a scaling bottleneck.
 Update-and-move phase
 Given the summary information the move-and-update phase identifies chunks that are ready to be filled and then moves the appropriate objects into those chunks. A chunk is ready to be filled if it is empty or if its objects are going to be compacted into that same chunk. There is always at least one of those. The move-and-update phase is done with multiple GC threads. One thing to note is that it is possible that there is only 1 ready chunk at any given time. If the chunks are A, B, C, D ... and the only dead object is in A and fits entirely in A, then A is the only ready chunk. The move-and-update phase will first fill any remaining space in A. Then B will be the only ready chunk. Obviously an extreme case but it makes the point. There is a technique to widen this "gap" of ready chunks. We expect to implement it, but it's not in this release.
 That's basically it. We're working on some scaling issues with this new collector, but it working pretty well as is.
 For more on this new collector see the slides for the GC presentation done at this year's JavaOne. They can be download from
 http://java.sun.com/javaone/
 The last time I looked, there was a "Click here" link for the technical sessions in the first paragraph. Following that link, download the Java SE bundle and in there our GC presentation is TS-1168. https://blogs.oracle.com/jonthecollector/entry/the_gc_whitepaper The GC whitepaper jonthecollector 2006-08-24T11:53:43+00:00 2006-08-24T19:01:13+00:00 There's a very nice whitepaper on our GC at
 http://java.sun.com/j2se/reference/whitepapers/memorymanagement_whitepaper.pdf
 It talks about basic concepts, some specifics about our GC, suggestions on tuning, and some of the latest improvements such as the parallel old collector. If you've looked at the GC tuning guides, some of it will look familiar, but it nicely pulls together information from several sources. https://blogs.oracle.com/jonthecollector/entry/the_real_thing The Real Thing jonthecollector 2006-08-15T08:18:26+00:00 2007-05-10T00:16:32+00:00 Here's a real treat dished up by Ross K. who hangs out 3 offices down the hall. A while back I wrote a blog about thread local allocation buffers (TLAB's). The short comings of my blog have probably been annoying Ross so much that he's written this one for us. If there are any discrepancies between my earlier blog and this one, believe this one. Thanks, Ross.

 A Thread Local Allocation Buffer (TLAB) is a region of Eden that is used for allocation by a single thread. It enables a thread to do object allocation using thread local top and limit pointers, which is faster than doing an atomic operation on a top pointer that is shared across threads. A thread acquires a TLAB at it's first object allocation after a GC scavenge. The size of the TLAB is computed via a somewhat complex process discribed below. The TLAB is released when it is full (or nearly so), or the next GC scavenge occurs. TLABs are allocated only in Eden, never from From-Space or the OldGen. Flags default description UseTLAB true Use thread-local object allocation ResizeTLAB false Dynamically resize tlab size for threads TLABSize (below) Default (or starting) size of a TLAB (in bytes) TLABWasteTargetPercent 1 Percentage of Eden that can be wasted PrintTLAB false Print various TLAB related information AggressiveHeap settings: TLABSize 256Kb ResizeTLAB true (Corrected 2007/05/09) Minor flags MinTLABSize 2\*K Minimum allowed TLAB size (in bytes) TLABAllocationWeight 35 Weight for exponential averaging of allocation TLABRefillWasteFraction 64 Max TLAB waste at a refill (internal fragmentation) TLABWasteIncrement 4 Increment allowed waste at slow allocation ZeroTLAB false Zero out the newly created TLAB These flags are for tuning the current implementation of TLABs and maybe disappear or change their initial value in a future release of the jvm. If it is not specified on the command-line (or specified as zero) via the -XX:TLABSize flag, the initial size of a TLAB is computed as: init_size = size_of_eden / (allocating_thread_count \* target_refills_per_epoch) where: a) Allocating_thread_count is the expected number of threads which will be actively allocating during the next epoch (an epoch is the mutator time between GC scavenges.) At jvm startup this is defined to be one. It is then recomputed at each GC scavenge from the number of threads that did at least one allocation of a tlab during the latest epoch. It's then exponentially averaged over the past epochs. b) Target_refills_per_epoch is the desired number of tlab allocations per thread during an epoch. It is computed from the value of TLABWasteTargetPercent which is the percentage of Eden allowed to be wasted due to TLAB fragmentation. From a mutator thread's perspective a GC scavenge can occur unexpectedly at any time. So, on average, only half of a thread's current TLAB will be allocated when a GC scavenge occurs. TLABWasteTargetPercent = 0.5 \* (1/target_refills_per_epoch) \* 100 Solving for target_refills_per_epoch: target_refills_per_epoch = ( 0.5 \* 100) / TLABWasteTargetPercent With the default value of 1 for TLABWasteTargetPercent target_refills_per_epoch = 50 When TLABResize is true (which it is by default) the tlab size is recomputed for each thread that did an allocation in the latest epoch. Threads that did not allocate in the latest epoch do not have their TLABs resized. The resize goal is to get the number of refills closer to the ideal: target_refills_per_epoch (default value is 50). For each thread, the number of refills in the latest epoch is exponentially averaged with values from previous epochs. If this average refill number is greater than target refills_per_epoch, then the tlab size is increased. If the average is less, the tlab size is decreased. The computation is (approximately): new_size = (old_size \* avg_refills_per_epoch) / target_refills_per_epoch It's actually computed from the fraction of the latest epoch's eden size used by this thread, because the next epoch may use a resized eden. To experiment with a specific TLAB size, two -XX flags need to be set, one to define the initial size, and one to disable the resizing: -XX:TLABSize=<bytesize> -XX:-ResizeTLAB The minimum size of a tlab is set with -XX:MinTLABSize which defaults to 2K bytes. The maximum size is the maximum size of an integer Java array, which is used to fill the unallocated portion of a TLAB when a GC scavenge occurs. Diagnostic Printing Options -XX:+PrintTLAB Prints at each scavenge one line for each thread (starts with "TLAB: gc thread: " without the "'s) and one summary line. Thread example: TLAB: gc thread: 0x0004ac00 [id: 2] size: 61KB slow allocs: 5 refill waste: 980B alloc: 0.99996 3072KB refills: 50 waste 0.1% gc: 0B slow: 4144B fast: 0B The tag "gc" indicates that this information was printed at a GC scavenge, after the tlabs have been filled. The "gc" tag doesn't mean a thread is a gc thread. Fields: thread: The address of the gc thread structure and it's system thread id. size: The size of the tlab in kilobytes. slow allocs: The number of allocations too large for remaining space in the TLAB. The allocation was done directly in eden space. refill waste: (in HeapWord units) The name is truncated in the dump, and should be: refill_waste_limit and is used to limit the amount of wasted space from internal fragmentation. If the remaining space in the TLAB is larger than this amount, and an allocation is requested that is too large to be allocated in the TLAB, then the allocation is done directly in Eden and the TLAB is not retired. If the remaining space is less than refill_waste_limit then the TLAB is retired, a new TLAB is allocated, and the object allocation is attempted in the new TLAB. After each allocation outside of the TLAB, the refill_waste_limit is incremented by TLABWasteIncrement to prevent an allocation of a size slightly less than refill_waste_limit from continually being allocated outside of the TLAB. alloc: [fraction] [sizeInBytes] Expected amount of eden allocated by this thread computed as a fraction of eden and number of heap words. refills: Number of tlab refills. waste [percent] gc: [bytes] slow: [bytes] fast: [bytes] Percentage of eden allocated to this thread that was wasted. Waste is the sum of three components: gc: unused space in the current TLAB when stopped for a scavenge. slow: sum of unused space in TLABs when they're retired to allocate a new one. fast: the client system can allocate a TLAB with a fast allocator. This is the amount of waste via that method. Summary example: TLAB totals: thrds: 1 refills: 50 max: 50 slow allocs: 5 max 5 waste: 0.1% gc: 0B max: 0B slow: 4144B max: 4144B fast: 0B max: 0B thrds: Number of threads that did an allocation. refills: [tt] max: [mm] Total number of TLAB refills by all threads, and maximun number of TLAB refills by a single thread. slow allocs: [ss] max [mm] Total number of allocations done outside of a TLAB, and maximum number by a single thread. waste [percent] gc: [bytes] slow: [bytes] max: [mm] fast: [bytes] max: [mm] Percentage of eden that was wasted across all threads. Waste is the sum of three components: gc: unused space in the current TLABs when scavenge starts. slow: sum of unused space in TLABs when they're retired to allocate a new ones. fast: the client system can allocate a TLAB with a fast allocator. This is the amount of waste via that method. For "slow" and "fast", the maximum value by a single thread is printed. More detail with addition of Verbose flag. -XX:+PrintTLAB -XX:+Verbose Using both: -XX:+PrintTLAB -XX:+Verbose will print the new tlab sizes for each thread when it is resized. Resizing is only done at GC scavenges. Example: TLAB new size: thread: 0x001eac00 [id: 19] refills 50 alloc: 0.402570 size: 19684 -> 18996 New size 18996. Previous size was: 19684. refills: Number of tlab refills for this thread. alloc: The expected fraction of eden this thread will use.
 https://blogs.oracle.com/jonthecollector/entry/location_location_location Location, Location, Location jonthecollector 2006-08-14T22:46:24+00:00 2006-08-15T05:46:24+00:00 Allocating a bunch of objects together and then using them together seems to be a good thing for cache utilization. Not always true, of course, but, if you're willing to take that as a quasi-truth, then you might be interested in how garbage collections can stir up that good locality.
 I'm going to talk first about our serial garbage collector. It is a generational garbage collector with a copying young generation collector and a mark-sweep-compact tenured generation collector. My comments generally apply to all our collectors but allow me to do the simplest one first. After that I'll comment on how doing the collection in parallel makes it even more interesting. By the way the techniques I'll be describing for the serial collector are pretty generic and are used by other garbage collectors.
 When a application thread T allocates objects, the allocations are generally done out of the young generation. Those allocations done by T together (i.e., at about the same time) are allocated in close proximity in the heap. This is mostly true even though allocations are being done in parallel by different application threads (different T's). Each T allocates from its own thread local allocation buffer (TLAB). I described recent improvement to TLAB's in an earlier blog with the title "A Little Thread Privacy, Please". If you read the first part of that one, you'll get the idea. At this point after the allocations are done, we have the good locality we've come to expect from the allocate-together, use-together practice. Then a garbage collection happens.
 When the young generation is collected, the surviving objects can be copied to either the survivor spaces or the tenured generation. The survivor spaces are areas in the young generation where young generation objects are kept if they survive a few collections. After surviving a few collections objects in the survivor spaces are copied (promoted) to the tenured generation where long lived objects are kept. Let's consider the very first collection and the case where all the objects that survive that first collection are copied to a survivor space.
 The young generation collection starts by finding objects being used directly by the application. By "directly" I mean the application has a reference to those objects A. That's as opposed to having a reference to an object (A) that contains a reference to a second object (B). The garbage collector scans the stacks of each application thread and, when it finds a reference to an object A, it copies A to a survivor space. Incidentally, that's also when the reference to A in the thread's stack is updated to point to the new location of A. Only A is copied at this point. If A contains references to another object B, the object B is not touched. The garbage collector continues to scan the stacks of the threads until it's found and copied all the objects directly referenced by the application. At that point the garbage collector starts looking inside the objects A that it has copied. It's then that the garbage collector would find B in A and would copy B to a survivor space. And as before if B contains a reference to object C, nothing is yet done about C.
 As an example, suppose an application thread T
 \* allocates object C
 \* allocates object B and puts a reference to C into B
 \* allocates object A and puts a reference to B into A
 repeatedly. The layout of objects in the heap would look something like
 ... C1 B1 A1 C2 B2 A2 C3 B3 A3 ...
 If thread T only kept a reference to the objects A (i.e., there are no reference directly to the B's or the C's on the thread stacks), then after the young generation collection the layout in the survivor space would look like
 ... A1 A2 A3 ... B1 B2 B3 ... C1 C2 C3 ...
 where the (...) represent other objects X found directly from T's stack or objects found from those X's. Here I'm being just a little generous in grouping the A's, B's, and C's together. That grouping assumes that A1 is found first, then A2 is found, and then A3 is found.
 Before the collection loading A1 into a cache line could reasonably also load B1 and C1 into the same cache line. After the collection B1 and C1 are in a galaxy far, far away. The A's, B's and C's have been separated so that their allocation order is no longer reflected in their location in the heap. The basic damage has been done.
 In the case where all the objects A, B, and C are promoted to the tenured generation, the layout in the tenured generation would be similar (just in the tenured generation instead of in a survivor space). In the case where some of the objects are copied to a survivor space and some are promoted, some of each of the A's, B's, and C's can end up in either a survivor space or the tenured generation. You might think that it would tend to look something like
 (survivor space) ... A1 A2 A3 ...
 (tenured generation) ... B1 B2 B3 ... C1 C2 C3 ...
 It might. Or it might be a bit messier. The above example would result if there was room in the survivor spaces for the A's but not for the B's and C's. If the B's were particularly large (so that they did not fit into the space remaining in the survivor space) and the C's were particularly small, then you might see
 (survivor space) ... A1 A2 A3 ... C1 C2 C3 ...
 (tenured generation) ... B1 B2 B3 ...
 If there are references in the application thread stack to the B's and C's as well as to the A's, and the collector found objects in the order
 C2 B3 A1 A2 A3
 then after the collection the object layout would look like
 ... C2 B3 A1 A2 A3 ... B1 B2 ... C1 C3 ...
 The mixtures are as endless as the ways in which the application references its objects.
 How does collecting in parallel affect things? Recall that the serial collector first looks at all the references found in the application thread stacks (in the simpliest case the A's) before it looks at references inside the A's (i.e., the B's). The parallel collector starts by looking at some of the A's, but then may look at some of the B's before all the A's have been found. A parallel GC thread copies an A and puts its new location on a stack. I'll refer to this stack as its grey stack so as not to confuse it with the stacks of the application threads. Each GC thread has its own grey stack. When its grey stack starts to get full, a GC thread will start taking the locations of the A's off its grey stack and looking inside them for B's to copy. It starts looking inside the A's so that its grey stack does not grow too large. In general we prefer to do some copying instead of having to malloc more space for the grey stack. Very quickly there are A's, B's and C's on the grey stacks and the order in which they are copied looks largely random from outside the garbage collector.
 When a parallel GC thread promotes an object, it promotes it into its own distinct area (let me call them PLAB's) in the tenured generation. A GC thread that has run out of objects to copy (no more application thread stacks to scan and no more objects on its own grey stack) can steal objects from another GC thread's grey stack. So A1 may be copied by one GC thread into its PLAB while B1 (if it were stolen) may be copied by another GC thread into a different PLAB.
 So collections can disrupt good object locality in a variety of different ways. We don't like that. We're thinking about ways to improve it.
 Hope this was understandable. Please ask if it was not. https://blogs.oracle.com/jonthecollector/entry/top_10_gc_reasons Top 10 GC Reasons jonthecollector 2006-05-03T15:57:09+00:00 2006-05-03T22:57:09+00:00 to upgrade to J2SE 5.0
 10) We fixed some bugs.
 9) Thread local allocation buffer dynamic tuning. (see "A Little Thread Privacy Please")
 8) Promotion failure handling was implemented for the low pause collector.
 Promotion failure handling is the ability to start a minor collection and then backout of it if there is not enough space in the tenured generation to promote all the objects that need to be promoted.
 In 5.0 the "young generation guarantee" need not apply for the low pause collector because we implemented promotion failure handling for that collector. The net effect of having promotion failure handling is that more minor collections can be done before a major collection is needed. Since minor collections are typically more efficient than major collections, it's better. The "young generation guarantee" is discussed in the document below.
 http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html
 It was particularly harsh on the low pause collector because the "young generation guarantee" required one contiguous chunk of space in the tenured generation. For the low pause collector that requirement could be difficult to satisfy because the low pause collector does not compact the tenured generation and so is subject to fragmentation.
 7) Parallel remark in the low pause collector.
 For the low pause collector there are two stop-the-world pauses associated with each collection of the tenured generation: the initial marking pause and the remarking pause. The latter is typically much longer than the former. In 5.0 the remarking is done with multiple threads in order to shorten it.
 6) Parallel reference processing in the low pause collector.
 For an application that uses Reference objects (see)
 http://java.sun.com/j2se/1.5.0/docs/api/java/lang/ref/Reference.html
 extensively, the GC work to process the Reference objects can be noticeable. It's not necessarily worse in the low pause collector than in the other collects, but it hurts more (because we're trying to keep the pauses low). Parallel reference processing is available for the low pause collector but is not on by default. Unless there are tons of Reference Objects, doing the reference processing serially is usually faster. Turn it on with the flag -XX:+ParallelRefProcEnabled if you make extensive use of Reference Objects (most applications don't).
 5) Server class machine defaults.
 On server class machines (machines with 2 or more cpus and 2G or more of physical memory) the default compiler, garbage collector and maximum heap size are different. Before 5.0 it was
 \* -client compiler
 \* serial garbage collector
 \* 64M maximum heap (on most 32 bit platforms)
 With 5.0 the defaults for server class machines are
 \* -server compiler
 \* throughput garbage collector
 \* maximum heap size the lesser of 1/4 of physical memory or 1G
 For the smaller machines the defaults did not change.
 The intent of this change was to provide better default performance on large machines that are typically dedicated to running 1 or a few applications and for which throughput is important. Desktops for which interactivity was expected to be more important were excluded from being server class machines.
 See
 http://java.sun.com/docs/hotspot/gc5.0/ergo5.html
 for more details.
 4) Low pause collector schedules remarking between minor collections.
 Recall that the pause due to object remarking is the longer of the two stop-the-world pauses. Before 5.0 it could happen that a remark pause would occur immediately after a minor pause. When this back-to-back minor pause and remark pause occurred it looked like one big fat pause. With 5.0 the remarking is scheduled so as to be about mid way between two minor pauses.
 3) Better scheduling of the start of a concurrent low pause collection.
 Prior to 5.0 a low pause collector started a collection based on the rate of allocation and the amount of free space in the tenured generation. It did the calculation and started a concurrent collection so as to finish before the tenured generation ran dry. This was good, but there were some end cases that needed to be recognized. For example, if the tenured generation had to be expanded in order to support promotions for a minor collection that just finished, a concurrent collection was started right away. Or if the next minor collection might not succeeded because of lack of free space in the tenured generation, then a concurrent collection was started. That latter example wouldn't happen if we could perfectly predict the rate at which the tenured generation is filling up. We're not perfect. Also a bump in the allocation rate might mess us up.
 2) GC ergonomics (See "It's Not Magic" at the start of my blogs)
 And the number 1 GC reason for upgrading to 5.0 is
 1) ???
 You can tell me what you've found to be a good reason to upgrade to 5.0 in a comment to this blog. Don't be shy. And thanks for any responses. https://blogs.oracle.com/jonthecollector/entry/what_the_heck_s_a What the Heck's a Concurrent Mode? jonthecollector 2006-04-13T09:15:45+00:00 2006-04-13T16:15:45+00:00 and why does it fail?
 If you use the low pause collector, have you ever seen a message that contained the phase "concurrent mode failure" such as this?
 <verbatim> 174.445: [GC 174.446: [ParNew: 66408K->66408K(66416K), 0.0000618 secs]174.446: [CMS (concurrent mode failure): 161928K->162118K(175104K), 4.0975124 secs] 228336K->162118K(241520K) </verbatim>
 This is from a 6.0 (still under development) JDK but the same type of message can come out of a 5.0 JDK.
 Recall that the low pause collector is a mostly concurrent collector: parts of the collection are done while the application is still running. The message "concurrent mode failure" signifies that the concurrent collection of the tenured generation did not finish before the tenured generation became full. Recall also that a concurrent collection attempts to start just-in-time to finish before the tenured generations becomes full. The low pause collector measures the rate at which the the tenured generation is filling and the expected amount of time until the next collection and starts a concurrent collection so that it finished just-in-time (JIT). Three things to note in that last sentence. The "rate" at which the tenured generation is filling is based on historical data as is the "expected amount of time" until the next collection. Either of those might incorrectly predict the future. Also the JIT is really JIT plus some amount of padding so as to get it right most of the time. When a concurrent mode failure happens, the low pause collector does a stop-the-world (STW) collection. All the application threads are stopped, a different algorithm is used to collect the tenured generation (our particular flavor of a mark-sweep-compact), the applications threads are started again, and life goes on. Except that the STW collection is not very low pause and there's the rub.
 At this point if you're asking "why does this happen", you've come to the right blog. There are several possibilites.
 The amount of live data that is in the tenured generation is too large. Specifically, there is not enough free space in the tenured generation to support the rate of allocation into the tenured generation. For an example in the extreme if there are only 2 words of free space in the tenured generation after a collection of the tenured generation, chances are those 2 words will be exhausted before another concurrent collection of the tenured generation can be done. If you are seeing lots of concurrent mode failures, chances are your heap is too small.
 Your application can change behaviors dramatically such that past behavior does not adequately predict future performance. If this is the problem you'll see the concurrent mode failures only near the change in behavior. After a few more collections the low pause collector adjusts its expectations to make better decisions. But to deal with the concurrent mode failures in the mean time, you'll usually be trading off better performance. You can tell the low pause collector to start a collection sooner. The flag -XX:CMSInitiatingOccupancyFraction=NN will cause a concurrent collection to start when NN percent of the tenured generation is full. If you use this option to deal with the concurrent mode failures that result from a change in the behavior of your application, much of the time (when the applications behavior is more steady state) you'll be starting collection too early and so doing more collections than necessary. If you set NN to 0, it will cause one concurrent collection to be followed as soon as possible by another. The next collection may not start immediately after the last because the check on when to start a collection is done only at particular points in the code, but the collection will start at the next opportunity.
 Your application may not have dramatic changes in behavior, but if it has a large variance in allocation rates, that can cause the JIT GC to not be JIT. You can add some more padding to the time at which a concurrent collection kicks off by using the flag -XX:CMSIncrementalSafetyFactor=NN. The default value for NN is 10 (i.e., a 10% padding on the start of the concurrent collection). Increasing NN to 100 starts a concurrent collection at the next opportunity. https://blogs.oracle.com/jonthecollector/entry/the_fault_with_defaults The Fault with Defaults jonthecollector 2006-04-04T07:47:04+00:00 2006-04-04T17:42:57+00:00 During early work on the low pause collector some of the applications we used for benchmarking the collector had the characteristic that objects had very short lifetimes or relatively long lifetimes with not much in between. All our current collectors use a copying collection for the young generation. This copying collection will copy the younger objects back into the young generation and copy the older objects into the tenured generation. The expectation is that more of the younger objects will die in the young generation if they are give a little more time. A consequence of this is that long lived objects are copied repeatedly into the young generation before finally being promoted into the tenured generation. Depending on the mix of lifetimes of your objects, this can be good (more objects die in the young generation) or this can be bad (extra copying into the young generation). Additionally at about the same time we were discussing the policy of never doing the copying but rather always copying any objects that survived a young generation collection immediately into the tenured generation. There are some collectors that do that and there was some anecdotal evidence that it was a good strategy. Given some example applications where never copying back into the young generations seemed to be the right strategy and the general discussion about what was the right policy for copying, we decided to make the default behavior for the low pause collector to always promote objects surviving a young generation collection into the tenured generation.
 This default policy still seems reasonable for some applications but the old adage about one size not fitting all certainly applies here. So here's what you should consider and what you can do.
 The young generation is acting like a filter to dispose of the shorter lived objects. Any objects that get through the filter are moved to the tenured generation and are henceforth handled by the mostly concurrent collections (which I'll just call the concurrent collections from here on out). If the concurrent collections are starting too often to suit your tastes, one possibility is to filter out more of the short lived objects by doing some copying during the young generation collections. This may slow down the growth in the number of objects in the tenured generation and lessen the need for too frequent concurrent collections. To do this try the command line flags
 -XX:MaxTenuringThreshold=15 -XX:SurvivorRatio=8
 MaxTenuringThreshold is the number of collections that an object must survive before being promoted into the tenured generation. When objects are copied back into the young generation during a young generation collection, the objects are copied into an part of the young generation that is referred to as a survivor space. SurvivorRatio set to 8 will mean that the survivor spaces are about 10% of the size of the young generation. The survivor spaces and SurvivorRatio are described in the 5.0 tuning guide that can be found under
 http://java.sun.com/docs/hotspot
 The FAQ attached to the 5.0 tuning guide has an entry about experimenting with MaxTenuringThreshold.
 By the way having concurrent collections starting "too often" can also be a sign that the tenured generation is smaller than is needed for good performance. You can always try increasing the size of the heap to reduce the frequency of the concurrent collections. The suggestions above may be able to get you the less frequent concurrent collections without the longer concurrent collections that typically come when the heap is increased.
 The default setting to always promote objects that survive a young generation collection is only the default for the low-pause collector. The other collectors by default will do some copying before promoting an object to the tenured generation. https://blogs.oracle.com/jonthecollector/entry/when_the_sum_of_the When the Sum of the Parts jonthecollector 2006-03-06T09:47:00+00:00 2006-03-06T17:47:00+00:00 doesn't equal a big enough hole.
 Did I mention that the low pause collector maintains free lists for the space available in the tenured generation and that fragmentation can become a problem? If you're using the low pause collector and things are going just peachy for days and days and then there is a huge (relatively speaking) pause, the cause may be fragmentation in the tenured generation.
 In 1.4.2 and older releases in order to do a young generation collection there was a requirement that there be a contiguous chunk of free space in the tenured generation that was big enough to hold all the the young generation. In the GC tuning documents at
 http://java.sun.com/docs/hotspot/
 this is referred to as the young generation guarantee. Basically during a young generation collection, any data that survives may have to be promoted into the tenured generation and we just don't know how much is going to survive. Being our usual conservative selves we assumed all of it would survive and so there needed to be room in the tenured generation for all of it. How does this cause a big pause? If the young generation is full and needs to be collected but there is not enough room in the tenured generation, then a full collection of both the young generation and the tenured generations are done. And this collection is a stop-the-world collection not a concurrent collection so you generally see a pause much longer than you want to. By the way this full collection is also a compacting collection so there is no fragmentation at the end of the full collection.
 In 5.0 we added the ability in the low pause collector to start a young generation collection and then to back out of it if there was not enough space in the tenured generation. Being able to backout of a young generation collection allowed us to make a couple of changes. We now keep an average of the amount of space that is used for promotions and use that (with some appropriate padding to be on the safe side) as the requirement for the space needed in the tenured generation. Additionally we no longer need a single contiguous chunk of space for the promotions so we look at the total amount of free space in the tenured generation in deciding if we can do a young generation collection. Not having to have a single contiguous chunk of space to support promotions is where fragmentation comes in (or rather where it doesn't come in as often). Yes, sometimes using the averages for the amount promoted and the total amount of free in the tenured generation tells us to go ahead and do a young generation collection and we get surprised (there really isn't enough space in tenured generation). In that situation we have to back out of the young generation collection. It's expensive to back out of a collection, but it's doable. That's a very long way of saying that fragmentation is less of a problem in 5.0. It still occurs, but we have better ways of dealling with it.
 What should you do if you run into a fragmentation problem?
 Try 5.0.
 Or you could try a larger total heap and/or smaller young generation. If your application is on the edge, it might give you just enough extra space to fit all your live data. But often it just delays the problem.
 Or you can try to make you application do a full, compacting collection at a time which will not disturb your users. If your application can go for a day without hitting a fragmentation problem, try a System.gc() in the middle of the night. That will compact the heap and you can hopefully go another day without hitting the fragmentation problem. Clearly no help for an application that does not have a logical "middle of the night".
 Or if by chance most of the data in the tenured generation is read in when your application first starts up and you can do a System.gc() after you complete initialization, that might help by compacting all data into a single chunk leaving the rest of the tenured generation available for promotions. Depending on the allocation pattern of the application, that might be adequate.
 Or you might want to start the concurrent collections earlier. The low pause collector tries to start a concurrent collection just in time (with some safety factor) to collect the tenured generation before it is full. If you are doing concurrent collections and freeing enough space, you can try starting a concurrent collection sooner so that it finishes before the fragmentation becomes a problem. The concurrent collections don't do a compaction, but they do coalese adjacent free blocks so larger chunks of free space can result from a concurrent collection. One of the triggers for starting a concurrent collection is the amount of free space in the tenured generation. You can cause a concurrent collection to occur early by setting the option -XX:CMSInitiatingOccupancyFraction=<NNN> where NNN is the percentage of the tenured generation that is in use above which a concurrent collection is started. This will increase the overall time you spend doing GC but may avoid the fragmentation problem. And this will be more effective with 5.0 because a single contiguous chunk of space is not required for promotions.
 By the way, I've increased the comment period for my blogs. I hadn't realized it was so short. https://blogs.oracle.com/jonthecollector/entry/so_what_s_it_worth So What's It Worth to You? jonthecollector 2006-02-15T14:08:43+00:00 2006-02-22T23:29:16+00:00 Have you ever gone to your favorite restaurant, had a great meal and sat back with your second cup of coffee and asked "Why can't those guys do a pausless garbage collector? How hard can it be?"
 Well, you're not the only one. That question came up around here recently.
 There are techniques for doing pauseless garbage collection. The problem is that it costs in terms of application performance or additional Java heap size. Threads in an application read and write objects. The garbage collector reads and writes (i.e., frees or moves) objects. All those reads and writes have to have some coordination. "Pauseless" GC very often means that the application as a whole is not paused, but individual threads are paused one at a time. Imagine two application threads that are furiously playing hot potato with an object. As soon as thread 1 passes the object A to thread 2, thread 1 forgets about A and thread 2 realizes it has A. Then thread 2 immediately passes it back to thread 1 with the analogous result. A pauseless collector pauses thread 1 to look at what objects it is holding, then restarts thread 1 (so only one thread is paused at a time) and pauses thread 2 to examine it. Did object A get passed from thread 2 to thread 1 in that window during which both were executing? If it did and that is the only reference to object A, then the collector will not know that object A is live without doing more work.
 I'm going to skip the details here. The point of this note is not to delve into the particular difficulties of pauseless GC, but rather to give you an idea of why it can be more costly.
 Also I'm going to consider collectors that compact objects (i.e., move them). Mostly because it's an easier example to talk about. Some collectors don't move objects during a collections. Not moving objects simplifies some aspects of a collection but has its own complications (e.g., maintaining free lists of objects, splitting and coalescing blocks of free space, and fragmentation).
 If the collector does compact the objects, it has to be sure that all references to live objects point to their new locations. A common way to catch object A when it has slipped through the window is to use what is fondly referred to as a read barrier. Any read of an object by the application goes through code (the read barrier) that looks at the object and decides whether something special has to be done. The "something special" depends on the particulars of the collector. Without getting into those particulars just having to do extra stuff on every read has got to hurt. Yes, I'm really over simplifying this example, but what can you expect from my really simple mind.
 So what's pauseless GC worth to you (i.e., in terms of the extra space and performance costs, the extra complexity, maybe special hardware)? It's definitely worth a bunch to some. But for many it isn't really necessary to have truly pauseless GC. Shorter is good enough.
 Why can't those guys do a pauseless garbage collector?
 Would "pauseless garbage collection" give us the biggest bang for the buck (i.e., most benefits to most user)? When we were deciding what to do for the next release, we asked ourselves that question. And we decided there were things we should do before "pauseless GC". Did I mention that I recently worked on the parallel collection of the tenured generation for the throughput collector? I'll tell you about it some time. https://blogs.oracle.com/jonthecollector/entry/tell_us_where_it_hurts Tell Us Where It Hurts jonthecollector 2006-02-01T09:35:56+00:00 2008-02-20T17:58:06+00:00 Here's a URL for asking GC questions.
 http://java.sun.com/docs/forms/gc-sendusmail.html
 Engineers in the GC group answer the questions so that's probably as good as it gets.
 Updated 2008/2/20
 The link above no longer takes you to the form for asking GC related questions. I'm leaving this information above just to save a little bit of confusion (i.e., you won't find yourself saying "Didn't this page used to ..."). The new channel for asking questions is explained below.
 Questions for the Hotspot[tm] JVM[tm] garbage collector group should be sent to the hotspot-gc-use mailing list. This is a mailing list that is open to the public. Please see the link below for information on subscribing to that mailing list.
 http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
 If you have questions which you would prefer not be answered in public, please work through you Sun support contact.
 Information on Sun support can be found at
 http://developers.sun.com/services/ https://blogs.oracle.com/jonthecollector/entry/why_not_a_grand_unified Why not a Grand Unified Garbage Collector? jonthecollector 2006-01-26T17:23:17+00:00 2006-01-27T01:23:17+00:00 At last count we have three garbage collectors.
 <l>
 the parallel collector
 the low pause collector
 the serial collector </l>
 Why is there more than one? Actually, the usual answer applies. Specialization often results in better performance. If you're interested in more particulars about our garbage collectors, read on.
 All three collectors are generational (young generation and tenured generation).
 Let's do the easy comparison first, why a parallel collector and a serial collector. Parallelism has overhead. Nuf said? Yeah, I used to read comic books when I was a kid. If you don't understand that reference, ignore it. I'm just older.
 As you might infer from the names, the serial collector uses 1 thread to do the GC work and the parallel collector uses multiple threads to do the same. As usual multiple threads doing the same tasks have to synchronize. That's pretty much it. On a single cpu machine the additional cost of the synchronization means that the parallel collector is slower than the serial collector. On a two cpu machine and a VM that has a small heap the parallel collector is as about fast as the serial collector. With two cpu's and large heaps the parallel collector will usually do better. We keep asking ourselves if we can get rid of the serial collector and use the parallel collector in its place. The answer so far keeps coming back no.
 More interesting is the case of the low pause collector versus the parallel collector. Above I made the remark about specialization and better performance. This is actually a case of more complexity and lesser performance. These two collectors do the collection of the young generation using almost the exact same techniques The differences in the collectors have to do with the collections of the tenured generation. The low pause collector does parts of that collection while the application continues to run. One way to do that is to not move the live objects when collecting the dead objects. The application tends to get confused if the objects it is using move around while the application is running. The other two collectors compact the heap during a collection of the tenured generation (i.e., live objects are moved so as to occupy one contiguous region of the heap). The low pause collector collects the dead objects and coalesces their space into blocks that are kept in free lists. Maintaining free lists and doing allocations from them takes effort so it's slower than having a heap that is compacted. Having applications run while a collection is happening means that new objects can be allocated during a collection. That leads to so more complexity. Also the collection of the tenured generation can be interrupted for a collection of the young generation. More complexity still. The bottom line is that the low pause collector has shorter GC pauses but it costs performance. That performance difference is not huge but it's large enough to keep us from ditching the parallel collector and always using the low pause collector.
 And last but not least, can we replace the serial collector with the low pause collector? Very tempting. The serial collector is used by default on desktop machines. We expect those to have 1 or 2 cpus and to be running applications that need 10's of megabytes of Java heap as opposed to 100's of megabytes. With small heaps the differences in collection times tend to make less difference. Even if the low pause collector was 10% slower than the serial collector, the difference between, for example, 70 ms and 77 ms often isn't large enough to matter. It would probably be a done deal except that the low pause collector has a larger memory footprint. It has additional data structures that it uses (for example to keep track of what references are being changed by the the running application while a collection is on going). It also usually needs a larger Java heap to run an application. Recall that the low pause collector uses free lists to keep track of the available space in the heap. Fragmentation can become a problem. The best bet is that we'll replace the serial collector with the low pause collector some day but not just yet. https://blogs.oracle.com/jonthecollector/entry/a_little_thread_privacy_please A Little Thread Privacy Please jonthecollector 2006-01-13T14:14:25+00:00 2006-01-13T22:14:25+00:00 This is not really about garbage collection but hopefully you'll find it interesting. And there is no punch line (i.e., cool command line flag that I can suggest to make your application run better). It's just a story about letting the VM do it.
 If you're using multiple threads (or one of the Java libraries is using multiple threads in your behalf) then threads in your application are doing allocations concurrently. All the threads are allocating from the same heap so some allowances have to be made for simultaneous allocations by 2 or more threads. In the simplest case (other than the case where no allowances are made and which is just plain wrong) each thread would grab a lock, do the allocation and release the lock. That gives the right answer but is just too slow. The slightly more complicated means would be to use some atomic operation (such as compare-and-swap) plus a little logic to safely do the allocation. Faster but still too slow. What is commonly done is to give each thread a buffer that is used exclusively by that thread to do allocations. You have to use some synchronization to allocate the buffer from the heap, but after that the thread can allocate from the buffer without synchronization. In the hotspot JVM we refer to these as thread local allocation buffers (TLAB's). They work well. But how large should the TLAB's be?
 Prior to 5.0 that was a difficult question to answer. TLAB's that were too large wasted space. If you had tons of threads and large TLAB's, you could conceivably fill up the heap with buffers that were mostly unused. Creating more threads might force a collection which would be unfortunate because the heap was mostly empty. TLAB's that were too small would fill quickly and would mean having to get a new TLAB which would require some form of synchronization. There was not a general recommendation that we could make on how large TLAB's should be. Yup, we were reduced to trial-and-error.
 Starting with 5.0 living large with TLAB's got much simpler - except for the guy down the hall that did the implementation. Here's what the VM does for you.
 Each thread starts with a small TLAB. Between the end of the last young generation collection and the start of the next (let me call that period an epoch), we keep track of the number of TLAB's a thread has used. We also know the size of the TLAB's for each thread. Averages for each thread are maintained for these two numbers (number and size of TLAB's). These averages are weighted toward the most recent epoch. Based on these averages the sizes of the TLAB's are adjusted so that a thread gets 50 TLAB's during a epoch. Why 50? All things being equal we figured that a thread would have used half its TLAB by the end of the epoch. Per thread that gave us 1/2 a TLAB not used (out of the magic 50) for a wastage of 1%. That seemed acceptable. Also if the young generation was not large enough to provide the desired TLAB's, the size of the young generation would be increased in order to make it so (within the usual heap size constraints, of course).
 The initial TLAB size is calculated from the number of threads doing allocation and the size of the heap. More threads pushes down the initial size of the TLAB's and larger heaps push up the initial size.
 An allocation that can not be made from a TLAB does not always mean that the thread has to get a new TLAB. Depending on the size of the allocation and the unused space remaining in the TLAB, the VM could decide to just do the allocation from the heap. That allocation from the heap would require synchronization but so would getting a new TLAB. If the allocation was considered large (some significant fraction of the current TLAB size), the allocation would always be done out of the heap. This cut down on wastage and gracefully handled the much-larger-than-average allocation. https://blogs.oracle.com/jonthecollector/entry/since_it_s_not_magic Since It's Not Magic jonthecollector 2006-01-02T13:29:09+00:00 2006-01-02T21:29:09+00:00
 So by now you've tried GC ergonomics and things are working pretty well. If you didn't have any GC related command line flags, your reactions could run the gamut from the "big yawn" to an eye-popping "that's cool". The "big yawn" means that GC probably doesn't matter for your application. The "that's cool" means that GC matters but nobody ever told you. If you did have a bunch of GC related command line flags and you threw them out in order to try GC ergonomics, then 80% of you are happy and just glad to forget those command line flags. But 20% of you are thinking "Hmmm, not quite as good". So what else do you need to know about GC ergonomics? By the way I invoke the 80/20 rules quite often, especially when I have no idea what the real numbers are.
 By default ergonomics gives you a larger maximum heap size. "Larger" is relative to the default maximum heap size you got prior to ergonomics in 5.0. This larger maximum heap size depends on the amount of physical memory on the system (the more physical memory you have, the larger the maximum heap size you get), but it is capped at 1G. If you had a command line flag that sets a larger maximum heap size, put it back. Ergonomics just doesn't know you're in the 20% that needs a larger heap. If you set a larger maximum heap size, ergonomics will still grow and shrink the generations for better performance but now will have more space to play with. By the way if you set a smaller maximum heap size, ergonomics still does its stuff but just within a smaller heap.
 Ergomonics has a default for the relative sizes of the young generation and the tenured generation in the heap. If you have command line flags to specify the sizes of the generations (probably something like -XX:NewRatio=ratio or -XX:MaxNewSize=bytes), use them. GC ergonomics normally does not change the maximum size of a generation so if you know something about how your application runs such that you know it benefits from a larger or smaller young generation, pass that information along on the command line. Again GC ergonomics just works within the limits of the heap it's given.