From kato-spec-returnlisa23@example.net Mon Jan 04 14:57:36 2010 Return-Path: <kato-spec-returnlisa23@example.net> Delivered-To: lisa23@example.net Received: (qmail 44030 invoked from network); 4 Jan 2010 14:57:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2010 14:57:35 -0000 Received: (qmail 31821 invoked by uid 500); 4 Jan 2010 14:57:35 -0000 Delivered-To: lisa23@example.net Received: (qmail 31783 invoked by uid 500); 4 Jan 2010 14:57:35 -0000 Mailing-List: contact lisa23@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:lisa23@example.net> List-Unsubscribe: <mailto:lisa23@example.net> List-Post: <mailto:lisa23@example.net> List-Id: <kato-spec.incubator.apache.org> Reply-To: lisa23@example.net Delivered-To: mailing list lisa23@example.net Received: (qmail 31773 invoked by uid 99); 4 Jan 2010 14:57:35 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 14:57:35 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lisa23@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-bw0-f210.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 14:57:26 +0000 Received: by bwz2 with SMTP id 2so8603534bwz.20 for <lisa23@example.net>; Mon, 04 Jan 2010 06:57:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=J+z7kY/iTsp1Rq+0faepVVKC3FXQT6NtSQwmpNW9Jh4=; b=Puk353QSg8jfUzUCTSeg4+mDtVyVN6noX6a7PoYAJGXhI3e8Fq6zikLXKnCcKSZh7G 5cvWP2Nq1H21qbgKBcCw9SLD5ZCcjBjql4OTXSFFV3dI6DyUe+qyjXL7keb8jIsVEdaS H35ilh4mLRFYMWxifpD92oMlbke/LxiAx/saw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=Fp2gkTZI3m8Yp0kvpCVUpUS2cvNpcrU0h2n0Ap8+SPjC9LPKmXS5mMfHnbQbjFoJ0f A6rlNTvIfG2oUgHhqyM2lE8iIbV7jV4NjGZ/WXPQx/VY8QeSNLdRHE/Xte5NKTd9G8WW IBmlCd4UoPfiBau34M/YAyBA+MzS+Uscd7NHs= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id z4mr7496843bkx.86.1262617024890; Mon, 04 Jan 2010 06:57:04 -0800 (PST) Date: Mon, 4 Jan 2010 14:57:04 +0000 Message-ID: <lisa23@example.net> Subject: Happy New Year From: Steve Poole <lisa23@example.net> To: lisa23@example.net Content-Type: multipart/alternative; boundary=000325553716418851047c57ef35 --000325553716418851047c57ef35 Content-Type: text/plain; charset=ISO-8859-1 Hi everyone - greetings at the start of a new year. I hope you all had a good vacation. As far as our work on JSR 326 is concerned we need to pick up speed on the Snapshot API design and move it to a conclusion. I have been working on a pure Java prototype implementation to help make it easier for us all to understand and experiment with this design. Discussion on that prototype will happen on the dev list. I'm going to post a couple of use cases for the Snapshot API which are more specific to application development rather than the monitoring ideas posted earlier. We need to include both types of information of course. Once we start the creative thoughts moving again I will organise a conference call where we can review our status. Cheers -- Steve --000325553716418851047c57ef35-- From kato-spec-returnlisa23@example.net Mon Jan 04 15:19:42 2010 Return-Path: <kato-spec-returnlisa23@example.net> Delivered-To: lisa23@example.net Received: (qmail 51366 invoked from network); 4 Jan 2010 15:19:42 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2010 15:19:42 -0000 Received: (qmail 53548 invoked by uid 500); 4 Jan 2010 15:19:42 -0000 Delivered-To: lisa23@example.net Received: (qmail 53512 invoked by uid 500); 4 Jan 2010 15:19:42 -0000 Mailing-List: contact lisa23@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:lisa23@example.net> List-Unsubscribe: <mailto:lisa23@example.net> List-Post: <mailto:lisa23@example.net> List-Id: <kato-spec.incubator.apache.org> Reply-To: lisa23@example.net Delivered-To: mailing list lisa23@example.net Received: (qmail 53502 invoked by uid 99); 4 Jan 2010 15:19:42 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 15:19:42 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lisa23@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-bw0-f210.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 15:19:33 +0000 Received: by bwz2 with SMTP id 2so8623317bwz.20 for <lisa23@example.net>; Mon, 04 Jan 2010 07:19:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=0zE+Uwe5ESeCj4nMsDK0SZIemNJ1sHOwU3Bi6LEZx8M=; b=ZDzNOKyd+5Y0McuTE4PSNalzuuOM1WvOBKxxIJTHWAljf39PqQmwloYEPTyxxXHyt/ OeG8jvD/ArMyx+2bHU8GuhLR/tM5DP+5E7WFS+XLiXHWQeb8qupHiZ1gxyfAJWaTz111 Mc3rMn3YIkLjK9K/JxKAvC18n2x07c+wTmLXQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=guSD5vYCgmITvkUVZt06qoiijdKjdob2nSI2cn3nL+BKP3OBA/cqEcJvxVPcCv2QSl ku4L4NZbM8cwMKRIMq4KE9opdGT/cVxZhtT9KhZYigG4JN2ws1SEziyja6lpgR9zLru5 QkASfTKhtAsjUE6rP27rGDxabJ2UOinbU+VNA= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id i3mr3342215bka.71.1262618351163; Mon, 04 Jan 2010 07:19:11 -0800 (PST) Date: Mon, 4 Jan 2010 15:19:11 +0000 Message-ID: <lisa23@example.net> Subject: Snapshot Application development usecases From: Steve Poole <lisa23@example.net> To: lisa23@example.net Content-Type: multipart/alternative; boundary=00032555a0be4edacb047c583e42 --00032555a0be4edacb047c583e42 Content-Type: text/plain; charset=ISO-8859-1 I want to present a few application development usecases for us to discuss to do with the Snapshot concept. First let's agree some basics An application snapshot is a deliberate subset of a running system. It's deliberate in that what is defined for inclusion / exclusion can be changed. To build an application snapshot three types of information are required 1 - root objects which determine where to start the snapshot from 2 - rules to restrict how "far" to search 3 - rules to describe what to include in the snapshot Example 1 A webserver session is behaving incorrectly. The application programmer wants to capture information to do with the session itself and some specific application details. The root object is an HTTPSession and the application code is all contained in packages that start org.acme.application The application programmer believes that the problem is in the application and so wants the minimum of system information included. Example 2 An error has occured and the application is about throw an exception. The application programmer wants to capture a series of objects related to the error and wants to see the Java stack including local variables. Example 3 A enterprise server is running slow and the application programmer would like to get a list of the instances of a specific interface that the server is managing, since the suspicion is that there are multiple instances of a nonthreadsafe instance of this interface when there should only be one. Example 4 A servlet server has run out of database connections and the suspicion is that managed objects have not been collected. Typically programmers use finalisers or depend on GC instead of using the Servlet.Destroy() method. the application programmer needs to get a list of objects in "Destroyed" state which obviously haven't been GC'd Cheers -- Steve --00032555a0be4edacb047c583e42-- From kato-spec-returnlisa23@example.net Mon Jan 04 17:45:22 2010 Return-Path: <kato-spec-returnlisa23@example.net> Delivered-To: lisa23@example.net Received: (qmail 2320 invoked from network); 4 Jan 2010 17:45:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2010 17:45:22 -0000 Received: (qmail 19866 invoked by uid 500); 4 Jan 2010 17:45:22 -0000 Delivered-To: lisa23@example.net Received: (qmail 19829 invoked by uid 500); 4 Jan 2010 17:45:22 -0000 Mailing-List: contact lisa23@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:lisa23@example.net> List-Unsubscribe: <mailto:lisa23@example.net> List-Post: <mailto:lisa23@example.net> List-Id: <kato-spec.incubator.apache.org> Reply-To: lisa23@example.net Delivered-To: mailing list lisa23@example.net Received: (qmail 19819 invoked by uid 99); 4 Jan 2010 17:45:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 17:45:22 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_HELO_PASS,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO moutng.kundenserver.de) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 17:45:12 +0000 Received: from [83.161.120.155] (gbibp9ph1--blueice1n1.emea.ibm.com [83.161.120.155]) by mrelayeu.kundenserver.de (node=mreu2) with ESMTP (Nemesis) id 0MIAjy-1NUbLJ3qgg-003sSz; Mon, 04 Jan 2010 18:44:52 +0100 Message-ID: <lisa23@example.net> Date: Mon, 04 Jan 2010 17:44:51 +0000 From: Stuart Monteith <lisa23@example.net> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:83.161.120.155) Gecko/20091206 SeaMonkey/2.0.1 MIME-Version: 1.0 To: lisa23@example.net Subject: Re: Snapshot Application development usecases References: <lisa23@example.net> In-Reply-To: <lisa23@example.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V01U2FsdGVkX19Pbm90WnoIxsSYxgjSTcKHl0fp/3Bi92jA9JF 9hnk0JdPBFuDiShEk8HzL5z/bwVcE5WBz3WrToRuQbwOkNsKBR 07vHJWwv2LN7QQpk8wnqSYoYICTz/na X-Virus-Checked: Checked by ClamAV on apache.org Happy New Year everyone! The usecases are interesting, the problem is how to find and dump what you want and no more. Ways in which objects may be selected for dumping: o By classloader This is can separate object instances at runtime by virtue of their classes having been defined by the same classloader. o By class package Simply filter dumped classes matching a given filter name. o By class annotation Classes could be specially marked with an annotation that is lisa23@example.net. o By object instance. A dump would have to be initiated /somewhere/. Realistically, we'd have to start from a particular object, class. An array could be passed to cater for multiple roots. o By Thread. Using a thread and all of its roots would be practicable if they were accessible. o By field value: Restrict dump to objects with a field containing a certain value, lisa23@example.net. o By distance from the root object. For example, given an object, only retrieve objects three references away. These could be used in various combinations. Dumping all objects of classes in package com.silly.donkey is all very well, but this could be restricted to only those annotated with @DumpMe and further restricted to only those reachable from a particular instance of "com.silly.donkey.HeeHaw" . Regards, Stuart Steve Poole wrote: > I want to present a few application development usecases for us to discuss > to do with the Snapshot concept. > > First let's agree some basics > > An application snapshot is a deliberate subset of a running system. It's > deliberate in that what is defined for inclusion / exclusion can be changed. > > To build an application snapshot three types of information are required > > 1 - root objects which determine where to start the snapshot from > 2 - rules to restrict how "far" to search > 3 - rules to describe what to include in the snapshot > > > Example 1 > > A webserver session is behaving incorrectly. The application programmer > wants to capture information > to do with the session itself and some specific application details. > > The root object is an HTTPSession and the application code is all contained > in packages that start org.acme.application > The application programmer believes that the problem is in the application > and so wants the minimum > of system information included. > > > Example 2 > > An error has occured and the application is about throw an exception. The > application programmer wants to capture a series > of objects related to the error and wants to see the Java stack including > local variables. > > > Example 3 > > A enterprise server is running slow and the application programmer would > like to get a list of the instances of a specific interface > that the server is managing, since the suspicion is that there are multiple > instances of a nonthreadsafe instance of this interface when there should > only be one. > > Example 4 > > A servlet server has run out of database connections and the suspicion is > that managed objects have not been > collected. Typically programmers use finalisers or depend on GC instead of > using the Servlet.Destroy() method. > the application programmer needs to get a list of objects in "Destroyed" > state which obviously haven't been GC'd > > > Cheers > > -- Stuart Monteith http://blog.stoo.me.uk/ From kato-spec-returnlisa23@example.net Mon Jan 04 17:53:56 2010 Return-Path: <kato-spec-returnlisa23@example.net> Delivered-To: lisa23@example.net Received: (qmail 6107 invoked from network); 4 Jan 2010 17:53:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2010 17:53:56 -0000 Received: (qmail 30177 invoked by uid 500); 4 Jan 2010 17:53:56 -0000 Delivered-To: lisa23@example.net Received: (qmail 30141 invoked by uid 500); 4 Jan 2010 17:53:56 -0000 Mailing-List: contact lisa23@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:lisa23@example.net> List-Unsubscribe: <mailto:lisa23@example.net> List-Post: <mailto:lisa23@example.net> List-Id: <kato-spec.incubator.apache.org> Reply-To: lisa23@example.net Delivered-To: mailing list lisa23@example.net Received: (qmail 30131 invoked by uid 99); 4 Jan 2010 17:53:56 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 17:53:56 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of lisa23@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-ew0-f220.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 17:53:48 +0000 Received: by ewy20 with SMTP id 20so3892025ewy.20 for <lisa23@example.net>; Mon, 04 Jan 2010 09:53:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=wjLMhtnEF69xqcYKyQKRTbjVc2rJ3+8O7x7CYJjHMbQ=; b=gvBdanOjyj0GilD1DiAtk0YoiJhVPMNSEjwlPm8LM6PRq3XdTxyTrbSZbQtvuRfBno VWoCP4dmX4GHkm0EfEZBu1vaKBt1aom5I7wdoJc3LutGjEzFo1/r5MsMbjrS/3Kfdhqp QhetD0COzVgwXaXjqJJlXF863vNqK1qcjP0hk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=l+W7Zkwppx9bjF6nj7Y3XiGm4+LoSXG+cIOOoux6eTWYuI86kDfgEbtU0HCp5pKAud aazIB9gVSalYHcn4FclqiR9mNgciWbjseMpPjbYxH16c3C28HkpAfJEbRbh2S0qETXtw K+H6sCfYq+Ps3XcMkjQEp54Nl4z1cMsKX1FtQ= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id v15mr10585170ebn.73.1262627608287; Mon, 04 Jan 2010 09:53:28 -0800 (PST) In-Reply-To: <lisa23@example.net> References: <lisa23@example.net> Date: Mon, 4 Jan 2010 17:53:28 +0000 Message-ID: <lisa23@example.net> Subject: Re: Snapshot Application development usecases From: David Griffiths <lisa23@example.net> To: lisa23@example.net Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Greetings. My name is Dave Griffiths and I've worked in IBM's Java Service group for many years. I have experience with Kato's predecessor DTFJ and various related technologies. I've been meaning to subscribe to this mailing list for some time so apologies if I'm a bit late to the party. Just some random thoughts on the concept of snapshot dumps. Problem analysis seems to fall into three main categories: 1) high cpu/deadlock where a simple snap of the thread stacks and monitors is all that's required. (Note: it's also vital to have the native stacks - for some reason this has often been problematic. From a debugging point of view getting this right dominates a lot of other stuff in terms of importance). 2) Java OOM conditions. For the vast majority of OOMs, a simple heap dump is all that is required. This should include the heap roots, class loaders and refs from one object to another but things like non-reference fields do not need to be included (see next category). (Disclaimer: I created the IBM PHD format which is clearly showing its age. But at the time it was thought to be important to have a fast and lightweight format). 3) Other. For more complex problem determination that doesn't fit into the above categories you almost always want a complete system dump. Storage is cheap and size is not an issue. Often the person who collects the dump (whether test or at a customer site) is not the person who later analyzes it. There is nothing worse than being given a dump that doesn't contain everything you need. So I'm a great fan of system dumps even for things like OOM failures (eg you get people saying "well can we see the contents of that String object?") The main thing is a nice easy API to let you get at the stuff you need in the dump. So for your four examples, for example 1 I would say something like the current java thread dump would suffice (depending on what you mean by behaving incorrectly - otherwise a system dump). Example 2 - not sure what you mean by "about to throw". Isn't Kato meant to be for post-mortem dump analysis? We have a way (at least at IBM) to force a system dump when an exception occurs and that's one way of doing it, but most app developers would either use logging or a debugger in this scenario? Example 3 - a heapdump should suffice. Example 4 - use a system dump since you want to see the non-reference fields of an object which are not stored in a heapdump. The format of the dump (thread, heap, system) is immaterial - the main thing is the consistent API to read them. So personally I would just stick to those three dump types and not try to over-egg the pudding. Cheers, Dave On Mon, Jan 4, 2010 at 3:19 PM, Steve Poole <lisa23@example.net> wrot= e: > I want to present a few application development usecases for us to discus= s > to do with the Snapshot concept. > > First let's agree some basics > > An application snapshot is a deliberate subset of a running system. =A0It= 's > deliberate in that what is defined for inclusion / exclusion can be chang= ed. > > To build an application snapshot three types of information are required > > 1 - root objects which determine where to start the snapshot from > 2 - rules to restrict how "far" to search > 3 - rules to describe what to include in the snapshot > > > Example 1 > > A webserver session is behaving incorrectly. =A0The application programme= r > wants to capture information > to do with the session itself and some specific application details. > > The root object is an HTTPSession and the application code is all contain= ed > in packages that start org.acme.application > The application programmer believes that the problem is in the applicatio= n > and so wants the minimum > of system information included. > > > Example 2 > > An error has occured and the application is about throw an exception. =A0= The > application programmer wants to capture a series > of objects related to the error and wants to see the Java stack including > local variables. > > > Example 3 > > A enterprise server is running slow and the application programmer would > like to get a list of the instances of a specific interface > that the server is managing, since the suspicion is that there are multip= le > instances of a nonthreadsafe instance of this interface when there should > only be one. > > Example 4 > > A servlet server has run out of database connections and the suspicion is > that managed objects have not been > collected. =A0Typically programmers use finalisers or depend on GC instea= d of > using the Servlet.Destroy() method. > the application programmer needs to get a list of objects in "Destroyed" > state which obviously haven't been GC'd > > > Cheers > > -- > Steve > From kato-spec-returnlisa23@example.net Mon Jan 04 22:23:57 2010 Return-Path: <kato-spec-returnlisa23@example.net> Delivered-To: lisa23@example.net Received: (qmail 6751 invoked from network); 4 Jan 2010 22:23:57 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2010 22:23:57 -0000 Received: (qmail 13754 invoked by uid 500); 4 Jan 2010 22:23:57 -0000 Delivered-To: lisa23@example.net Received: (qmail 13667 invoked by uid 500); 4 Jan 2010 22:23:57 -0000 Mailing-List: contact lisa23@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:lisa23@example.net> List-Unsubscribe: <mailto:lisa23@example.net> List-Post: <mailto:lisa23@example.net> List-Id: <kato-spec.incubator.apache.org> Reply-To: lisa23@example.net Delivered-To: mailing list lisa23@example.net Received: (qmail 13646 invoked by uid 99); 4 Jan 2010 22:23:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 22:23:57 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lisa23@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-bw0-f210.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 22:23:48 +0000 Received: by mail-bw0-f210.google.com with SMTP id 2so8932532bwz.20 for <lisa23@example.net>; Mon, 04 Jan 2010 14:23:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=Tj4r3EXxz9EQ3zHVldqnousdd3vEVhr76P0PKrXzWCg=; b=mggDD0z2OqTwYSSnRfn8+ogW+JY5U0K23Ufx7ys432oLQQ+Ebfa8zv0aSfdkejfd8k s+vQmkDupzDvvepLHnzM/zMJc8XGiWiVmyWJ8w9KYzGMm3+WUOLUrMIBsv3ycq6J4fTu rAeJrq+aiBEXMN2sht8MvXk27zatLkyL+mzis= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=GD/5NZo6cYeNbx0TujmUt1vvo8tMxkhTadgUjG4AphUpbdaKVnVoOx+rvYzkfBrvTZ QVCCxVeFe6IoBHHBHQ6EogcwH+P4GMYPauUThwFddDIVJv+oFUaW2dmPuO322zNPnpbt V+TYU/aezLKN1Np/jfguDFVF/Azak9bV2bSp0= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id 15mr3720270bkw.89.1262643806811; Mon, 04 Jan 2010 14:23:26 -0800 (PST) In-Reply-To: <lisa23@example.net> References: <lisa23@example.net> <lisa23@example.net> Date: Mon, 4 Jan 2010 22:23:26 +0000 Message-ID: <lisa23@example.net> Subject: Re: Snapshot Application development usecases From: Steve Poole <lisa23@example.net> To: lisa23@example.net Content-Type: multipart/alternative; boundary=00151758864e953883047c5e2ba5 --00151758864e953883047c5e2ba5 Content-Type: text/plain; charset=ISO-8859-1 Thanks Stuart - like the donkey. I would imagine, based on your examples, having methods like selectClassesLoadedBy(ClassLoader c) selectPackageName("com.silly.donkey.*") selectByAnnotation(Annotation a) selectReferencedObjects(Thread t); addFieldFilter(java.lang.reflect.Field f,Object value); selectReferencedObjectsWithin(Object root,int distance); I like to get down the the method definitions as quickly as possible as it makes it real :-) On Mon, Jan 4, 2010 at 5:44 PM, Stuart Monteith <lisa23@example.net> wrote: > Happy New Year everyone! > > The usecases are interesting, the problem is how to find and dump what you > want and no more. > > Ways in which objects may be selected for dumping: > > o By classloader > This is can separate object instances at runtime by virtue of their > classes having been > defined by the same classloader. > > o By class package > Simply filter dumped classes matching a given filter name. > > o By class annotation > Classes could be specially marked with an annotation that is visible > at runtime. > > o By object instance. > A dump would have to be initiated /somewhere/. Realistically, we'd > have to start from a particular > object, class. An array could be passed to cater for multiple roots. > > o By Thread. > Using a thread and all of its roots would be practicable if they > were accessible. > > o By field value: > Restrict dump to objects with a field containing a certain value, > lisa23@example.net. > > o By distance from the root object. > For example, given an object, only retrieve objects three references > away. > > These could be used in various combinations. Dumping all objects of classes > in package com.silly.donkey > is all very well, but this could be restricted to only those annotated with > @DumpMe and further restricted > to only those reachable from a particular instance of > "com.silly.donkey.HeeHaw" . > > > Regards, > Stuart > > > Steve Poole wrote: > >> I want to present a few application development usecases for us to discuss >> to do with the Snapshot concept. >> >> First let's agree some basics >> >> An application snapshot is a deliberate subset of a running system. It's >> deliberate in that what is defined for inclusion / exclusion can be >> changed. >> >> To build an application snapshot three types of information are required >> >> 1 - root objects which determine where to start the snapshot from >> 2 - rules to restrict how "far" to search >> 3 - rules to describe what to include in the snapshot >> >> >> Example 1 >> >> A webserver session is behaving incorrectly. The application programmer >> wants to capture information >> to do with the session itself and some specific application details. >> >> The root object is an HTTPSession and the application code is all >> contained >> in packages that start org.acme.application >> The application programmer believes that the problem is in the application >> and so wants the minimum >> of system information included. >> >> >> Example 2 >> >> An error has occured and the application is about throw an exception. The >> application programmer wants to capture a series >> of objects related to the error and wants to see the Java stack including >> local variables. >> >> >> Example 3 >> >> A enterprise server is running slow and the application programmer would >> like to get a list of the instances of a specific interface >> that the server is managing, since the suspicion is that there are >> multiple >> instances of a nonthreadsafe instance of this interface when there should >> only be one. >> >> Example 4 >> >> A servlet server has run out of database connections and the suspicion is >> that managed objects have not been >> collected. Typically programmers use finalisers or depend on GC instead >> of >> using the Servlet.Destroy() method. >> the application programmer needs to get a list of objects in "Destroyed" >> state which obviously haven't been GC'd >> >> >> Cheers >> >> >> > > -- > Stuart Monteith > http://blog.stoo.me.uk/ > > -- Steve --00151758864e953883047c5e2ba5-- From kato-spec-returnlisa23@example.net Mon Jan 04 23:44:16 2010 Return-Path: <kato-spec-returnlisa23@example.net> Delivered-To: lisa23@example.net Received: (qmail 36579 invoked from network); 4 Jan 2010 23:44:16 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2010 23:44:16 -0000 Received: (qmail 99172 invoked by uid 500); 4 Jan 2010 23:44:16 -0000 Delivered-To: lisa23@example.net Received: (qmail 99134 invoked by uid 500); 4 Jan 2010 23:44:16 -0000 Mailing-List: contact lisa23@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:lisa23@example.net> List-Unsubscribe: <mailto:lisa23@example.net> List-Post: <mailto:lisa23@example.net> List-Id: <kato-spec.incubator.apache.org> Reply-To: lisa23@example.net Delivered-To: mailing list lisa23@example.net Received: (qmail 99124 invoked by uid 99); 4 Jan 2010 23:44:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 23:44:16 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lisa23@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-bw0-f210.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 23:44:07 +0000 Received: by bwz2 with SMTP id 2so8966678bwz.20 for <lisa23@example.net>; Mon, 04 Jan 2010 15:43:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=EYM/KnH3FRnEJyv4ytOanseb1wtIScxKae5YXUJ+uJ0=; b=Wx/o/C48Fp2edHgLLBXnkhaEYYgelhNWS8HZHe4X+DimnyA6TWlasX4Ola19kxg1RQ f+MHXiyr7pji8ISRYBMYUgYD3Nz7R2q2+dgagQFfaN+34aHTq841BetQoBq8zkEmrl2w yiptOqXAVLim/D2G1+tzktO2TyykayW7/z0Uk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=K28kUU3alI4kmfZp7XxzpJTaqWX1Gu6FKkTCNkNZu7HEprsnlBd8pAo6jM6egvgF6q HVPRRJHdW3/utFDvUzdUmBouCI/STmfIBcRNrCDQpLV3PO6ngI/8z7jLBHvy0CkquNcO jZEa/eZ+VB/35+iNdC3/mKkI8gbOrNFBZGavU= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id 16mr7730092bki.58.1262648625080; Mon, 04 Jan 2010 15:43:45 -0800 (PST) In-Reply-To: <lisa23@example.net> References: <lisa23@example.net> <lisa23@example.net> Date: Mon, 4 Jan 2010 23:43:45 +0000 Message-ID: <lisa23@example.net> Subject: Re: Snapshot Application development usecases From: Steve Poole <lisa23@example.net> To: lisa23@example.net Content-Type: multipart/alternative; boundary=0015174c1d20c629e9047c5f4a86 --0015174c1d20c629e9047c5f4a86 Content-Type: text/plain; charset=ISO-8859-1 Hi David and welcome to the party - never too late... I started writing this a quick note but it has turned into a sermon+rant. It's not aimed at you but hopefully it's still informative. I should write up a better introduction and post it to the wiki. For background you need to understand that we are not creating a new DTFJ. The main driver now is towards helping Java developers solve their application problems - not to help JVM developers solve JVM problems. Its a postmortem process for sure but we are considering how to address two other requirements : the need to deal with dumps that are getting larger and are already unwieldy plus the need to provide a subset of information from a running system for tend analysis. You also need to consider that we have to build a reference implementation for the API and since this is an Apache project we are doing that in an open way using open data. The proprietary information held within any IBM dump is inaccessible to us unless IBM chooses to publish the data. That's sort of true for the Sun JVM too - we can't write a dump reader that understands the intricacies of a Sun JVM in Apache as the code available to us is GPL. It would be possible for someone to start a GPL licenced project and produce the required code to match the JSR 326 API but that would be GPL code too so again of limited use to us. The two options available to us are a) a Licencee to the Sun JVM code writes an implementation and makes it available under a suitable licence b) We reduce our dependency on the data we need from a JVM to a level where it is reasonable to declare it an optional part of the API. I should also point out that none of the Sun JVM Licensees I've spoken to see the value in providing the capability of reading Java info from a corefile. It may be when the Sun/Oracle acquisition is completed that we can revisit this idea. For now we are left with having to deal with the dumps and APIs that we have today. If we need more information from the JVM than we have available today then we need to be considerate on how and where this information would be exposed to us. There is nothing wrong with us specifying APIs to be added or extended in the JVM but if they never get implemented it's a pointless exercise and could jeopardise the project. Where does that leave us? As of today the JSR will finesse out the relationship between the JavaRuntime API and the Image API so that its not a requirement for implementions to have to do the whole of the API. Since the only existing public dump format available to us is HPROF and that doesn't contain enough data for us the Apache Kato project is producing its own dump(s). Thats inline with what we needed to do anyway for the RI (to support the Snapshot requirements) its just a shame that we've not had the buy-in from JVM vendors we expected. The snapshot API is a runtime API which will be used to trigger dumps in a standard, programmatic, way and will allow an application programmer to restrict the data that will appear in the dump. We also need to consider what extra information (such as statistical and/or JVM runtime ) is needed but it not accessable to us today. The data selection mechanism has to be declaritive as we shouldn't dictate how the implementation would work. So this thread is about the mechanism that would be used in a running JVM by an application programmer to trigger a dump with just the information that they need to solve their particular problem. I want to work out how such an declarative mechanism would work in practise. More comments below On Mon, Jan 4, 2010 at 5:53 PM, David Griffiths <lisa23@example.net>wrote: > Greetings. My name is Dave Griffiths and I've worked in IBM's Java > Service group for many years. I have experience with Kato's > predecessor DTFJ and various related technologies. I've been meaning > to subscribe to this mailing list for some time so apologies if I'm a > bit late to the party. > > Just some random thoughts on the concept of snapshot dumps. > > Problem analysis seems to fall into three main categories: > > 1) high cpu/deadlock where a simple snap of the thread stacks and > monitors is all that's required. (Note: it's also vital to have the > native stacks - for some reason this has often been problematic. From > a debugging point of view getting this right dominates a lot of other > stuff in terms of importance). > > Yep - agree with this idea and the level of importance. How , as a reference implementation do we get at the native stacks? Note that Its possible , if we can keep the size of the requirement small enough ,to finesse this from the spec point of view and just make it an optional capability that the RI does not implement. > 2) Java OOM conditions. For the vast majority of OOMs, a simple heap > dump is all that is required. This should include the heap roots, > class loaders and refs from one object to another but things like > non-reference fields do not need to be included (see next category). > (Disclaimer: I created the IBM PHD format which is clearly showing its > age. But at the time it was thought to be important to have a fast and > lightweight format). > > Is there any reason why we might want to restrict the contents of dump that is being used to solve OOMs? If not then this is probably not something to consider for the Snapshot discussion. > 3) Other. For more complex problem determination that doesn't fit into > the above categories you almost always want a complete system dump. > Storage is cheap and size is not an issue. Often the person who > collects the dump (whether test or at a customer site) is not the > person who later analyzes it. There is nothing worse than being given > a dump that doesn't contain everything you need. So I'm a great fan of > system dumps even for things like OOM failures (eg you get people > saying "well can we see the contents of that String object?") The main > thing is a nice easy API to let you get at the stuff you need in the > dump. > > As I said at the beginning the system dump ( or at least any Java info in it) is inacessable to us. In this particular topic we are actually exploring how to define something that is by its very nature not a system dump :-) So for your four examples, for example 1 I would say something like > the current java thread dump would suffice (depending on what you mean > by behaving incorrectly - otherwise a system dump). > > Example 2 - not sure what you mean by "about to throw". Isn't Kato > meant to be for post-mortem dump analysis? We have a way (at least at > IBM) to force a system dump when an exception occurs and that's one > way of doing it, but most app developers would either use logging or a > debugger in this scenario? > > If you imagine - in IBM terms we are thinking about a sort of FFDC for application developers. Example 3 - a heapdump should suffice. > > Example 4 - use a system dump since you want to see the non-reference > fields of an object which are not stored in a heapdump. > > The format of the dump (thread, heap, system) is immaterial - the main > thing is the consistent API to read them. > > So personally I would just stick to those three dump types and not try > to over-egg the pudding. > > Cheers, > > Dave > > On Mon, Jan 4, 2010 at 3:19 PM, Steve Poole <lisa23@example.net> > wrote: > > I want to present a few application development usecases for us to > discuss > > to do with the Snapshot concept. > > > > First let's agree some basics > > > > An application snapshot is a deliberate subset of a running system. It's > > deliberate in that what is defined for inclusion / exclusion can be > changed. > > > > To build an application snapshot three types of information are required > > > > 1 - root objects which determine where to start the snapshot from > > 2 - rules to restrict how "far" to search > > 3 - rules to describe what to include in the snapshot > > > > > > Example 1 > > > > A webserver session is behaving incorrectly. The application programmer > > wants to capture information > > to do with the session itself and some specific application details. > > > > The root object is an HTTPSession and the application code is all > contained > > in packages that start org.acme.application > > The application programmer believes that the problem is in the > application > > and so wants the minimum > > of system information included. > > > > > > Example 2 > > > > An error has occured and the application is about throw an exception. > The > > application programmer wants to capture a series > > of objects related to the error and wants to see the Java stack including > > local variables. > > > > > > Example 3 > > > > A enterprise server is running slow and the application programmer would > > like to get a list of the instances of a specific interface > > that the server is managing, since the suspicion is that there are > multiple > > instances of a nonthreadsafe instance of this interface when there should > > only be one. > > > > Example 4 > > > > A servlet server has run out of database connections and the suspicion is > > that managed objects have not been > > collected. Typically programmers use finalisers or depend on GC instead > of > > using the Servlet.Destroy() method. > > the application programmer needs to get a list of objects in "Destroyed" > > state which obviously haven't been GC'd > > > > > > Cheers > > > > -- > > Steve > > > -- Steve --0015174c1d20c629e9047c5f4a86-- From kato-spec-returnlisa23@example.net Tue Jan 05 09:38:48 2010 Return-Path: <kato-spec-returnlisa23@example.net> Delivered-To: lisa23@example.net Received: (qmail 25713 invoked from network); 5 Jan 2010 09:38:48 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jan 2010 09:38:48 -0000 Received: (qmail 73245 invoked by uid 500); 5 Jan 2010 09:38:48 -0000 Delivered-To: lisa23@example.net Received: (qmail 73209 invoked by uid 500); 5 Jan 2010 09:38:48 -0000 Mailing-List: contact lisa23@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:lisa23@example.net> List-Unsubscribe: <mailto:lisa23@example.net> List-Post: <mailto:lisa23@example.net> List-Id: <kato-spec.incubator.apache.org> Reply-To: lisa23@example.net Delivered-To: mailing list lisa23@example.net Received: (qmail 73199 invoked by uid 99); 5 Jan 2010 09:38:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 09:38:48 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of lisa23@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO ey-out-1920.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 09:38:40 +0000 Received: by ey-out-1920.google.com with SMTP id 3so3661584eyh.8 for <lisa23@example.net>; Tue, 05 Jan 2010 01:38:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=gLcj8HNmDm/TNleDFscmtR/0P8qfmV5y+/5HCV7wSB0=; b=CFuW3KASSfr5XiCKMoxEzpff3ZD53V4ZVWR+fJk31yt2ApmWjY3isfXQBmDluVVCmt MxegGxZiTWwVTJDSRVCCEUdUXzHjKRr/8WjIVvEscP+JfT3wZ5YisUmajP9+JVBl/3jv BZCFj7FLkwz60FKUX80BhE12YueM5NULmbWvc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=pPWlPok1ilBXIsr8KTW/hmiZXwWcNlkEeMNkVKe80z1Vu+X+NtOYHlccfIzvNy1ENI UYn4l2vEpFAnZQ2ERiIYPSw3A+ZajmoYSjr7lYjNduPuisDWkm7CdLe5tEmtxqgrTV+/ Sr9xqVhLKo5LiXt/Zbt2A0LRsTNNdSMd5hrBg= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id o17mr9015848ebe.38.1262684299784; Tue, 05 Jan 2010 01:38:19 -0800 (PST) In-Reply-To: <lisa23@example.net> References: <lisa23@example.net> <lisa23@example.net> <lisa23@example.net> Date: Tue, 5 Jan 2010 09:38:19 +0000 Message-ID: <lisa23@example.net> Subject: Re: Snapshot Application development usecases From: David Griffiths <lisa23@example.net> To: lisa23@example.net Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org Hi Steve, thanks for the clarification. A lot of the issues we assist customers with turn out to be problems in third-party code and for these problems a post-mortem dump of some sort is the starting point. I'm surprised there's so little interest in obtaining Java info from core dumps. A lot of our customers use Java thread dumps and heapdumps to solve their own problems and I'm sure they would do more with core dumps if the tooling hadn't in the past been so unwieldy (eg jextract). Anyway, my perspective on all this is sufficiently different that I'll try to refrain from blundering in with any more dumb comments :) Cheers, Dave From kato-spec-returnlisa23@example.net Tue Jan 05 10:33:48 2010 Return-Path: <kato-spec-returnlisa23@example.net> Delivered-To: lisa23@example.net Received: (qmail 47046 invoked from network); 5 Jan 2010 10:33:48 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jan 2010 10:33:48 -0000 Received: (qmail 31860 invoked by uid 500); 5 Jan 2010 10:33:48 -0000 Delivered-To: lisa23@example.net Received: (qmail 31819 invoked by uid 500); 5 Jan 2010 10:33:48 -0000 Mailing-List: contact lisa23@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:lisa23@example.net> List-Unsubscribe: <mailto:lisa23@example.net> List-Post: <mailto:lisa23@example.net> List-Id: <kato-spec.incubator.apache.org> Reply-To: lisa23@example.net Delivered-To: mailing list lisa23@example.net Received: (qmail 31809 invoked by uid 99); 5 Jan 2010 10:33:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 10:33:48 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lisa23@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-bw0-f210.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 10:33:38 +0000 Received: by bwz2 with SMTP id 2so9155968bwz.20 for <lisa23@example.net>; Tue, 05 Jan 2010 02:33:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=pzHQ5MaQBTNi+b1RuR6jd0Ouyi2taDfz++3cLP0kuM4=; b=KFvl4sT7GYS3RZENCcJM1B3FPQ5UQOIxQvU1d8/a6LM2I/KFvREJBxbSuzA35J27Gt Dyg+CAlY0VeZ28XW8HstwQS+siStksRsEyupcYZAuRKItq8CQxW/fPbTb1Fem+lEZcHP XldTxouuvzLDnTSF+hRBTSqfj3f883Y4Nx2o0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=Bn4a2W2iO1BPb6vGFlk8G1r4qHf73krBm7n+PxTz1zkX63iEPu0Xa7i5r4IFm1CQgb bDiZcz+zX8TeRmrZKb42MkoneCy6zRVwyMSN/xxGAngmuDeOIsPR9Qfv0g8wnxlwFMCQ EUIngv5cL87I/QCty2v/Anq7KKABODtMjNSpA= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id n17mr5621512bka.102.1262687597260; Tue, 05 Jan 2010 02:33:17 -0800 (PST) In-Reply-To: <lisa23@example.net> References: <lisa23@example.net> <lisa23@example.net> <lisa23@example.net> <lisa23@example.net> Date: Tue, 5 Jan 2010 10:33:17 +0000 Message-ID: <lisa23@example.net> Subject: Re: Snapshot Application development usecases From: Steve Poole <lisa23@example.net> To: lisa23@example.net Content-Type: multipart/alternative; boundary=00504502438eb270cd047c685d43 --00504502438eb270cd047c685d43 Content-Type: text/plain; charset=ISO-8859-1 On Tue, Jan 5, 2010 at 9:38 AM, David Griffiths <lisa23@example.net>wrote: > Hi Steve, thanks for the clarification. A lot of the issues we assist > customers with turn out to be problems in third-party code and for > these problems a post-mortem dump of some sort is the starting point. > I'm surprised there's so little interest in obtaining Java info from > core dumps. A lot of our customers use Java thread dumps and heapdumps > to solve their own problems and I'm sure they would do more with core > dumps if the tooling hadn't in the past been so unwieldy (eg > jextract). > > In all the discussions we've had with developers the first hurdle has always been getting them to understand that there is value to* them* in using dumps. The Java developer community has grown up without the dump concept (If you do get a developer to acknowledge the existence of dumps then they see them as either an opaque thing to be given to some other support team or as a data collection mechanism for profiling) Once you do get the dump idea across you then have to explain that actually, the most prevalent dump (ie HPROF) doesn't have all the data needed to help solve the problems developers want to see solved. Which is where this project comes in - we need new dump artifacts that can help fill in the missing gaps and which work with the Sun JVM. We had hoped to get a corefile reader that could inspect the Sun JVM stuctures etc but that has not happened ( and as I explained it not something we can do here) Anyway, my perspective on all this is sufficiently different that I'll > try to refrain from blundering in with any more dumb comments :) > > I don't think you made any dumb comments - but perhaps your perspective was initially too IBM centric. We do welcome any comments , suggestions , contributions etc on any part of this project. I don't want to scare you off so if something is unclear just ask away and thanks for taking time to comment in the first place. > Cheers, > > Dave > -- Steve --00504502438eb270cd047c685d43-- From kato-spec-returnlisa23@example.net Tue Jan 05 13:51:49 2010 Return-Path: <kato-spec-returnlisa23@example.net> Delivered-To: lisa23@example.net Received: (qmail 30966 invoked from network); 5 Jan 2010 13:51:49 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jan 2010 13:51:49 -0000 Received: (qmail 62023 invoked by uid 500); 5 Jan 2010 13:51:49 -0000 Delivered-To: lisa23@example.net Received: (qmail 61987 invoked by uid 500); 5 Jan 2010 13:51:48 -0000 Mailing-List: contact lisa23@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:lisa23@example.net> List-Unsubscribe: <mailto:lisa23@example.net> List-Post: <mailto:lisa23@example.net> List-Id: <kato-spec.incubator.apache.org> Reply-To: lisa23@example.net Delivered-To: mailing list lisa23@example.net Received: (qmail 61977 invoked by uid 99); 5 Jan 2010 13:51:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 13:51:48 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_HELO_PASS,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO moutng.kundenserver.de) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 13:51:39 +0000 Received: from [83.161.120.155] (gbibp9ph1--blueice3n1.emea.ibm.com [83.161.120.155]) by mrelayeu.kundenserver.de (node=mrbap2) with ESMTP (Nemesis) id 0MMpSB-1NZ3hS2JrD-008khX; Tue, 05 Jan 2010 14:51:17 +0100 Message-ID: <lisa23@example.net> Date: Tue, 05 Jan 2010 13:51:17 +0000 From: Stuart Monteith <lisa23@example.net> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:83.161.120.155) Gecko/20091206 SeaMonkey/2.0.1 MIME-Version: 1.0 To: lisa23@example.net Subject: Re: Snapshot Application development usecases References: <lisa23@example.net> <lisa23@example.net> <lisa23@example.net> In-Reply-To: <lisa23@example.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V01U2FsdGVkX1/l7GyTBcDR0+y8AY7+JM2hpHkQVf5bkU4hfUM bd3bFd89x6L2F+rDbSsZmLkt7M1lCMI9EV/ZcV6aw/BfT6g5wb a+CiHk3MoPA414yOTnhfWtfPgmlaSwt Given a method: dump() to generate a dump, this could be expanded with: dump(Object obj) which would dump that object and what it refers too. Of course, doing the following: Object[] obj; dump(obj) I would expect that a Dump object would be created and used in multiple places, configured for use by whatever. The object passed to the dump method would be used as the context in which the dump was to be taken. Commonly, we might expect: Object[] context = new Object[2]; context[0] = Thread.getCurrentThread(); context[1]= HTTPSession; dumpObj.dump(context); which the dumpObj has already been configured for. class Dump { // Generate a dump. void dump(); // Generate a dump, including the passed root object. void dump(Object root); // Get the roots already Set getRoots(); // Sets the number of references void setDepth(int depth); // Sets the objects to start scanning from void setRoots(Set col) ; // Get the set of classloaders dumps are filtered against. Set<ClassLoader> getClassLoadersFilter(); // Set the classloaders to filter against. void setClassLoadersFilter(Set<ClassLoader> loaders); // Set the packages to filter against Set<String> getPackagesFilter(); void setPackagesFilter(Set<String> packages); // Filter by sets of Threads. void setThreadsFilter(Set<Thread> thread); Set<Thread> getThreadsFilter(); ... etc.. } Of course, filtering on annotations, classloader and packages are essentially filtering by operations on classes. Presuming the filtering is performed in Java, we could simply provide an interface to determine this: interface ClassFilter { // Return true if class is to be included. boolean filter(Class clazz); } We could provide implementations of ClassFilters to perform the operations. I am a little concerned about how to implement this stuff. If it is Java, we need a virtual machine, and the one we are in may be failing in such a way we can't execute - an OutOfMemoryError or StackOverFlowError are both obvious examples. Regards, Stuart Steve Poole wrote: > Thanks Stuart - like the donkey. > > I would imagine, based on your examples, having methods like > > selectClassesLoadedBy(ClassLoader c) > selectPackageName("com.silly.donkey.*") > selectByAnnotation(Annotation a) > selectReferencedObjects(Thread t); > addFieldFilter(java.lang.reflect.Field f,Object value); > selectReferencedObjectsWithin(Object root,int distance); > > > I like to get down the the method definitions as quickly as possible as it > makes it real :-) > > > On Mon, Jan 4, 2010 at 5:44 PM, Stuart Monteith<lisa23@example.net> wrote: > > >> Happy New Year everyone! >> >> The usecases are interesting, the problem is how to find and dump what you >> want and no more. >> >> > >> Ways in which objects may be selected for dumping: >> >> o By classloader >> This is can separate object instances at runtime by virtue of their >> classes having been >> defined by the same classloader. >> >> o By class package >> Simply filter dumped classes matching a given filter name. >> >> o By class annotation >> Classes could be specially marked with an annotation that is visible >> at runtime. >> >> o By object instance. >> A dump would have to be initiated /somewhere/. Realistically, we'd >> have to start from a particular >> object, class. An array could be passed to cater for multiple roots. >> >> o By Thread. >> Using a thread and all of its roots would be practicable if they >> were accessible. >> >> o By field value: >> Restrict dump to objects with a field containing a certain value, >> lisa23@example.net. >> >> o By distance from the root object. >> For example, given an object, only retrieve objects three references >> away. >> >> These could be used in various combinations. Dumping all objects of classes >> in package com.silly.donkey >> is all very well, but this could be restricted to only those annotated with >> @DumpMe and further restricted >> to only those reachable from a particular instance of >> "com.silly.donkey.HeeHaw" . >> >> >> Regards, >> Stuart >> >> >> Steve Poole wrote: >> >> >>> I want to present a few application development usecases for us to discuss >>> to do with the Snapshot concept. >>> >>> First let's agree some basics >>> >>> An application snapshot is a deliberate subset of a running system. It's >>> deliberate in that what is defined for inclusion / exclusion can be >>> changed. >>> >>> To build an application snapshot three types of information are required >>> >>> 1 - root objects which determine where to start the snapshot from >>> 2 - rules to restrict how "far" to search >>> 3 - rules to describe what to include in the snapshot >>> >>> >>> Example 1 >>> >>> A webserver session is behaving incorrectly. The application programmer >>> wants to capture information >>> to do with the session itself and some specific application details. >>> >>> The root object is an HTTPSession and the application code is all >>> contained >>> in packages that start org.acme.application >>> The application programmer believes that the problem is in the application >>> and so wants the minimum >>> of system information included. >>> >>> >>> Example 2 >>> >>> An error has occured and the application is about throw an exception. The >>> application programmer wants to capture a series >>> of objects related to the error and wants to see the Java stack including >>> local variables. >>> >>> >>> Example 3 >>> >>> A enterprise server is running slow and the application programmer would >>> like to get a list of the instances of a specific interface >>> that the server is managing, since the suspicion is that there are >>> multiple >>> instances of a nonthreadsafe instance of this interface when there should >>> only be one. >>> >>> Example 4 >>> >>> A servlet server has run out of database connections and the suspicion is >>> that managed objects have not been >>> collected. Typically programmers use finalisers or depend on GC instead >>> of >>> using the Servlet.Destroy() method. >>> the application programmer needs to get a list of objects in "Destroyed" >>> state which obviously haven't been GC'd >>> >>> >>> Cheers >>> >>> >>> >>> >> -- >> Stuart Monteith >> http://blog.stoo.me.uk/ >> >> >> > > -- Stuart Monteith http://blog.stoo.me.uk/ From kato-spec-returnlisa23@example.net Tue Jan 05 15:34:37 2010 Return-Path: <kato-spec-returnlisa23@example.net> Delivered-To: lisa23@example.net Received: (qmail 76648 invoked from network); 5 Jan 2010 15:34:37 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jan 2010 15:34:37 -0000 Received: (qmail 92300 invoked by uid 500); 5 Jan 2010 15:34:37 -0000 Delivered-To: lisa23@example.net Received: (qmail 92264 invoked by uid 500); 5 Jan 2010 15:34:37 -0000 Mailing-List: contact lisa23@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:lisa23@example.net> List-Unsubscribe: <mailto:lisa23@example.net> List-Post: <mailto:lisa23@example.net> List-Id: <kato-spec.incubator.apache.org> Reply-To: lisa23@example.net Delivered-To: mailing list lisa23@example.net Received: (qmail 92254 invoked by uid 99); 5 Jan 2010 15:34:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 15:34:37 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of lisa23@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-pw0-f41.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 15:34:27 +0000 Received: by pwj4 with SMTP id 4so2596438pwj.20 for <lisa23@example.net>; Tue, 05 Jan 2010 07:34:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=8ooSlKza71KXuSt0JhJA56r011FA9gHlZW3lh1GXLIs=;
{'source': 'Apache Software Foundation Public Mail Archives'}