From server-devugarcia@example.com Thu Apr 01 07:17:06 2010 Return-Path: <server-devugarcia@example.com> Delivered-To: ugarcia@example.com Received: (qmail 60695 invoked from network); 1 Apr 2010 07:17:06 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Apr 2010 07:17:06 -0000 Received: (qmail 95859 invoked by uid 500); 1 Apr 2010 07:17:06 -0000 Delivered-To: ugarcia@example.com Received: (qmail 95824 invoked by uid 500); 1 Apr 2010 07:17:03 -0000 Mailing-List: contact ugarcia@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:ugarcia@example.com> List-Help: <mailto:ugarcia@example.com> List-Post: <mailto:ugarcia@example.com> List-Id: "James Developers List" <server-dev.james.apache.org> Reply-To: "James Developers List" <ugarcia@example.com> Delivered-To: mailing list ugarcia@example.com Received: (qmail 95816 invoked by uid 99); 1 Apr 2010 07:17:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 07:17:03 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mailrelay003.isp.belgacom.be) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 07:16:57 +0000 X-Belgacom-Dynamic: yes X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApEBANPks0tbsJiO/2dsb2JhbAAH0CeCbwqCCAQ Received: from 142.152-176-91.adsl-dyn.isp.belgacom.be (HELO [83.161.120.155]) ([83.161.120.155]) by relay.skynet.be with ESMTP; 01 Apr 2010 09:16:36 +0200 Message-ID: <ugarcia@example.com> Date: Thu, 01 Apr 2010 09:16:36 +0200 From: Eric Charles <ugarcia@example.com> User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:83.161.120.155) Gecko/20100322 Thunderbird/3.0.3 MIME-Version: 1.0 To: James Developers List <ugarcia@example.com> Subject: Re: OutOfMemory exception References: <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> In-Reply-To: <ugarcia@example.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi Norman, Tks for the smpt-source pointer. I looked at it, but finally, I think I will develop those little classes to have more control on it. I will put it on github. I launched latest trunk james yesterday evening. During the night, 3 hours after the launch, the following occured: KahDB slower and slower... INFO 23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 2330 INFO 23:38:04,591 | james.mailetcontext | Error while storing mail. org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be parsed.; nested exception is: org.apache.james.imap.mailbox.StorageException: failed. Transaction commit failed.; nested exception is: <openjpa-1.2.1-r752877:753278 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265) at org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130) ... 55 more Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt 1363215207 INSERT INTO Message (id, bodyStartOctet, content, contentOctets, mediaType, subType, textu alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int) 1260, (byte[]) [B@7a00e70c, (long) 47887, (String) text, (String) html, (long) 876]} [code=0, state=XJ001] at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192) at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57) at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866) at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269) So still a OOM exception that was shown by yet-another-component (in this case, the StoreMailbox). There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its job). All 4 files were 0 bytes. I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB max memory). With the previous parameters ( -Xmx512m), the process was taking the whole 512MB. Tks, Eric On 03/31/2010 08:26 PM, Norman Maurer wrote: > Hi eric, > > thx for all your help, maybe smtp-source ( which is included in > postfix ) can help you with stress testing. > > Bye > Norman > > 2010/3/31, Eric Charles<ugarcia@example.com>: > >> Hi Norman, >> >> There are not so much .m64 files, so even it it shows that a mail >> processing didn't reach the end and may have cause a leak, I don't think >> it would make crash my process so quick (between 10 and 24 hours). >> >> I quickly analysed a few dumps last week : there is always a class that >> took 30/40% of the memory. Sometimes a activemq class, sometimes not >> (don't remember the details). >> Also notable, the stack trace (the place where the OOM gives problems) >> varies. >> Today stack was: >> org.apache.camel.RuntimeCamelException: >> org.apache.camel.CamelExchangeException: Error processing Exchange. >> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102, >> responseRequired = true, messageId = >> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null, >> originalTransactionId = null, producerId = >> ID:srv001-51032-1270011735798-2:0:24:1, destination = >> queue://processor.root, transactionId = null, expiration = 0, timestamp >> = 1270014325685, arrival = 0, brokerInTime = 1270014326893, >> brokerOutTime = 1270014333023, correlationId = null, replyTo = null, >> persistent = true, type = null, priority = 4, groupID = null, >> groupSequence = 0, targetConsumerId = null, compressed = false, userID = >> null, content = org.apache.activemq.util.ByteSequence@69b568d0, >> marshalledProperties = null, dataStructure = null, redeliveryCounter = >> 0, size = 10718, properties = null, readOnlyProperties = true, >> readOnlyBody = true, droppable = false}]. Caused by: >> [java.lang.OutOfMemoryError - Java heap space] >> at >> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055) >> at >> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154) >> ... >> >> >> >> I also think the tmp files are not the main cause. >> >> I have now james running in eclipse and I made already made a little >> trip in the code. Quite impressive since last time I ugarcia@example.com. >> I will try to stress james with a small smtp/pop3 client (I used >> postage, but a simple class with commons-net behind) may be easier and >> see what happens (eventually with eclipse profiler). >> >> It will be for this weekend for me. >> >> Bye, >> >> Eric >> >> >> On 03/31/2010 07:11 PM, Norman Maurer wrote: >> >>> Hi Eric, >>> >>> thx for keeping us in the loop... I'm still not sure why the .m64 >>> files are still in the tmp folder sometimes.. But I suspect the files >>> are not the cause of the OOM. >>> >>> I didn't get a OOM since yesterday morning (the time I deployed >>> current trunk version), but just found to new .m64 files.. >>> >>> So I'm still searching for the real cause. If nothing helps I will >>> need to use a profiler to find th leak. >>> >>> Bye, >>> Norman >>> >>> 2010/3/31 Eric Charles<ugarcia@example.com>: >>> >>> >>>> Hi Norman, >>>> >>>> I had defined the Null mailet >>>> >>>> <mailet match="HostIsLocal" class="Null"> >>>> <processor> local-address-error</processor> >>>> <notice>550 - Requested action not taken: no such user here</notice> >>>> </mailet> >>>> >>>> but now, I use the standard config >>>> >>>> <mailet match="HostIsLocal" class="ToProcessor"> >>>> <processor> local-address-error</processor> >>>> <notice>550 - Requested action not taken: no such user here</notice> >>>> </mailet> >>>> >>>> I still have the OOM. >>>> >>>> I will now deploy a fresh svn with your last commits and enable the >>>> ValidRcptHandler. >>>> I keep you posted with the result, >>>> >>>> Tks, >>>> Eric >>>> >>>> >>>> On 03/30/2010 06:49 AM, Norman Maurer wrote: >>>> >>>> >>>>> Hi Eric, >>>>> >>>>> you said all the files are related to address-errors , could you show >>>>> me your address error processor config? >>>>> Does the Problem still exist when you enable the ValidRcptHandler in >>>>> the smtpserver.xml file? >>>>> >>>>> Thx >>>>> Norman >>>>> >>>>> 2010/3/30, Eric Charles<ugarcia@example.com>: >>>>> >>>>> >>>>> >>>>>> Oops, no, the files are still there (only ugarcia@example.com). >>>>>> Eric >>>>>> >>>>>> >>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote: >>>>>> >>>>>> >>>>>> >>>>>>> Hi Norman, >>>>>>> >>>>>>> I just deployed your new commit with the new camel DisposeProcess. >>>>>>> Good news : I don't see anymore the m64 file in tmp (well I see 1 >>>>>>> file, on the second after, it is no more there, so the dispose works >>>>>>> as it should). >>>>>>> >>>>>>> I keep you posted with our eventual future OOM. >>>>>>> >>>>>>> Tks, >>>>>>> Eric >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Hi Norman, >>>>>>>> >>>>>>>> The .m64 are all to "unkown user" but to "well known domain" (so for >>>>>>>> <ugarcia@example.com>). >>>>>>>> They are from various size (with and without attachment). >>>>>>>> They are well formed (I can open the downloaded file with >>>>>>>> thunderbird) >>>>>>>> >>>>>>>> However, when I sent a mail to ugarcia@example.com, I don't see it in >>>>>>>> the /tmp >>>>>>>> >>>>>>>> Tks, >>>>>>>> Eric >>>>>>>> >>>>>>>> >>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Hi Eric, >>>>>>>>> >>>>>>>>> sure.. we have to find the OOM cause. What would be interesting, >>>>>>>>> could >>>>>>>>> you check somehow if the .m64 files are files which are related to >>>>>>>>> successfully delivered mail ? >>>>>>>>> >>>>>>>>> Thx, >>>>>>>>> Norman >>>>>>>>> >>>>>>>>> >>>>>>>>> 2010/3/29 Eric Charles<ugarcia@example.com>: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Norman, >>>>>>>>>> >>>>>>>>>> Done. Now, we have to wait... >>>>>>>>>> >>>>>>>>>> I saw some created *.m64 that were removed. >>>>>>>>>> But there are other ones that remains in /tmp. >>>>>>>>>> >>>>>>>>>> I sometimes run a jmap (java memory map) to produce a heap dump and >>>>>>>>>> analyse >>>>>>>>>> it. >>>>>>>>>> At the beginning, everything seems ok, and often, when I come back, >>>>>>>>>> the OOM >>>>>>>>>> has already occured. >>>>>>>>>> >>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a quicker exception (if >>>>>>>>>> any, >>>>>>>>>> let's hope not). >>>>>>>>>> >>>>>>>>>> The *.m64 are a clue and we should ensure that they are removed in >>>>>>>>>> all >>>>>>>>>> circumstances. >>>>>>>>>> This may be the easy part, as we have some visible clues (the >>>>>>>>>> files). >>>>>>>>>> We should also ensure after that there are no other causes to the >>>>>>>>>> leaks. >>>>>>>>>> >>>>>>>>>> Tks, >>>>>>>>>> Eric >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Hi Eric, >>>>>>>>>>> >>>>>>>>>>> I found the cause for the not deleted temporary files. Hopefully >>>>>>>>>>> this >>>>>>>>>>> is the cause of the OOM. Could try to svn up and run again ? >>>>>>>>>>> >>>>>>>>>>> Thx, >>>>>>>>>>> Norman >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2010/3/29 Norman Maurer<ugarcia@example.com>: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Hi Eric, >>>>>>>>>>>> >>>>>>>>>>>> thx for the report. I see exact the same problem today here.. >>>>>>>>>>>> (The >>>>>>>>>>>> OOM). Didn't notice the files in the tmp folder, but I think >>>>>>>>>>>> thats >>>>>>>>>>>> a >>>>>>>>>>>> good pointer. I will try to debug the problem later or tomorrow. >>>>>>>>>>>> But I >>>>>>>>>>>> suspect you are right about the tmp files and jms producers.. Did >>>>>>>>>>>> you >>>>>>>>>>>> run a kill -3 pid to see what threads are active etc ? >>>>>>>>>>>> >>>>>>>>>>>> Thx, >>>>>>>>>>>> Norman >>>>>>>>>>>> >>>>>>>>>>>> Ps: Patches are welcome :) >>>>>>>>>>>> >>>>>>>>>>>> 2010/3/29 Eric Charles<ugarcia@example.com>: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> The last two weeks, I deployed various trunk snapshots. >>>>>>>>>>>>> After james running less than 1 day, I always ran into a >>>>>>>>>>>>> OutOfMemory >>>>>>>>>>>>> Exception. >>>>>>>>>>>>> >>>>>>>>>>>>> I analysed the logs (for example STACK TRACE 1 in annex), and >>>>>>>>>>>>> always >>>>>>>>>>>>> found >>>>>>>>>>>>> Camel complaining when trying to deliver on JMS queue. >>>>>>>>>>>>> I tried to adapt the ActiveMQ configuration based on >>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html but nothing >>>>>>>>>>>>> helped. >>>>>>>>>>>>> >>>>>>>>>>>>> I also analyzed various heap dump :sometimes, 65% was used by >>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession, sometimes >>>>>>>>>>>>> by a >>>>>>>>>>>>> activemq class,... >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: ugarcia@example.com >>>>>>> For additional commands, e-mail: ugarcia@example.com >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: ugarcia@example.com >>>>>> For additional commands, e-mail: ugarcia@example.com >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: ugarcia@example.com >>>>> For additional commands, e-mail: ugarcia@example.com >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: ugarcia@example.com >>>> For additional commands, e-mail: ugarcia@example.com >>>> >>>> >>>> >>>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: ugarcia@example.com >>> For additional commands, e-mail: ugarcia@example.com >>> >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: ugarcia@example.com >> For additional commands, e-mail: ugarcia@example.com >> >> >> > --------------------------------------------------------------------- > To unsubscribe, e-mail: ugarcia@example.com > For additional commands, e-mail: ugarcia@example.com > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: ugarcia@example.com For additional commands, e-mail: ugarcia@example.com From server-devugarcia@example.com Thu Apr 01 07:25:06 2010 Return-Path: <server-devugarcia@example.com> Delivered-To: ugarcia@example.com Received: (qmail 62310 invoked from network); 1 Apr 2010 07:25:06 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Apr 2010 07:25:06 -0000 Received: (qmail 452 invoked by uid 500); 1 Apr 2010 07:25:06 -0000 Delivered-To: ugarcia@example.com Received: (qmail 347 invoked by uid 500); 1 Apr 2010 07:25:04 -0000 Mailing-List: contact ugarcia@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:ugarcia@example.com> List-Help: <mailto:ugarcia@example.com> List-Post: <mailto:ugarcia@example.com> List-Id: "James Developers List" <server-dev.james.apache.org> Reply-To: "James Developers List" <ugarcia@example.com> Delivered-To: mailing list ugarcia@example.com Received: (qmail 338 invoked by uid 99); 1 Apr 2010 07:25:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 07:25:03 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ugarcia@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-ww0-f49.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 07:24:57 +0000 Received: by wwa36 with SMTP id 36so602990wwa.22 for <ugarcia@example.com>; Thu, 01 Apr 2010 00:24:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:content-type :content-transfer-encoding; bh=onyHI8c9EV7dbAv1pgY7H+acx7C2XA670+gL5TYsN0c=; b=WqJT9+DskYKjIErs3XQo0Nfj2a+O4Ymib5C61fvHdnX+xyXRAHFiHqrrBu4S7xNC2R lQVmA2bLQfmDheYc/fomWf5Eq7ajMXqNhzKEmTLCf1IISgcDJDkW1qgTjHwxWQfayQfy cug8rR5UmM1NikQC2JD+oVViZrs1agqR2T8yk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=LGVxLH7G2z6zd5Xddf7couBPsSIHSuidVrMwCQpVPQ2AKuuAEo0waPWniV4vKPGzjE qlnsjOJPg7Deiq5mwQWB2G6y2ilepwTD34ZW9T1snt9+VWJJXgAOvwv7ljkPO8Xs/swc AGwiRDi5cssF6I+Ut7h7+ExBpVuq8nljlc5Ys= MIME-Version: 1.0 Received: by 83.161.120.155 with HTTP; Thu, 1 Apr 2010 00:24:36 -0700 (PDT) In-Reply-To: <ugarcia@example.com> References: <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> Date: Thu, 1 Apr 2010 09:24:36 +0200 Received: by 83.161.120.155 with SMTP id x53mr215336wee.144.1270106676899; Thu, 01 Apr 2010 00:24:36 -0700 (PDT) Message-ID: <ugarcia@example.com> Subject: Re: OutOfMemory exception From: Norman Maurer <ugarcia@example.com> To: James Developers List <ugarcia@example.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Hi Eric, thx again for your tests... So I think we have two independent problems here: 1) The .m64 files which are not deleted always: My guess here is that there is something wrong with processing Mail objects with state Null. I will investigate futher.. I also found a possible problem with .m64 files which get not deleted when a Exception is throws while processing the camel routes. I will patch this soon. 2) The OOM My guess here is that ActiveMQ is not freeing up memory. Not sure why yet I think 2) is the more criticial problem and should get higher priority to get fixed.. Bye, Norman 2010/4/1 Eric Charles <ugarcia@example.com>: > Hi Norman, > > Tks for the smpt-source pointer. > I looked at it, but finally, I think I will develop those little classes = to > have more control on it. > I will put it on github. > > I launched latest trunk james yesterday evening. > During the night, 3 hours after the launch, the following occured: > > KahDB slower and slower... > > INFO =A023:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase |= Slow > KahaDB access: cleanup took 2330 > INFO =A023:38:04,591 | james.mailetcontext | Error while storing mail. > org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be > parsed.; > =A0nested exception is: > =A0 =A0org.apache.james.imap.mailbox.StorageException: failed. Transactio= n > commit failed.; > =A0nested exception is: > <openjpa-1.2.1-r752877:753278 fatal store error> > org.apache.openjpa.persistence.RollbackException: The transaction has bee= n > rolled back. =A0See the nested exceptions for details on the errors that > occurred. > =A0 =A0at > org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:= 265) > > =A0 =A0at > org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreMan= ager.java:130) > =A0 =A0... 55 more > Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java > exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt > 1363215207 INSERT INTO Message (id, bodyStartOctet, content, contentOctet= s, > mediaType, subType, textu > alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=3D(long) 2678, (int) 12= 60, > (byte[]) [B@7a00e70c, (long) 47887, (String) text, (String) html, (long) > 876]} [code=3D0, state=3DXJ001] > =A0 =A0at > org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnec= tionDecorator.java:192) > =A0 =A0at > org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(Logging= ConnectionDecorator.java:57) > =A0 =A0at > org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$= LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866) > =A0 =A0at > org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(Del= egatingPreparedStatement.java:269) > > So still a OOM exception that was shown by yet-another-component (in this > case, the StoreMailbox). > > > There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its > job). > All 4 files were 0 bytes. > > I have now launched with EXTRA_JVM_ARGUMENTS=3D"-Xms512m -Xmx4g" (so 4GB = max > memory). > > With the previous parameters ( -Xmx512m), the process was taking the whol= e > 512MB. > > Tks, > Eric > > > On 03/31/2010 08:26 PM, Norman Maurer wrote: >> >> Hi eric, >> >> thx for all your help, maybe smtp-source ( which is included in >> postfix ) can help you with stress testing. >> >> Bye >> Norman >> >> 2010/3/31, Eric Charles<ugarcia@example.com>: >> >>> >>> Hi Norman, >>> >>> There are not so much .m64 files, so even it it shows that a mail >>> processing didn't reach the end and may have cause a leak, I don't thin= k >>> it would make crash my process so quick (between 10 and 24 hours). >>> >>> I quickly analysed a few dumps last week : there is always a class that >>> took 30/40% of the memory. Sometimes a activemq class, sometimes not >>> (don't remember the details). >>> Also notable, the stack trace (the place where the OOM gives problems) >>> varies. >>> Today stack was: >>> org.apache.camel.RuntimeCamelException: >>> org.apache.camel.CamelExchangeException: Error processing Exchange. >>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId =3D 2102, >>> responseRequired =3D true, messageId =3D >>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination =3D nul= l, >>> originalTransactionId =3D null, producerId =3D >>> ID:srv001-51032-1270011735798-2:0:24:1, destination =3D >>> queue://processor.root, transactionId =3D null, expiration =3D 0, times= tamp >>> =3D 1270014325685, arrival =3D 0, brokerInTime =3D 1270014326893, >>> brokerOutTime =3D 1270014333023, correlationId =3D null, replyTo =3D nu= ll, >>> persistent =3D true, type =3D null, priority =3D 4, groupID =3D null, >>> groupSequence =3D 0, targetConsumerId =3D null, compressed =3D false, u= serID =3D >>> null, content =3D org.apache.activemq.util.ByteSequence@69b568d0, >>> marshalledProperties =3D null, dataStructure =3D null, redeliveryCounte= r =3D >>> 0, size =3D 10718, properties =3D null, readOnlyProperties =3D true, >>> readOnlyBody =3D true, droppable =3D false}]. Caused by: >>> [java.lang.OutOfMemoryError - Java heap space] >>> =A0 =A0 =A0at >>> >>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelp= er.java:1055) >>> =A0 =A0 =A0at >>> >>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWi= thoutResult(TransactionErrorHandler.java:154) >>> ... >>> >>> >>> >>> I also think the tmp files are not the main cause. >>> >>> I have now james running in eclipse and I made already made a little >>> trip in the code. Quite impressive since last time I ugarcia@example.com. >>> I will try to stress james with a small smtp/pop3 client (I used >>> postage, but a simple class with commons-net behind) may be easier and >>> see what happens (eventually with eclipse profiler). >>> >>> It will be for this weekend for me. >>> >>> Bye, >>> >>> Eric >>> >>> >>> On 03/31/2010 07:11 PM, Norman Maurer wrote: >>> >>>> >>>> Hi Eric, >>>> >>>> thx for keeping us in the loop... I'm still not sure why the .m64 >>>> files are still in the tmp folder sometimes.. But I suspect the files >>>> are not the cause of the OOM. >>>> >>>> I didn't get a OOM since yesterday morning (the time I deployed >>>> current trunk version), but just found to new .m64 files.. >>>> >>>> So I'm still searching for the real cause. If nothing helps I will >>>> need to use a profiler to find th leak. >>>> >>>> Bye, >>>> Norman >>>> >>>> 2010/3/31 Eric Charles<ugarcia@example.com>: >>>> >>>> >>>>> >>>>> Hi Norman, >>>>> >>>>> I had defined the Null mailet >>>>> >>>>> <mailet match=3D"HostIsLocal" class=3D"Null"> >>>>> <processor> =A0 local-address-error</processor> >>>>> <notice>550 - Requested action not taken: no such user here</notice> >>>>> </mailet> >>>>> >>>>> but now, I use the standard config >>>>> >>>>> <mailet match=3D"HostIsLocal" class=3D"ToProcessor"> >>>>> <processor> =A0 local-address-error</processor> >>>>> <notice>550 - Requested action not taken: no such user here</notice> >>>>> </mailet> >>>>> >>>>> I still have the OOM. >>>>> >>>>> I will now deploy a fresh svn with your last commits and enable the >>>>> ValidRcptHandler. >>>>> I keep you posted with the result, >>>>> >>>>> Tks, >>>>> Eric >>>>> >>>>> >>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote: >>>>> >>>>> >>>>>> >>>>>> Hi Eric, >>>>>> >>>>>> you said all the files are related to address-errors , could you sho= w >>>>>> me your address error processor config? >>>>>> Does the Problem still exist when you enable the ValidRcptHandler in >>>>>> the smtpserver.xml file? >>>>>> >>>>>> Thx >>>>>> Norman >>>>>> >>>>>> 2010/3/30, Eric Charles<ugarcia@example.com>: >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Oops, no, the files are still there (only ugarcia@example.com). >>>>>>> Eric >>>>>>> >>>>>>> >>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Hi Norman, >>>>>>>> >>>>>>>> I just deployed your new commit with the new camel DisposeProcess. >>>>>>>> Good news : I don't see anymore the m64 file in tmp (well I see 1 >>>>>>>> file, on the second after, it is no more there, so the dispose wor= ks >>>>>>>> as it should). >>>>>>>> >>>>>>>> I keep you posted with our eventual future OOM. >>>>>>>> >>>>>>>> Tks, >>>>>>>> Eric >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Hi Norman, >>>>>>>>> >>>>>>>>> The .m64 are all to "unkown user" but to "well known domain" (so >>>>>>>>> for >>>>>>>>> <ugarcia@example.com>). >>>>>>>>> They are from various size (with and without attachment). >>>>>>>>> They are well formed (I can open the downloaded file with >>>>>>>>> thunderbird) >>>>>>>>> >>>>>>>>> However, when I sent a mail to ugarcia@example.com, I don't see it = in >>>>>>>>> the /tmp >>>>>>>>> >>>>>>>>> Tks, >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> >>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi Eric, >>>>>>>>>> >>>>>>>>>> sure.. we have to find the OOM cause. What would be interesting, >>>>>>>>>> could >>>>>>>>>> you check somehow if the .m64 files are files which are related = to >>>>>>>>>> successfully delivered mail ? >>>>>>>>>> >>>>>>>>>> Thx, >>>>>>>>>> Norman >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2010/3/29 Eric Charles<ugarcia@example.com>: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Norman, >>>>>>>>>>> >>>>>>>>>>> Done. Now, we have to wait... >>>>>>>>>>> >>>>>>>>>>> I saw some created *.m64 that were removed. >>>>>>>>>>> But there are other ones that remains in /tmp. >>>>>>>>>>> >>>>>>>>>>> I sometimes run a jmap (java memory map) to produce a heap dump >>>>>>>>>>> and >>>>>>>>>>> analyse >>>>>>>>>>> it. >>>>>>>>>>> At the beginning, everything seems ok, and often, when I come >>>>>>>>>>> back, >>>>>>>>>>> the OOM >>>>>>>>>>> has already occured. >>>>>>>>>>> >>>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a quicker exception >>>>>>>>>>> (if >>>>>>>>>>> any, >>>>>>>>>>> let's hope not). >>>>>>>>>>> >>>>>>>>>>> The *.m64 are a clue and we should ensure that they are removed >>>>>>>>>>> in >>>>>>>>>>> all >>>>>>>>>>> circumstances. >>>>>>>>>>> This may be the easy part, as we have some visible clues (the >>>>>>>>>>> files). >>>>>>>>>>> We should also ensure after that there are no other causes to t= he >>>>>>>>>>> leaks. >>>>>>>>>>> >>>>>>>>>>> Tks, >>>>>>>>>>> Eric >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Eric, >>>>>>>>>>>> >>>>>>>>>>>> I found the cause for the not deleted temporary files. Hopeful= ly >>>>>>>>>>>> this >>>>>>>>>>>> is the cause of the OOM. Could try to svn up and run again ? >>>>>>>>>>>> >>>>>>>>>>>> Thx, >>>>>>>>>>>> Norman >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2010/3/29 Norman Maurer<ugarcia@example.com>: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Eric, >>>>>>>>>>>>> >>>>>>>>>>>>> thx for the report. I see exact the same problem today here.. >>>>>>>>>>>>> (The >>>>>>>>>>>>> OOM). Didn't notice the files in the tmp folder, but I think >>>>>>>>>>>>> thats >>>>>>>>>>>>> a >>>>>>>>>>>>> good pointer. I will try to debug the problem later or >>>>>>>>>>>>> tomorrow. >>>>>>>>>>>>> But I >>>>>>>>>>>>> suspect you are right about the tmp files and jms producers.. >>>>>>>>>>>>> Did >>>>>>>>>>>>> you >>>>>>>>>>>>> run a kill -3 pid to see what threads are active etc ? >>>>>>>>>>>>> >>>>>>>>>>>>> Thx, >>>>>>>>>>>>> Norman >>>>>>>>>>>>> >>>>>>>>>>>>> Ps: Patches are welcome :) >>>>>>>>>>>>> >>>>>>>>>>>>> 2010/3/29 Eric Charles<ugarcia@example.com>: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> The last two weeks, I deployed various trunk snapshots. >>>>>>>>>>>>>> After james running less than 1 day, I always ran into a >>>>>>>>>>>>>> OutOfMemory >>>>>>>>>>>>>> Exception. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I analysed the logs (for example STACK TRACE 1 in annex), an= d >>>>>>>>>>>>>> always >>>>>>>>>>>>>> found >>>>>>>>>>>>>> Camel complaining when trying to deliver on JMS queue. >>>>>>>>>>>>>> I tried to adapt the ActiveMQ configuration based on >>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html but >>>>>>>>>>>>>> nothing >>>>>>>>>>>>>> helped. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I also analyzed various heap dump :sometimes, 65% was used b= y >>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession, >>>>>>>>>>>>>> sometimes >>>>>>>>>>>>>> by a >>>>>>>>>>>>>> activemq class,... >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ------------------------------------------------------------------= --- >>>>>>>> To unsubscribe, e-mail: ugarcia@example.com >>>>>>>> For additional commands, e-mail: ugarcia@example.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -------------------------------------------------------------------= -- >>>>>>> To unsubscribe, e-mail: ugarcia@example.com >>>>>>> For additional commands, e-mail: ugarcia@example.com >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------= - >>>>>> To unsubscribe, e-mail: ugarcia@example.com >>>>>> For additional commands, e-mail: ugarcia@example.com >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: ugarcia@example.com >>>>> For additional commands, e-mail: ugarcia@example.com >>>>> >>>>> >>>>> >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: ugarcia@example.com >>>> For additional commands, e-mail: ugarcia@example.com >>>> >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: ugarcia@example.com >>> For additional commands, e-mail: ugarcia@example.com >>> >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: ugarcia@example.com >> For additional commands, e-mail: ugarcia@example.com >> >> >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: ugarcia@example.com > For additional commands, e-mail: ugarcia@example.com > > --------------------------------------------------------------------- To unsubscribe, e-mail: ugarcia@example.com For additional commands, e-mail: ugarcia@example.com From server-devugarcia@example.com Thu Apr 01 07:35:14 2010 Return-Path: <server-devugarcia@example.com> Delivered-To: ugarcia@example.com Received: (qmail 64288 invoked from network); 1 Apr 2010 07:35:14 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Apr 2010 07:35:14 -0000 Received: (qmail 9155 invoked by uid 500); 1 Apr 2010 07:35:14 -0000 Delivered-To: ugarcia@example.com Received: (qmail 8960 invoked by uid 500); 1 Apr 2010 07:35:12 -0000 Mailing-List: contact ugarcia@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:ugarcia@example.com> List-Help: <mailto:ugarcia@example.com> List-Post: <mailto:ugarcia@example.com> List-Id: "James Developers List" <server-dev.james.apache.org> Reply-To: "James Developers List" <ugarcia@example.com> Delivered-To: mailing list ugarcia@example.com Received: (qmail 8948 invoked by uid 99); 1 Apr 2010 07:35:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 07:35:11 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=10.0 tests=AWL,MSGID_FROM_MTA_HEADER,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mail.devtech.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 07:35:04 +0000 Message-ID: <ugarcia@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-MessageIsInfected: false Received: from cpe-024-168-243-113.sc.res.rr.com. ([83.161.120.155]) by mail.devtech.com (JAMES SMTP Server 2.3.1-dev) with SMTP ID 219 for <ugarcia@example.com>; Thu, 1 Apr 2010 03:34:42 -0400 (EDT) Date: Thu, 1 Apr 2010 03:34:42 -0400 (EDT) Sender: ugarcia@example.com To: JAMES Server Development <ugarcia@example.com> From: JAMES Nightly Build System <ugarcia@example.com> Subject: JAMES Server Nightly Build Report An automated nightly build of JAMES has been posted to http://people.apache.org/builds/james/nightly/ -------------------------------------------------------------------- Any unit test errors from the build should be reported below: -------------------------------------------------------------------- BUILD SUCCESSFUL Total time: 4 minutes 32 seconds -------------------------------------------------------------------- Done. Packages Will Be Uploaded To The Nightly Repository. -------------------------------------------------------------------- Please review http://people.apache.org/builds/james/nightly/BUILD.log. --------------------------------------------------------------------- To unsubscribe, e-mail: ugarcia@example.com For additional commands, e-mail: ugarcia@example.com From server-devugarcia@example.com Thu Apr 01 07:38:47 2010 Return-Path: <server-devugarcia@example.com> Delivered-To: ugarcia@example.com Received: (qmail 64809 invoked from network); 1 Apr 2010 07:38:47 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Apr 2010 07:38:47 -0000 Received: (qmail 11119 invoked by uid 500); 1 Apr 2010 07:38:47 -0000 Delivered-To: ugarcia@example.com Received: (qmail 10915 invoked by uid 500); 1 Apr 2010 07:38:45 -0000 Mailing-List: contact ugarcia@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:ugarcia@example.com> List-Help: <mailto:ugarcia@example.com> List-Post: <mailto:ugarcia@example.com> List-Id: "James Developers List" <server-dev.james.apache.org> Reply-To: "James Developers List" <ugarcia@example.com> Delivered-To: mailing list ugarcia@example.com Received: (qmail 10907 invoked by uid 99); 1 Apr 2010 07:38:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 07:38:45 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mailrelay003.isp.belgacom.be) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 07:38:38 +0000 X-Belgacom-Dynamic: yes X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApEBAEjps0tbsJiO/2dsb2JhbAAH0BqCbwqCCAQ Received: from 142.152-176-91.adsl-dyn.isp.belgacom.be (HELO [83.161.120.155]) ([83.161.120.155]) by relay.skynet.be with ESMTP; 01 Apr 2010 09:38:18 +0200 Message-ID: <ugarcia@example.com> Date: Thu, 01 Apr 2010 09:38:17 +0200 From: Eric Charles <ugarcia@example.com> User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:83.161.120.155) Gecko/20100322 Thunderbird/3.0.3 MIME-Version: 1.0 To: James Developers List <ugarcia@example.com> Subject: Re: OutOfMemory exception References: <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> In-Reply-To: <ugarcia@example.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org 2) Launching james, memory usage is 300MB. This seems normal and holding the traffic well. At a certain time, memory jumps to 3GB (I gave much memory via -Xmx). Maybe a certain condition produces this. Debugging in eclipse shows me that many threads are created by camel. I also had the impression that some threads were falling and other ones were recreated. If this is the case, is it normal. I'm going to look at activemq/camel (I don't know them really), and also still read the http://activemq.apache.org/javalangoutofmemory.html, http://activemq.apache.org/my-producer-blocks.html,... I have the impression that everything is working fine, but at a certain moment, there is a condition that makes the memory usage rise extensively (not a linear growth, but really a stair). Bye, Eric On 04/01/2010 09:24 AM, Norman Maurer wrote: > Hi Eric, > > thx again for your tests... > > So I think we have two independent problems here: > > 1) The .m64 files which are not deleted always: > > My guess here is that there is something wrong with processing Mail > objects with state Null. I will investigate futher.. I also found a > possible problem with .m64 files which get not deleted when a > Exception is throws while processing the camel routes. I will patch > this soon. > > 2) The OOM > > My guess here is that ActiveMQ is not freeing up memory. Not sure why yet > > I think 2) is the more criticial problem and should get higher > priority to get fixed.. > > Bye, > Norman > > > > 2010/4/1 Eric Charles<ugarcia@example.com>: > >> Hi Norman, >> >> Tks for the smpt-source pointer. >> I looked at it, but finally, I think I will develop those little classes to >> have more control on it. >> I will put it on github. >> >> I launched latest trunk james yesterday evening. >> During the night, 3 hours after the launch, the following occured: >> >> KahDB slower and slower... >> >> INFO 23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase | Slow >> KahaDB access: cleanup took 2330 >> INFO 23:38:04,591 | james.mailetcontext | Error while storing mail. >> org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be >> parsed.; >> nested exception is: >> org.apache.james.imap.mailbox.StorageException: failed. Transaction >> commit failed.; >> nested exception is: >> <openjpa-1.2.1-r752877:753278 fatal store error> >> org.apache.openjpa.persistence.RollbackException: The transaction has been >> rolled back. See the nested exceptions for details on the errors that >> occurred. >> at >> org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265) >> >> at >> org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130) >> ... 55 more >> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java >> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt >> 1363215207 INSERT INTO Message (id, bodyStartOctet, content, contentOctets, >> mediaType, subType, textu >> alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int) 1260, >> (byte[]) [B@7a00e70c, (long) 47887, (String) text, (String) html, (long) >> 876]} [code=0, state=XJ001] >> at >> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192) >> at >> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57) >> at >> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866) >> at >> org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269) >> >> So still a OOM exception that was shown by yet-another-component (in this >> case, the StoreMailbox). >> >> >> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its >> job). >> All 4 files were 0 bytes. >> >> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB max >> memory). >> >> With the previous parameters ( -Xmx512m), the process was taking the whole >> 512MB. >> >> Tks, >> Eric >> >> >> On 03/31/2010 08:26 PM, Norman Maurer wrote: >> >>> Hi eric, >>> >>> thx for all your help, maybe smtp-source ( which is included in >>> postfix ) can help you with stress testing. >>> >>> Bye >>> Norman >>> >>> 2010/3/31, Eric Charles<ugarcia@example.com>: >>> >>> >>>> Hi Norman, >>>> >>>> There are not so much .m64 files, so even it it shows that a mail >>>> processing didn't reach the end and may have cause a leak, I don't think >>>> it would make crash my process so quick (between 10 and 24 hours). >>>> >>>> I quickly analysed a few dumps last week : there is always a class that >>>> took 30/40% of the memory. Sometimes a activemq class, sometimes not >>>> (don't remember the details). >>>> Also notable, the stack trace (the place where the OOM gives problems) >>>> varies. >>>> Today stack was: >>>> org.apache.camel.RuntimeCamelException: >>>> org.apache.camel.CamelExchangeException: Error processing Exchange. >>>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102, >>>> responseRequired = true, messageId = >>>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null, >>>> originalTransactionId = null, producerId = >>>> ID:srv001-51032-1270011735798-2:0:24:1, destination = >>>> queue://processor.root, transactionId = null, expiration = 0, timestamp >>>> = 1270014325685, arrival = 0, brokerInTime = 1270014326893, >>>> brokerOutTime = 1270014333023, correlationId = null, replyTo = null, >>>> persistent = true, type = null, priority = 4, groupID = null, >>>> groupSequence = 0, targetConsumerId = null, compressed = false, userID = >>>> null, content = org.apache.activemq.util.ByteSequence@69b568d0, >>>> marshalledProperties = null, dataStructure = null, redeliveryCounter = >>>> 0, size = 10718, properties = null, readOnlyProperties = true, >>>> readOnlyBody = true, droppable = false}]. Caused by: >>>> [java.lang.OutOfMemoryError - Java heap space] >>>> at >>>> >>>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055) >>>> at >>>> >>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154) >>>> ... >>>> >>>> >>>> >>>> I also think the tmp files are not the main cause. >>>> >>>> I have now james running in eclipse and I made already made a little >>>> trip in the code. Quite impressive since last time I ugarcia@example.com. >>>> I will try to stress james with a small smtp/pop3 client (I used >>>> postage, but a simple class with commons-net behind) may be easier and >>>> see what happens (eventually with eclipse profiler). >>>> >>>> It will be for this weekend for me. >>>> >>>> Bye, >>>> >>>> Eric >>>> >>>> >>>> On 03/31/2010 07:11 PM, Norman Maurer wrote: >>>> >>>> >>>>> Hi Eric, >>>>> >>>>> thx for keeping us in the loop... I'm still not sure why the .m64 >>>>> files are still in the tmp folder sometimes.. But I suspect the files >>>>> are not the cause of the OOM. >>>>> >>>>> I didn't get a OOM since yesterday morning (the time I deployed >>>>> current trunk version), but just found to new .m64 files.. >>>>> >>>>> So I'm still searching for the real cause. If nothing helps I will >>>>> need to use a profiler to find th leak. >>>>> >>>>> Bye, >>>>> Norman >>>>> >>>>> 2010/3/31 Eric Charles<ugarcia@example.com>: >>>>> >>>>> >>>>> >>>>>> Hi Norman, >>>>>> >>>>>> I had defined the Null mailet >>>>>> >>>>>> <mailet match="HostIsLocal" class="Null"> >>>>>> <processor> local-address-error</processor> >>>>>> <notice>550 - Requested action not taken: no such user here</notice> >>>>>> </mailet> >>>>>> >>>>>> but now, I use the standard config >>>>>> >>>>>> <mailet match="HostIsLocal" class="ToProcessor"> >>>>>> <processor> local-address-error</processor> >>>>>> <notice>550 - Requested action not taken: no such user here</notice> >>>>>> </mailet> >>>>>> >>>>>> I still have the OOM. >>>>>> >>>>>> I will now deploy a fresh svn with your last commits and enable the >>>>>> ValidRcptHandler. >>>>>> I keep you posted with the result, >>>>>> >>>>>> Tks, >>>>>> Eric >>>>>> >>>>>> >>>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote: >>>>>> >>>>>> >>>>>> >>>>>>> Hi Eric, >>>>>>> >>>>>>> you said all the files are related to address-errors , could you show >>>>>>> me your address error processor config? >>>>>>> Does the Problem still exist when you enable the ValidRcptHandler in >>>>>>> the smtpserver.xml file? >>>>>>> >>>>>>> Thx >>>>>>> Norman >>>>>>> >>>>>>> 2010/3/30, Eric Charles<ugarcia@example.com>: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Oops, no, the files are still there (only ugarcia@example.com). >>>>>>>> Eric >>>>>>>> >>>>>>>> >>>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Hi Norman, >>>>>>>>> >>>>>>>>> I just deployed your new commit with the new camel DisposeProcess. >>>>>>>>> Good news : I don't see anymore the m64 file in tmp (well I see 1 >>>>>>>>> file, on the second after, it is no more there, so the dispose works >>>>>>>>> as it should). >>>>>>>>> >>>>>>>>> I keep you posted with our eventual future OOM. >>>>>>>>> >>>>>>>>> Tks, >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Hi Norman, >>>>>>>>>> >>>>>>>>>> The .m64 are all to "unkown user" but to "well known domain" (so >>>>>>>>>> for >>>>>>>>>> <ugarcia@example.com>). >>>>>>>>>> They are from various size (with and without attachment). >>>>>>>>>> They are well formed (I can open the downloaded file with >>>>>>>>>> thunderbird) >>>>>>>>>> >>>>>>>>>> However, when I sent a mail to ugarcia@example.com, I don't see it in >>>>>>>>>> the /tmp >>>>>>>>>> >>>>>>>>>> Tks, >>>>>>>>>> Eric >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Hi Eric, >>>>>>>>>>> >>>>>>>>>>> sure.. we have to find the OOM cause. What would be interesting, >>>>>>>>>>> could >>>>>>>>>>> you check somehow if the .m64 files are files which are related to >>>>>>>>>>> successfully delivered mail ? >>>>>>>>>>> >>>>>>>>>>> Thx, >>>>>>>>>>> Norman >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2010/3/29 Eric Charles<ugarcia@example.com>: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Norman, >>>>>>>>>>>> >>>>>>>>>>>> Done. Now, we have to wait... >>>>>>>>>>>> >>>>>>>>>>>> I saw some created *.m64 that were removed. >>>>>>>>>>>> But there are other ones that remains in /tmp. >>>>>>>>>>>> >>>>>>>>>>>> I sometimes run a jmap (java memory map) to produce a heap dump >>>>>>>>>>>> and >>>>>>>>>>>> analyse >>>>>>>>>>>> it. >>>>>>>>>>>> At the beginning, everything seems ok, and often, when I come >>>>>>>>>>>> back, >>>>>>>>>>>> the OOM >>>>>>>>>>>> has already occured. >>>>>>>>>>>> >>>>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a quicker exception >>>>>>>>>>>> (if >>>>>>>>>>>> any, >>>>>>>>>>>> let's hope not). >>>>>>>>>>>> >>>>>>>>>>>> The *.m64 are a clue and we should ensure that they are removed >>>>>>>>>>>> in >>>>>>>>>>>> all >>>>>>>>>>>> circumstances. >>>>>>>>>>>> This may be the easy part, as we have some visible clues (the >>>>>>>>>>>> files). >>>>>>>>>>>> We should also ensure after that there are no other causes to the >>>>>>>>>>>> leaks. >>>>>>>>>>>> >>>>>>>>>>>> Tks, >>>>>>>>>>>> Eric >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Hi Eric, >>>>>>>>>>>>> >>>>>>>>>>>>> I found the cause for the not deleted temporary files. Hopefully >>>>>>>>>>>>> this >>>>>>>>>>>>> is the cause of the OOM. Could try to svn up and run again ? >>>>>>>>>>>>> >>>>>>>>>>>>> Thx, >>>>>>>>>>>>> Norman >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2010/3/29 Norman Maurer<ugarcia@example.com>: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Eric, >>>>>>>>>>>>>> >>>>>>>>>>>>>> thx for the report. I see exact the same problem today here.. >>>>>>>>>>>>>> (The >>>>>>>>>>>>>> OOM). Didn't notice the files in the tmp folder, but I think >>>>>>>>>>>>>> thats >>>>>>>>>>>>>> a >>>>>>>>>>>>>> good pointer. I will try to debug the problem later or >>>>>>>>>>>>>> tomorrow. >>>>>>>>>>>>>> But I >>>>>>>>>>>>>> suspect you are right about the tmp files and jms producers.. >>>>>>>>>>>>>> Did >>>>>>>>>>>>>> you >>>>>>>>>>>>>> run a kill -3 pid to see what threads are active etc ? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thx, >>>>>>>>>>>>>> Norman >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ps: Patches are welcome :) >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2010/3/29 Eric Charles<ugarcia@example.com>: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The last two weeks, I deployed various trunk snapshots. >>>>>>>>>>>>>>> After james running less than 1 day, I always ran into a >>>>>>>>>>>>>>> OutOfMemory >>>>>>>>>>>>>>> Exception. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I analysed the logs (for example STACK TRACE 1 in annex), and >>>>>>>>>>>>>>> always >>>>>>>>>>>>>>> found >>>>>>>>>>>>>>> Camel complaining when trying to deliver on JMS queue. >>>>>>>>>>>>>>> I tried to adapt the ActiveMQ configuration based on >>>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html but >>>>>>>>>>>>>>> nothing >>>>>>>>>>>>>>> helped. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I also analyzed various heap dump :sometimes, 65% was used by >>>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession, >>>>>>>>>>>>>>> sometimes >>>>>>>>>>>>>>> by a >>>>>>>>>>>>>>> activemq class,... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: ugarcia@example.com >>>>>>>>> For additional commands, e-mail: ugarcia@example.com >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: ugarcia@example.com >>>>>>>> For additional commands, e-mail: ugarcia@example.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: ugarcia@example.com >>>>>>> For additional commands, e-mail: ugarcia@example.com >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: ugarcia@example.com >>>>>> For additional commands, e-mail: ugarcia@example.com >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: ugarcia@example.com >>>>> For additional commands, e-mail: ugarcia@example.com >>>>> >>>>> >>>>> >>>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: ugarcia@example.com >>>> For additional commands, e-mail: ugarcia@example.com >>>> >>>> >>>> >>>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: ugarcia@example.com >>> For additional commands, e-mail: ugarcia@example.com >>> >>> >>> >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: ugarcia@example.com >> For additional commands, e-mail: ugarcia@example.com >> >> >> > --------------------------------------------------------------------- > To unsubscribe, e-mail: ugarcia@example.com > For additional commands, e-mail: ugarcia@example.com > > --------------------------------------------------------------------- To unsubscribe, e-mail: ugarcia@example.com For additional commands, e-mail: ugarcia@example.com From server-devugarcia@example.com Thu Apr 01 07:42:44 2010 Return-Path: <server-devugarcia@example.com> Delivered-To: ugarcia@example.com Received: (qmail 66208 invoked from network); 1 Apr 2010 07:42:44 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Apr 2010 07:42:44 -0000 Received: (qmail 14679 invoked by uid 500); 1 Apr 2010 07:42:43 -0000 Delivered-To: ugarcia@example.com Received: (qmail 14575 invoked by uid 500); 1 Apr 2010 07:42:41 -0000 Mailing-List: contact ugarcia@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:ugarcia@example.com> List-Help: <mailto:ugarcia@example.com> List-Post: <mailto:ugarcia@example.com> List-Id: "James Developers List" <server-dev.james.apache.org> Reply-To: "James Developers List" <ugarcia@example.com> Delivered-To: mailing list ugarcia@example.com Received: (qmail 14425 invoked by uid 99); 1 Apr 2010 07:42:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 07:42:41 +0000 X-ASF-Spam-Status: No, hits=-0.2 required=10.0 tests=AWL,FREEMAIL_FROM,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ugarcia@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-wy0-f177.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 07:42:36 +0000 Received: by wyb40 with SMTP id 40so406332wyb.22 for <ugarcia@example.com>; Thu, 01 Apr 2010 00:42:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:content-type :content-transfer-encoding; bh=h5xJGXdONGE0I4N9RpHlA5OwF3WyN92++Tg5QiXgcAI=; b=HUISlE7udHRg7bidbsCdetqSHXax9GCQPrm/Cw9KQ0oB/IrAo/IORxnlUh3yxfVox7 64/0TuaP/ipRCQ4a7bCCPyR+tY5bQB+T9xVvIPhd+rrtQ7VXSUHPLxUNVQ9lobsjH000 jrNFaWin30e9iuY0LTPLSDz3hbGV/JZAsIR2I= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=SRcdYv2Guq/rWNjXxbPutsMAKfh1Noia+6spg7cXXFuH8CeR8k5uNcQRb+iVf9epyK k78p9o02K5xVTSRB1VsR97WSaLViLlBg7G63CMp1dhsPEsX3S8WwNeg4q/QiYf3P7UIW sw9ASZyuDJIFg5m4Z42VSnS9/+JpcJy582tTE= MIME-Version: 1.0 Received: by 83.161.120.155 with HTTP; Thu, 1 Apr 2010 00:42:14 -0700 (PDT) In-Reply-To: <ugarcia@example.com> References: <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> <ugarcia@example.com> Date: Thu, 1 Apr 2010 09:42:14 +0200 Received: by 83.161.120.155 with SMTP id z14mr216801wee.129.1270107735090; Thu, 01 Apr 2010 00:42:15 -0700 (PDT) Message-ID: <ugarcia@example.com> Subject: Re: OutOfMemory exception From: Norman Maurer <ugarcia@example.com> To: James Developers List <ugarcia@example.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Eric, could you try if this helps... In spring-beans.xml replace the jmsConnectionFactory block with: <!-- jms connection pooling --> <bean id=3D"jmsConnectionFactory" class=3D"org.apache.activemq.pool.Poo= ledConnec tionFactory" destroy-method=3D"stop"> <property name=3D"connectionFactory"> <bean class=3D"org.apache.activemq.ActiveMQConnectionFactory"> <property name=3D"brokerURL"> <value>vm://localhost?broker.useJmx=3Dfalse&amp;jms.prefetchPol= icy.all =3D50</value> </property> </bean> </property> </bean> Bye, Norman 2010/4/1 Eric Charles <ugarcia@example.com>: > 2) Launching james, memory usage is 300MB. This seems normal and holding = the > traffic well. At a certain time, memory jumps to 3GB (I gave much memory = via > -Xmx). > Maybe a certain condition produces this. > Debugging in eclipse shows me that many threads are created by camel. I a= lso > had the impression that some threads were falling and other ones were > recreated. If this is the case, is it normal. > I'm going to look at activemq/camel (I don't know them really), and also > still read the http://activemq.apache.org/javalangoutofmemory.html, > http://activemq.apache.org/my-producer-blocks.html,... > > I have the impression that everything is working fine, but at a certain > moment, there is a condition that makes the memory usage rise extensively > (not a linear growth, but really a stair). > > Bye, > Eric > > > On 04/01/2010 09:24 AM, Norman Maurer wrote: >> >> Hi Eric, >> >> thx again for your tests... >> >> So I think we have two independent problems here: >> >> 1) The .m64 files which are not deleted always: >> >> My guess here is that there is something wrong with processing Mail >> objects with state Null. I will investigate futher.. I also found a >> possible problem with .m64 files which get not deleted when a >> Exception is throws while processing the camel routes. I will patch >> this soon. >> >> 2) The OOM >> >> My guess here is that ActiveMQ is not freeing up memory. Not sure why ye= t >> >> I think 2) is the more criticial problem and should get higher >> priority to get fixed.. >> >> Bye, >> Norman >> >> >> >> 2010/4/1 Eric Charles<ugarcia@example.com>: >> >>> >>> Hi Norman, >>> >>> Tks for the smpt-source pointer. >>> I looked at it, but finally, I think I will develop those little classe= s >>> to >>> have more control on it. >>> I will put it on github. >>> >>> I launched latest trunk james yesterday evening. >>> During the night, 3 hours after the launch, the following occured: >>> >>> KahDB slower and slower... >>> >>> INFO =A023:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase= | >>> Slow >>> KahaDB access: cleanup took 2330 >>> INFO =A023:38:04,591 | james.mailetcontext | Error while storing mail. >>> org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be >>> parsed.; >>> =A0nested exception is: >>> =A0 =A0org.apache.james.imap.mailbox.StorageException: failed. Transact= ion >>> commit failed.; >>> =A0nested exception is: >>> <openjpa-1.2.1-r752877:753278 fatal store error> >>> org.apache.openjpa.persistence.RollbackException: The transaction has >>> been >>> rolled back. =A0See the nested exceptions for details on the errors tha= t >>> occurred. >>> =A0 =A0at >>> >>> org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.jav= a:265) >>> >>> =A0 =A0at >>> >>> org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreM= anager.java:130) >>> =A0 =A0... 55 more >>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java >>> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt >>> 1363215207 INSERT INTO Message (id, bodyStartOctet, content, >>> contentOctets, >>> mediaType, subType, textu >>> alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=3D(long) 2678, (int) >>> 1260, >>> (byte[]) [B@7a00e70c, (long) 47887, (String) text, (String) html, (long= ) >>> 876]} [code=3D0, state=3DXJ001] >>> =A0 =A0at >>> >>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConn= ectionDecorator.java:192) >>> =A0 =A0at >>> >>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(Loggi= ngConnectionDecorator.java:57) >>> =A0 =A0at >>> >>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnectio= n$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:86= 6) >>> =A0 =A0at >>> >>> org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(D= elegatingPreparedStatement.java:269) >>> >>> So still a OOM exception that was shown by yet-another-component (in th= is >>> case, the StoreMailbox). >>> >>> >>> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its >>> job). >>> All 4 files were 0 bytes. >>> >>> I have now launched with EXTRA_JVM_ARGUMENTS=3D"-Xms512m -Xmx4g" (so 4G= B >>> max >>> memory). >>> >>> With the previous parameters ( -Xmx512m), the process was taking the >>> whole >>> 512MB. >>> >>> Tks, >>> Eric >>> >>> >>> On 03/31/2010 08:26 PM, Norman Maurer wrote: >>> >>>> >>>> Hi eric, >>>> >>>> thx for all your help, maybe smtp-source ( which is included in >>>> postfix ) can help you with stress testing. >>>> >>>> Bye >>>> Norman >>>> >>>> 2010/3/31, Eric Charles<ugarcia@example.com>: >>>> >>>> >>>>> >>>>> Hi Norman, >>>>> >>>>> There are not so much .m64 files, so even it it shows that a mail >>>>> processing didn't reach the end and may have cause a leak, I don't >>>>> think >>>>> it would make crash my process so quick (between 10 and 24 hours). >>>>> >>>>> I quickly analysed a few dumps last week : there is always a class th= at >>>>> took 30/40% of the memory. Sometimes a activemq class, sometimes not >>>>> (don't remember the details). >>>>> Also notable, the stack trace (the place where the OOM gives problems= ) >>>>> varies. >>>>> Today stack was: >>>>> org.apache.camel.RuntimeCamelException: >>>>> org.apache.camel.CamelExchangeException: Error processing Exchange. >>>>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId =3D 2102, >>>>> responseRequired =3D true, messageId =3D >>>>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination =3D n= ull, >>>>> originalTransactionId =3D null, producerId =3D >>>>> ID:srv001-51032-1270011735798-2:0:24:1, destination =3D >>>>> queue://processor.root, transactionId =3D null, expiration =3D 0, tim= estamp >>>>> =3D 1270014325685, arrival =3D 0, brokerInTime =3D 1270014326893, >>>>> brokerOutTime =3D 1270014333023, correlationId =3D null, replyTo =3D = null, >>>>> persistent =3D true, type =3D null, priority =3D 4, groupID =3D null, >>>>> groupSequence =3D 0, targetConsumerId =3D null, compressed =3D false,= userID >>>>> =3D >>>>> null, content =3D org.apache.activemq.util.ByteSequence@69b568d0, >>>>> marshalledProperties =3D null, dataStructure =3D null, redeliveryCoun= ter =3D >>>>> 0, size =3D 10718, properties =3D null, readOnlyProperties =3D true, >>>>> readOnlyBody =3D true, droppable =3D false}]. Caused by: >>>>> [java.lang.OutOfMemoryError - Java heap space] >>>>> =A0 =A0 =A0at >>>>> >>>>> >>>>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHe= lper.java:1055) >>>>> =A0 =A0 =A0at >>>>> >>>>> >>>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransaction= WithoutResult(TransactionErrorHandler.java:154) >>>>> ... >>>>> >>>>> >>>>> >>>>> I also think the tmp files are not the main cause. >>>>> >>>>> I have now james running in eclipse and I made already made a little >>>>> trip in the code. Quite impressive since last time I ugarcia@example.com. >>>>> I will try to stress james with a small smtp/pop3 client (I used >>>>> postage, but a simple class with commons-net behind) may be easier an= d >>>>> see what happens (eventually with eclipse profiler). >>>>> >>>>> It will be for this weekend for me. >>>>> >>>>> Bye, >>>>> >>>>> Eric >>>>> >>>>> >>>>> On 03/31/2010 07:11 PM, Norman Maurer wrote: >>>>> >>>>> >>>>>> >>>>>> Hi Eric, >>>>>> >>>>>> thx for keeping us in the loop... I'm still not sure why the .m64 >>>>>> files are still in the tmp folder sometimes.. But I suspect the file= s >>>>>> are not the cause of the OOM. >>>>>> >>>>>> I didn't get a OOM since yesterday morning (the time I deployed >>>>>> current trunk version), but just found to new .m64 files.. >>>>>> >>>>>> So I'm still searching for the real cause. If nothing helps I will >>>>>> need to use a profiler to find th leak.
{'source': 'Apache Software Foundation Public Mail Archives'}