Title: Blocking in classloading while quering hazelcast
Tags: classloader;hazelcast
Question: We are using hazelcast as a distributed cache. After the application runs for a certain time we start to get blocking in classloading. Following is the stacktrace :

java.lang.Thread.State: BLOCKED (on object monitor)      at java.lang.ClassLoader.loadClass(ClassLoader.java:404)      - locked <
0x00002acaac4c4718> (a java.lang.Object)      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)      at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:124)      at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:97)      at com.hazelcast.nio.IOUtil$1.resolveClass(IOUtil.java:113)      at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)      at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)      at com.hazelcast.nio.serialization.DefaultSerializers$ObjectSerializer.read(DefaultSerializers.java:196)      at com.hazelcast.nio.serialization.StreamSerializerAdapter.toObject(StreamSerializerAdapter.java:65)      at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:260)      at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:186)      at com.hazelcast.map.impl.AbstractMapServiceContextSupport.toObject(AbstractMapServiceContextSupport.java:42)      at com.hazelcast.map.impl.DefaultMapServiceContext.toObject(DefaultMapServiceContext.java:28)      at com.hazelcast.map.impl.proxy.MapProxySupport.toObject(MapProxySupport.java:1038)      at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:84)

Hazelcast is loading the class every-time it de-serializes an object. I am not sure why classloading is required each time.

Can somebody please help.

Comment: how did you address this issue ?


Here is another answer: This is not Hazelcast specific, whenever you create an instance you have to ask the classloader for the class, no matter you use reflection or a new call. The problem is actually when synchronized classloaders come into the play (like in webapps or stuff). Hazelcast, obviously, has to deserialize a lot and therefore requests a lot of classes.

The internal deserialization is kind of optimized by now (by caching the conctructor instances - as far as I remember) but the Java standard serialization (the one you use) always wants the class and classes aren't yet cached.

Comment for this answer: There is no way around it, as I mentioned, this is a Java issue and not specific to Hazelcast. You can only hack / wrap the ClassLoader and implement a caching strategy on your own. Afterwards you pass the wrapping ClassLoader instance to Hazelcast's config as the config classloader.
Comment for this answer: The problem with that approach is (and the reason it wasn't merged), it won't work for certain containers. Especially containers like OSGi and application servers are a burden to keep working. The constructor cache was kind of the middle way to go but it is not a perfect solution, I agree. The problem is, the whole classloading thing needs to be rewritten to make it more adjustable and less complex - providing support for further frameworks always break another one.
Comment for this answer: Maybe it changed by now but trying it in the past I had lots of weird memleaks
Comment for this answer: so what do you suggest to get out of this problem? I've started facing the same issue and as you mentioned, there is a synchronized block in classloader.
Comment for this answer: Ok. Instead of wrapping the class loader, I kind of replicated the `ConstructorCache` class in `ClassLoaderUtil` in hz to create `ClassCacheMap`, which will cache the classes.  Later I saw that almost identical changes were done by 'Mr.Easy' too against some PR, but never got integrated with hz codeline (I presume).  I hope that it might give me some good results.
I'm about to run some tests with the patched jar now.
Comment for this answer: please let me know if you see any problems with approach.
Comment for this answer: Hi noctarius, I understood the second part where you said that whole class loading code in runtime needs to be re-written. However, in the first part, what is the problem that you faced with OSGI container. I'm running the modified code in an OSGI container and my preliminary tests shows that it is working fine. So could you please elaborate on the problem that you faced so that I can get a head start there ?
Comment for this answer: Thanks @noctarius, I'll keep an eye on memory leaks in the system, going forward.

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 1, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2015:00:04/21/15', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<classloader><hazelcast>', 'Title': 'Blocking in classloading while quering hazelcast', 'ViewCount': 579, 'id': 29771612, 'user_AboutMe': '<p>Software developer at Impetus Infotech(india) pvt. ltd.</p>\n', 'user_AccountId': 1684773.0, 'user_DisplayName': 'anurag arya', 'user_Reputation': 71.0, 'user_Views': 21.0, 'user_id': 1548279.0}}