title: weblogic context lookup error : java.rmi.UnmarshalException: error unmarshalling arguments
body: <p>We are facing an issue in our production env. We have searched the net high and low and we were not able to come up with any answers.
  This error(stacktrace below) occurs when an ejb lookup is made from managed server 1 to manager server 2. Virtual ip is used for the lookup. It occurs intermittently and at random intervals. We are not able to identify any pattern and If the ejb call is attempted two or three times, it gets through successfully.</p>
  
  <p>Env details :
  server : weblogic 10.0 MP1 running on java 1.5
  os : solaris</p>
  
  <p>Pls revert if any other details are required.</p>
  
  <p>Source used for lookup :</p>
  
  <pre><code>private TreControlRemote getController() throws Exception {
      Context context = null;
      Properties p = new Properties();
      TreControlHome treHome = null;
      TreControlRemote remote = null;
      ConfigurationLoader lAppLoader = null;
      try {
          mLog.debug("Entering");
          lAppLoader = PropertiesFileLoader.getInstance("context.properties");
          p.put(Context.INITIAL_CONTEXT_FACTORY, lAppLoader.getValue("INITIAL_CONTEXT_FACTORY"));
          p.put(Context.PROVIDER_URL, lAppLoader.getValue("PROVIDER_URL"));
          context = new InitialContext(p);
          mLog.debug("context : " + context.getEnvironment());
          remote = null;
          treHome = (TreControlHome) context.lookup("CONTROL");
          mLog.debug("Object ---&gt;&gt;&gt;&gt;" + treHome);
          remote = (TreControlRemote) treHome.create();
          mLog.debug("Leaving");
      } catch (Exception ex) {
          mLog.fatal("Exception while getting remote", ex);
          ex.printStackTrace();
          throw ex;
      } finally {
          lAppLoader = null;
      }
      return remote;
  }
  </code></pre>
  
  <p>The url is a virtual ip pointing to managed server 2 and it contains a ejb with jndi "CONTROL". The problem is that it successful on certain occassions and fails randomly with the error:</p>
  
  <p>stack trace of the error :</p>
  
  <pre><code>*javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
  java.io.StreamCorruptedException]
  at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:74)
  at weblogic.jndi.internal.WLContextImpl.translateException(WLContextImpl.java:426)
  at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:382)
  at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:367)
  at javax.naming.InitialContext.lookup(InitialContext.java:351)
  ```````````````````````````````````````````````````````````````````
  Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
  java.io.StreamCorruptedException
  at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:221)
  at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:338)
  at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:252)
  at weblogic.jndi.internal.ServerNamingNode_1001_WLStub.lookup(Unknown Source)
  at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:379)
  ... 33 more
  Caused by: java.io.StreamCorruptedException
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1332)
  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
  at weblogic.utils.io.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:195)
  at weblogic.rjvm.MsgAbbrevInputStream.readObject(MsgAbbrevInputStream.java:565)
  at weblogic.utils.io.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:191)
  at weblogic.jndi.internal.RootNamingNode_WLSkel.invoke(Unknown Source)
  at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:589)
  at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:224)
  at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:479)
  at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
  at weblogic.security.service.SecurityManager.runAs(Unknown Source)
  at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:475)
  at weblogic.rmi.internal.BasicServerRef.access$300(BasicServerRef.java:59)
  at weblogic.rmi.internal.BasicServerRef$BasicExecuteRequest.run(BasicServerRef.java:1016)
  ... 2 more*
  </code></pre>
  
  <p>Obtained the below mentioned stacktrace from the weblogic log. Could this error be related to our problem mentioned above?</p>
  
  <pre><code>*####&lt;Aug 25, 2009 2:11:04 AM BST&gt; &lt;Info&gt; &lt;RJVM&gt; &lt;pkssv049&gt; &lt;M1AP4&gt; &lt;ACTIVE ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'&gt; &lt;&lt;WLS Kernel&gt;&gt; &lt;1251162664181&gt; &lt;BEA-000513&gt; &lt;Failure in heartbeat trigger for RJVM: 5433424963141690658S:169.93.73.0:10040,10040,-1,-1,-1,-1,-1:pkssv049.***.net:10240,pkssv049.***.net:10241,pkssv050.***.net:10240,pkssv050.***.net:10241:LIQP1_LMSDomain:M1AP3
  java.io.IOException: The connection manager to ConnectionManager for: 'weblogic.rjvm.RJVMImpl@189ed0e - id: '5433424963141690658S:169.93.73.0:10040,10040,-1,-1,-1,-1,-1:pkssv049.***.net:10240,pkssv049.***.net:10241,pkssv050.***.net:10240,pkssv050.***.net:10241:LIQP1_LMSDomain:M1AP3' connect time: 'Mon Aug 24 20:24:02 BST 2009'' has already been shut down.
  java.io.IOException: The connection manager to ConnectionManager for: 'weblogic.rjvm.RJVMImpl@189ed0e - id: '5433424963141690658S:169.93.73.0:10040,10040,-1,-1,-1,-1,-1:pkssv049.***.net:10240,pkssv049.***.net:10241,pkssv050.***.net:10240,pkssv050.***.net:10241:LIQP1_LMSDomain:M1AP3' connect time: 'Mon Aug 24 20:24:02 BST 2009'' has already been shut down
  at weblogic.rjvm.ConnectionManager.getOutputStream(ConnectionManager.java:1686)
  at weblogic.rjvm.ConnectionManager.createHeartbeatMsg(ConnectionManager.java:1629)
  at weblogic.rjvm.ConnectionManager.sendHeartbeatMsg(ConnectionManager.java:607)
  at weblogic.rjvm.RJVMImpl$HeartbeatChecker.timerExpired(RJVMImpl.java:1540)
  at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
  at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:464)
  at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
  at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)*
  </code></pre>
  
  <p>Any help would be greatly appreciated.</p>
  
  <p>Here is some additional info..</p>
  
  <p>Is the problem intermittent, or does reproduce every single time? If the problem is intermittent, do you know what conditions it occurs under?
  <em>It occurs intermittently and we are not able to observe any pattern.</em></p>
  
  <p>Are there any other errors/warnings logged either on the local server or on the remote server?
  <em>We see a lot of connection refused errors in the weblogic log</em></p>
  
  <p>Are both the managed servers in the same domain?
  <em>Yes</em></p>