title: On stopping and restarting my application, I receive an address already in use error for JMX
body: <p>I'm writing a script to stop and restart my service.  Ideally, it will be called by the JVM options, set in my gradle.build file, in the event of an out of memory error.  So far, it has been called and successfully killed the process, but when it calls the start script, I receive an error that says that the JMX port I also have set in my build file is already in use:</p>
  
  <pre><code>Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 9010; nested exception is:
          java.net.BindException: Address already in use (Bind failed)
  sun.management.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 9010; nested exception is:
          java.net.BindException: Address already in use (Bind failed)
          at sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:480)
          at sun.management.Agent.startAgent(Agent.java:262)
          at sun.management.Agent.startAgent(Agent.java:452)
  Caused by: java.rmi.server.ExportException: Port already in use: 9010; nested exception is:
          java.net.BindException: Address already in use (Bind failed)
          at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:346)
          at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:254)
          at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411)
          at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
          at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:236)
          at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:213)
          at sun.rmi.registry.RegistryImpl.&lt;init&gt;(RegistryImpl.java:173)
          at sun.management.jmxremote.SingleEntryRegistry.&lt;init&gt;(SingleEntryRegistry.java:49)
          at sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:816)
          at sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:468)
          ... 2 more
  Caused by: java.net.BindException: Address already in use (Bind failed)
  </code></pre>
  
  <p>This does not make any sense to me, as I assume killing the service should also free up the port.  </p>
  
  <p>Here are my JVM options:</p>
  
  <pre><code>task appStartScripts(type: CreateStartScripts) {
      def tplName = 'startTemplate.sh'
      assert project.file(tplName).exists()
      defaultJvmOpts = ["-XX:+HeapDumpOnOutOfMemoryError",
                        "-XX:HeapDumpPath=\$HOME/log/",
                        "-Dcom.sun.management.jmxremote",
                        "-Dcom.sun.management.jmxremote.port=9010",
                        "-Dcom.sun.management.jmxremote.authenticate=false",
                        "-Dcom.sun.management.jmxremote.ssl=false",
                        "-Djava.rmi.server.hostname=testHost",
                        "-XX:OnOutOfMemoryError=./restart.sh",
                        "-Xms64m", "-Xmx124m"]
      dependsOn shadowJar
      applicationName = 'start'
      defaultJvmOpts += ["-Dspring.profiles.active=development"]
      classpath = startShadowScripts.classpath
      mainClassName = startShadowScripts.mainClassName
      outputDir = new File(project.buildDir, 'scriptsShadow')
  
      doLast {
          // IMPORTANT! needed to ensure HOME environment variable is expanded correctly
          unixScript.text = unixScript.text.replace('\\$HOME', '\'"$HOME"\'')
  }
  </code></pre>
  
  <p>And here is my restart script:</p>
  
  <pre><code>#!/usr/bin/env sh
  
  envDeploy=`whoami`
  
  if [ "$envDeploy" == "dev_account" ]; then
      envName=development
  elif [ "$envDeploy" == "dev_account" ]
  then
      envName=quality
  elif [ "$envDeploy" == "prod_account" ]
  then
      envName=production
  fi
  
  PID=`pgrep -f application.jar`
  
  kill -9 $PID
  sleep 15
  echo "Restarting application for \$envName environment"
  ./start.sh $envName;
  </code></pre>
  
  <p>In theory, the service should be completely killed before calling start.sh, so I'm a bit baffled on what else I can do to resolve this.</p>
  
  <p>EDIT:
  I should also mention that the restart script is able to kill and restart the service fine when called on its own:</p>
  
  <pre><code>./restart.sh #typing this into the console works as expected
  </code></pre>
  
  <p>This behavior is only observed when it is called on an out of memory exception, that is, when this line is called:</p>
  
  <pre><code>-XX:OnOutOfMemoryError=./restart.sh
  </code></pre>
  
  <p>Edit2:
  I've made some modifications to my restart script to try to find where it is failing:</p>
  
  <pre><code>#!/usr/bin/env sh
  
  envDeploy=`whoami`
  
  if [ "$envDeploy" == "dev_account" ]; then
      envName=development
  elif [ "$envDeploy" == "dev_account" ]
  then
      envName=quality
  elif [ "$envDeploy" == "prod_account" ]
  then
      envName=production
  fi
  
  PID=`pgrep -f application.jar` &gt;&gt; check.log
  
  echo "PID is $PID" &gt;&gt; check.log
  
  netstat -nlp | grep $PID &gt;&gt; check.log
  kill -9 $PID &gt;&gt; check.log
  ps $PID &gt;&gt; check.log
  netstat -nlp | grep $PID &gt;&gt; check.log
  kill -9 $PID &gt;&gt; check.log
  echo "checking defunct processes" &gt;&gt; check.log
  ps -ef | grep defunct &gt;&gt; check.log
  sleep 15
  echo "===========" &gt;&gt; check.log
  echo "checking jmx port" &gt;&gt; check.log
  netstat -nlp | grep $PID &gt;&gt; check.log
  netstat -nlp | grep 9010 &gt;&gt; check.log
  kill $PID &gt;&gt; check.log
  netstat -tupln |grep ":&lt;Your_Port_Here&gt;" &gt;&gt; check.log
  sleep 5 &gt;&gt; check.log
  jps &gt;&gt; check.log
  echo "Restarting application for $envName environment" &gt;&gt; check.log
  ./start.sh $envName; &gt;&gt; check.log
  </code></pre>
  
  <p>After adding in <code>netstat -nlp | grep 9010 &gt;&gt; check.log</code>, I was able to see the following output from check.log:</p>
  
  <pre><code>PID is 11959
  tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      11959/java
  tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN      11959/java
  tcp        0      0 0.0.0.0:9010            0.0.0.0:*               LISTEN      11959/java
  tcp        0      0 0.0.0.0:38933           0.0.0.0:*               LISTEN      11959/java
  tcp        0      0 0.0.0.0:36257           0.0.0.0:*               LISTEN      11959/java
    PID TTY      STAT   TIME COMMAND
  11959 pts/0    Zl     0:38 [java] &lt;defunct&gt;
  checking defunct processes
  sysibrt+ 13296 13287  0 17:43 pts/0    00:00:00 grep defunct
  ===========
  checking jmx port
  tcp        0      0 0.0.0.0:9010            0.0.0.0:*               LISTEN      13287/sh
  13370 Jps
  Restarting automation-cfg-svc for qa environment
  </code></pre>
  
  <p>The process ID for port 9010 changed after the OOm error, indicating that the service had restarted.  This should not happen, as there is nothing before <code>./start.sh $envName;</code> to restart the service.</p>