title: Why is Selenium Grid behaving differently when starting it from within Java code than outside?
body: <p>I want to start and shutdown a selenium grid from within a spring boot application. <strong>The grid is accessed periodically in parallel using several threads to get the response from several websites with java script support.</strong>
  When i start the grid <strong>before</strong> i start the application, everything is working fine, however when i start the grid from within the java code, <strong>the first time the grid is used by some threads it is working as intended</strong>, however after the thread pool is finished + terminated and the process is repeated <strong>the program throws following error</strong>:</p>
  <pre><code>org.openqa.selenium.WebDriverException: Error forwarding the new session Error forwarding the request timeout
  Command duration or timeout: 30.05 seconds
      at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64)
      at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
      at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
      at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
      at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
      at org.openqa.selenium.remote.JsonWireProtocolResponse.lambda$errorHandler$0(JsonWireProtocolResponse.java:54)
      at org.openqa.selenium.remote.HandshakeResponse.lambda$getResponseFunction$0(HandshakeResponse.java:30)
      at org.openqa.selenium.remote.ProtocolHandshake.lambda$createSession$0(ProtocolHandshake.java:126)
      at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
      at java.base/java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958)
      at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127)
      at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502)
      at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488)
      at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
      at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150)
      at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:543)
      at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:128)
      at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:74)
      at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:136)
      at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)
      at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:213)
      at org.openqa.selenium.remote.RemoteWebDriver.&lt;init&gt;(RemoteWebDriver.java:131)
      at org.openqa.selenium.remote.RemoteWebDriver.&lt;init&gt;(RemoteWebDriver.java:144)
      at net.JsSeleniumGrid.getHtml(JsSeleniumGrid.java:100)
      at net.CrawlCareerLinkRunner.run(CrawlCareerLinkRunner.java:27)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
      at java.base/java.lang.Thread.run(Thread.java:832)
  Caused by: org.openqa.selenium.WebDriverException: Error forwarding the new session Error forwarding the request timeout
  Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'
  System info: host: 'olw', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '5.11.0-34-generic', java.version: '15.0.2'
  Driver info: driver.version: unknown
      at org.openqa.grid.web.servlet.handler.RequestHandler.process(RequestHandler.java:118)
      at org.openqa.grid.web.servlet.DriverServlet.process(DriverServlet.java:85)
      at org.openqa.grid.web.servlet.DriverServlet.doPost(DriverServlet.java:69)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      at org.seleniumhq.jetty9.servlet.ServletHolder.handle(ServletHolder.java:865)
      at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle(ServletHandler.java:535)
      at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
      at org.seleniumhq.jetty9.security.SecurityHandler.handle(SecurityHandler.java:548)
      at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
      at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
      at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
      at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
      at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
      at org.seleniumhq.jetty9.servlet.ServletHandler.doScope(ServletHandler.java:473)
      at org.seleniumhq.jetty9.server.session.SessionHandler.doScope(SessionHandler.java:1564)
      at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
      at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
      at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
      at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      at org.seleniumhq.jetty9.server.Server.handle(Server.java:503)
      at org.seleniumhq.jetty9.server.HttpChannel.handle(HttpChannel.java:364)
      at org.seleniumhq.jetty9.server.HttpConnection.onFillable(HttpConnection.java:260)
      at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
      at org.seleniumhq.jetty9.io.FillInterest.fillable(FillInterest.java:103)
      at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
      at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
      at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
      at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
      at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
      at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
      at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
      at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
      at java.lang.Thread.run(Thread.java:829)
  </code></pre>
  <p>I seem to be able to still instantiate the <code>RemoteWebDriver</code> object since the logger prints:</p>
  <pre><code>2021-09-15 16:28:22.300  INFO 39465 --- [ null to remote] o.o.selenium.remote.ProtocolHandshake    : Detected dialect: W3C
  </code></pre>
  <p>The class i am using to start or shutdown the grid is following:</p>
  <pre><code>@Configuration
  @Slf4j
  public class SeleniumGrid {
  
      public Process hub;
      public Process node;
      public String gridPath;
      private final ProcessBuilder processBuilder;
      private final ChromeOptions chromeOptions;
  
      public SeleniumGrid(ChromeOptions chromeOptions){
          gridPath = String.format(System.getProperty(&quot;user.dir&quot;)+&quot;/src/main/resources/grid/&quot;);
          log.info(&quot;Directory path of Selenium Grid set to {}&quot;,gridPath);
          processBuilder = new ProcessBuilder().directory(new File(gridPath));
          this.chromeOptions = chromeOptions;
      }
  
      public boolean restartSeleniumGrid(){
          return (shutdownSeleniumGrid() &amp;&amp; startSeleniumGrid());
      }
  
      public boolean startSeleniumGrid(){
          try {
              this.hub = executeCommand(&quot;./startHub.sh&quot;);
              this.node = executeCommand(&quot;./startNode.sh&quot;);
              log.info(&quot;Selenium grid started successfully.&quot;);
              Thread.sleep(20000);
          } catch (IOException | InterruptedException e) {
              e.printStackTrace();
              return false;
          } 
  
          return true;
      }
  
      public boolean shutdownSeleniumGrid(){
          if(this.node != null){
              this.node.children().forEach((child) -&gt; child.destroy());
              this.node.destroy();
          }
          if(this.hub != null){
              this.hub.children().forEach((child) -&gt; child.destroy());
              this.hub.destroy();
          }
          try {
              Thread.sleep(10000);
          } catch (InterruptedException e) {
              e.printStackTrace();
              return false;
          }
  
          return true;
      }
  
      private Process executeCommand(String cmd) throws IOException {
          processBuilder.command(cmd);
          return processBuilder.start();
      }
  }
  </code></pre>
  <p>Whereas ./startHub.sh and ./startNode.sh are just wrapper scripts for the longer commands needed to start the grid.</p>
  <p>startHub.sh:</p>
  <pre><code>#!/bin/bash
  echo 'Starting Selenium Grid in HUB mode'
  java -jar selenium-server-standalone-3.141.59.jar -role hub -hubConfig hubConfig.json
  </code></pre>
  <p>startNode.sh</p>
  <pre><code>#!/bin/bash
  echo 'Starting Selenium Grid in NODE mode'
  java -jar selenium-server-standalone-3.141.59.jar -role node -nodeConfig nodeConfig.json
  </code></pre>
  <p>When i debug i can connect to the Grid interface completely normal with <code>http://localhost:4444/</code>, everything behaves fine up until that second attempt. The problem clearly does not occur because of some configuration within the <code>WebDriver</code> or the Grid / Node since it is working perfectly fine when i manually start it before the program execution in 2 separate shell sessions (1 Hub + 1 Node). Any ideas what could be the possible problem ?</p>