title: elasticbeanstalk failed to deploy worker environment /var/pids/web.pid: no such file or directory
body: <p>I am struggling with EBS for what I figured has a relatively simple architecture. I have a Django app. This Django app will contain code for both my worker environment and my WebServer.I have been deploying to a WebServer just fine, and only have this issue with my Worker environment as I try and incorporate it. <a href="https://stackoverflow.com/questions/42435032/separate-environment-config-elastic-beanstalk">This poor chap</a> had the same issue, <a href="https://forums.aws.amazon.com/message.jspa?messageID=685803" rel="noreferrer">which the community</a> doesn't seem to have a good answer for.</p>
  <p>This was mistake number 1 I learned, due to not having a great way to separate configurations.</p>
  <p>It began with me getting this error:</p>
  <blockquote>
  <p>2020/09/03 20:17:59.065285 [ERROR] update processes [web sqsd nginx healthd cfn-hup] pid symlinks failed with error Read pid source file /var/pids/web.pid failed with error:open /var/pids/web.pid: no such file or directory</p>
  </blockquote>
  <p>which from <a href="https://forums.aws.amazon.com/thread.jspa?threadID=326968" rel="noreferrer">this link</a>, I thought was due to my having my <code>.platform</code> with an nginx overwrite (only modifying gzip to turn it on) was somehow causing this. I had removed it and redeployed and the error wasn't there, so it seemed logical. Unfortunately it seems rather temperamental, because it's come back (after I have already deployed a few times!!!!).</p>
  <p>My first attempt to fix this was using <code>saved_configs/</code></p>
  <pre><code>├── .elasticbeanstalk
  │   ├── config.yml
  │   └── saved_configs
  │       ├── web.cfg.yml
  │       └── worker.cfg.yml
  </code></pre>
  <p>and calling it like this:</p>
  <pre><code>eb create testweb --cfg web
  eb create testwrkr -t worker --cfg worker
  eb deploy testweb
  eb deploy testwrkr
  </code></pre>
  <p>which seemed to work, but I still couldn't deploy my worker environment. Next I tried to create a Makefile since removing the .platform fixed the error for me yesterday.</p>
  <pre><code>ENVS = 'production development'
  ifndef env
  $(error &quot;env&quot; is not specified. Please use one of: $(ENVS))
  endif
  
  create_web: clean copy_web_configuration
      @echo &quot;Creating Environment: $(env)&quot;
      -eb create $(env)
  
  create_worker: clean copy_worker_configuration
      @echo &quot;Creating Worker Environment: $(env)&quot;
      -eb create $(env) -t worker
  
  deploy_web: clean copy_web_configuration
      @echo &quot;Deploying to: $(env)&quot;
      -eb deploy $(env) $(args)
      @$(MAKE) clean
  
  deploy_worker: clean copy_worker_configuration
      @echo &quot;Deploying worker to: $(env)&quot;
      -eb deploy $(env) $(args)
      @$(MAKE) clean
  
  copy_web_configuration:
      @cp -r config/ebs/extensions/shared/ .ebextensions/
      @cp -r config/ebs/extensions/web/ .ebextensions/
      @cp -r config/ebs/platform/web/ .platform/
  
  copy_worker_configuration:
      @cp -r config/ebs/extensions/shared/ .ebextensions/
      @cp -r config/ebs/extensions/worker/ .ebextensions/
  
  clean:
      @find .ebextensions/ -maxdepth 1 -type f -exec rm -f {} \;
      @rm -rf .platform/nginx
      @find .platform/ -maxdepth 1 -type f -exec rm -f {} \;
  </code></pre>
  <p>so I could be 100% sure that whatever was in <code>.ebextensions/</code> or <code>.platform/</code> was for the intended platform.</p>
  <pre><code>My new file tree looks like this:
  .ebextensions/
  .platform/
  config/
  ├── __init__.py
  ├── ebs
  │   ├── extensions
  │   │   ├── shared
  │   │   │   ├── 01_packages.config
  │   │   │   ├── appslog.config
  │   │   │   └── django.config
  │   │   ├── web
  │   │   │   ├── db-migrate.config
  │   │   │   ├── securelistener-clb.config
  │   │   │   └── static.config
  │   │   └── worker
  │   │       └── worker.config
  │   └── platform
  │       └── web
  │           └── nginx
  │               └── nginx.conf
  ├── settings
  │   ├── __init__.py
  │   ├── base.py
  │   ├── local.py
  │   └── production.py
  ├── urls.py
  └── wsgi.py
  
  </code></pre>
  <p>And now, when I deploy (after adding in <code>cron.yaml</code>), I get my old friend again</p>
  <blockquote>
  <p>2020/09/04 22:12:21.485660 [INFO] Executing instruction: Track pids in healthd
  2020/09/04 22:12:21.485677 [INFO] This is an enhanced health env...
  2020/09/04 22:12:21.485697 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf aws-eb.target | cut -d= -f2
  2020/09/04 22:12:21.491871 [INFO] nginx.service healthd.service cfn-hup.service sqsd.service</p>
  <p>2020/09/04 22:12:21.491894 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf eb-app.target | cut -d= -f2
  2020/09/04 22:12:21.496690 [INFO] web.service</p>
  <p>2020/09/04 22:12:21.496761 [ERROR] update processes [web nginx healthd cfn-hup sqsd] pid symlinks failed with error Read pid source file /var/pids/web.pid failed with error:open /var/pids/web.pid: no such file or directory
  2020/09/04 22:12:21.496772 [ERROR] An error occurred during execution of command [app-deploy] - [Track pids in healthd]. Stop running the command. Error: update processes [web nginx healthd cfn-hup sqsd] pid symlinks failed with error Read pid source file /var/pids/web.pid failed with error:open /var/pids/web.pid: no such file or directory</p>
  <p>2020/09/04 22:12:21.496776 [INFO] Executing cleanup logic
  2020/09/04 22:12:21.496861 [INFO] CommandService Response: {&quot;status&quot;:&quot;FAILURE&quot;,&quot;api_version&quot;:&quot;1.0&quot;,&quot;results&quot;:[{&quot;status&quot;:&quot;FAILURE&quot;,&quot;msg&quot;:&quot;Engine execution has encountered an error.&quot;,&quot;returncode&quot;:1,&quot;events&quot;:[{&quot;msg&quot;:&quot;Instance deployment successfully generated a 'Procfile'.&quot;,&quot;timestamp&quot;:1599257531,&quot;severity&quot;:&quot;INFO&quot;},{&quot;msg&quot;:&quot;Instance deployment failed. For details, see 'eb-engine.log'.&quot;,&quot;timestamp&quot;:1599257541,&quot;severity&quot;:&quot;ERROR&quot;}]}]}</p>
  </blockquote>
  <p>My worker configuration doesn't have much in it, so I'm at a loss for words on why this won't deploy. Has anyone seen this issue before?
  The only resources I have found online are:</p>
  <ul>
  <li><a href="https://forums.aws.amazon.com/thread.jspa?messageID=955011&amp;#955011" rel="noreferrer">https://forums.aws.amazon.com/thread.jspa?messageID=955011&amp;#955011</a> (no answer available)</li>
  <li><a href="https://forums.aws.amazon.com/thread.jspa?threadID=326968" rel="noreferrer">https://forums.aws.amazon.com/thread.jspa?threadID=326968</a></li>
  </ul>
  <hr />
  <p>I monitored the logs at the same time and saw in an order of operations that first I get this error:</p>
  <pre><code>Sep  4 22:42:18 ip-172-31-7-235 web: File &quot;/usr/lib64/python3.7/importlib/__init__.py&quot;, line 127, in import_module
  Sep  4 22:42:18 ip-172-31-7-235 web: return _bootstrap._gcd_import(name[level:], package, level)
  Sep  4 22:42:18 ip-172-31-7-235 web: File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 1006, in _gcd_import
  Sep  4 22:42:18 ip-172-31-7-235 web: File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 983, in _find_and_load
  Sep  4 22:42:18 ip-172-31-7-235 web: File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 965, in _find_and_load_unlocked
  Sep  4 22:42:18 ip-172-31-7-235 web: ModuleNotFoundError: No module named 'application'
  Sep  4 22:42:18 ip-172-31-7-235 web: [2020-09-04 22:42:18 +0000] [9303] [INFO] Worker exiting (pid: 9303)
  Sep  4 22:42:18 ip-172-31-7-235 web: [2020-09-04 22:42:18 +0000] [9296] [INFO] Shutting down: Master
  Sep  4 22:42:18 ip-172-31-7-235 web: [2020-09-04 22:42:18 +0000] [9296] [INFO] Reason: Worker failed to boot.
  </code></pre>
  <p>and then</p>
  <pre><code>2020/09/04 22:42:21.717799 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf eb-app.target | cut -d= -f2
  2020/09/04 22:42:21.722604 [INFO] web.service
  
  2020/09/04 22:42:21.722678 [ERROR] update processes [web healthd nginx sqsd cfn-hup] pid symlinks failed with error Read pid source file /var/pids/web.pid failed with error:open /var/pids/web.pid: no such file or directory
  2020/09/04 22:42:21.722689 [ERROR] An error occurred during execution of command [app-deploy] - [Track pids in healthd]. Stop running the command. Error: update processes [web healthd nginx sqsd cfn-hup] pid symlinks failed with error Read pid source file /var/pids/web.pid failed with error:open /var/pids/web.pid: no such file or directory 
  
  2020/09/04 22:42:21.722694 [INFO] Executing cleanup logic
  2020/09/04 22:42:21.722778 [INFO] CommandService Response: {&quot;status&quot;:&quot;FAILURE&quot;,&quot;api_version&quot;:&quot;1.0&quot;,&quot;results&quot;:[{&quot;status&quot;:&quot;FAILURE&quot;,&quot;msg&quot;:&quot;Engine execution has encountered an error.&quot;,&quot;returncode&quot;:1,&quot;events&quot;:[{&quot;msg&quot;:&quot;Instance deployment successfully generated a 'Procfile'.&quot;,&quot;timestamp&quot;:1599259331,&quot;severity&quot;:&quot;INFO&quot;},{&quot;msg&quot;:&quot;Instance deployment failed. For details, see 'eb-engine.log'.&quot;,&quot;timestamp&quot;:1599259341,&quot;severity&quot;:&quot;ERROR&quot;}]}]}
  </code></pre>
  <p>so I'm guessing the <code>web.service</code> that isn't being found is because the <code>ModuleNotFound</code> error. What I don't understand is I am using the exact same code repo as the Web environment.. so how could the Worker environment fail? The configuration looks okay to me. My tree is above, and the relevant beanstalk configuration is this:</p>
  <pre><code>  aws:elasticbeanstalk:container:python:
      WSGIPath: config.wsgi:application
  </code></pre>