<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 3.1.25 (Cipater)</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
CELERY_ACCEPT_CONTENT: ['pickle', 'json', 'msgpack', 'yaml']
CELERY_RESULT_BACKEND: 'djcelery.backends.database:DatabaseBackend'
BROKER_URL: 'redis://redis:6379/1'
</p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 3.6.8</li>
<li><strong>Minimal Celery Version</strong>: 3.1.25</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: redis 2.10.6</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
amqp==1.4.9
anyjson==0.3.3
asgi-redis==1.4.3
asgiref==1.1.2
asn1crypto==0.24.0
attrs==19.1.0
autobahn==19.7.1
Automat==0.7.0
beautifulsoup4==4.6.3
billiard==3.3.0.23
blinker==1.4
celery==3.1.25
certifi==2019.6.16
cffi==1.11.5
channels==1.1.8
chardet==3.0.4
Click==7.0
clinner==1.12.3
colorlog==3.2.0
constantly==15.1.0
coreapi==2.3.3
coreschema==0.0.4
cryptography==2.4.2
daphne==1.3.0
ddt==1.2.1
decorator==4.4.0
Django==1.11.12
django-accredible==0.5.0
django-activity-stream==0.6.5
django-anonymizer===0.5.0.16-bw
django-appconf==1.0.2
django-autocomplete-light==3.2.9
django-autoslug==1.9.3
django-backupdb==0.7.0
django-bootstrap-breadcrumbs==0.8.2
django-bootstrap-pagination==1.6.2
django-bootstrap3==9.0.0
django-braces==1.11.0
django-celery==3.2.1
django-channels-presence==0.0.9
django-cors-headers==3.0.2
django-exo-mentions==1.0.0
django-extensions==2.1.6
django-extra-fields==0.9
django-filter==1.0.4
django-form-utils==1.0.3
django-generic-m2m==0.3.1
django-guardian==1.4.9
django-markdown==0.8.4
django-model-utils==3.0.0
django-multiselectfield==0.1.8
django-password-reset==0.9
django-ratings-models==0.1.10
django-redis-sessions==0.6.1
django-rest-params==1.0.2
django-rest-swagger==2.1.2
django-sizefield==0.9.1
django-stats-redis==0.1.1
django-storages==1.6.3
django-tagulous==0.14.0
django-timezone-field==2.1
django-timezone-utils==0.11
django-typeform-feedback==0.1.5
django-versionfield2==0.5.0
djangorestframework==3.9.4
djangorestframework-camel-case==1.0.3
djangorestframework-jwt==1.11.0
drf-nested-routers==0.90.0
drf-openapi==1.3.0
exo-changelog==0.1.3
factory-boy==2.7.0
fake-factory==0.5.7
Faker==0.7.7
filestack-python==2.3.1
flatbuffers==1.11
future==0.17.1
gitdb==0.6.4
gitdb2==2.0.5
GitPython==2.1.12
graphene==1.4.1
graphene-django==1.3
graphql-core==1.1
graphql-relay==0.4.5
gunicorn==19.9.0
health-check==3.4.1
hyperlink==19.0.0
icalendar==4.0.1
idna==2.7
incremental==17.5.0
inflection==0.3.1
ipython==6.1.0
ipython-genutils==0.2.0
iso8601==0.1.12
itypes==1.1.0
jedi==0.14.1
Jinja2==2.10.1
kombu==3.0.37
lib-exo-messages==1.3.2
lxml==4.2.5
Markdown==3.1.1
MarkupSafe==1.1.1
meld3==1.0.2
mimeparse==0.1.3
msgpack-python==0.5.6
olefile==0.43
openapi-codec==1.3.2
parso==0.5.1
pexpect==4.7.0
pickleshare==0.7.5
pilkit==2.0
Pillow==5.3.0
promise==2.2.1
prompt-toolkit==1.0.16
psycopg2==2.7.4
psycopg2-binary==2.7.4
ptyprocess==0.6.0
pyaccredible==0.6.0
pyasn1==0.4.5
pyasn1-modules==0.2.5
pycairo==1.16.3
pycparser==2.19
pycrypto==2.6.1
pycurl==7.43.0.2
Pygments==2.4.2
pygobject==3.28.2
PyHamcrest==1.9.0
PyJWT==1.7.1
pyOpenSSL==18.0.0
python-dateutil==2.7.5
pytz==2019.1
PyYAML==3.12
redis==2.10.6
requests==2.20.0
requests-toolbelt==0.9.1
sentry-sdk==0.10.2
service-identity==18.1.0
setproctitle==1.1.10
simplegeneric==0.8.1
simplejson==3.16.0
singledispatch==3.4.0.3
six==1.12.0
smmap==0.9.0
smmap2==2.0.5
supervisor==4.0.4
trafaret==1.2.0
traitlets==4.3.2
Twisted==19.2.1
txaio==18.8.1
uritemplate==3.0.0
urllib3==1.24.3
wcwidth==0.1.7
XlsxWriter==1.1.8
zope.interface==4.6.0
</p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="docker exec service sh
./manage.py shell

from example.celery import debug_task
from datetime import datetime
debug_task.apply_async(eta=datetime(2019,10,10,10))

exit container
docker restart containerID
"><pre class="notranslate"><span class="pl-s1">docker</span> <span class="pl-s1">exec</span> <span class="pl-s1">service</span> <span class="pl-s1">sh</span>
.<span class="pl-c1">/</span><span class="pl-s1">manage</span>.<span class="pl-s1">py</span> <span class="pl-s1">shell</span>

<span class="pl-k">from</span> <span class="pl-s1">example</span>.<span class="pl-s1">celery</span> <span class="pl-k">import</span> <span class="pl-s1">debug_task</span>
<span class="pl-k">from</span> <span class="pl-s1">datetime</span> <span class="pl-k">import</span> <span class="pl-s1">datetime</span>
<span class="pl-s1">debug_task</span>.<span class="pl-en">apply_async</span>(<span class="pl-s1">eta</span><span class="pl-c1">=</span><span class="pl-en">datetime</span>(<span class="pl-c1">2019</span>,<span class="pl-c1">10</span>,<span class="pl-c1">10</span>,<span class="pl-c1">10</span>))

<span class="pl-s1">exit</span> <span class="pl-s1">container</span>
<span class="pl-s1">docker</span> <span class="pl-s1">restart</span> <span class="pl-s1">containerID</span></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">After docker container restarts should have:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="celery@18346e3912df: OK
    * {'eta': '2019-10-10T10:00:00+00:00', 'priority': 6, 'request': {'id': '306f5dec-0bcd-4894-a2da-4a1234e3c10d', 'name': 'exolever.celery.debug_task', 'args': '[]', 'kwargs': '{}', 'hostname': 'celery@18346e3912df', 'time_start': None, 'acknowledged': False, 'delivery_info': {'exchange': 'celery', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}, 'worker_pid': None}}"><pre class="notranslate"><code class="notranslate">celery@18346e3912df: OK
    * {'eta': '2019-10-10T10:00:00+00:00', 'priority': 6, 'request': {'id': '306f5dec-0bcd-4894-a2da-4a1234e3c10d', 'name': 'exolever.celery.debug_task', 'args': '[]', 'kwargs': '{}', 'hostname': 'celery@18346e3912df', 'time_start': None, 'acknowledged': False, 'delivery_info': {'exchange': 'celery', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}, 'worker_pid': None}}
</code></pre></div>
<h1 dir="auto">Actual Behavior</h1>

<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="-&gt; celery@18346e3912df: OK
    - empty -"><pre class="notranslate"><code class="notranslate">-&gt; celery@18346e3912df: OK
    - empty -
</code></pre></div>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?utf8=%E2%9C%93&amp;q=is%3Aissue+label%3A%22Category%3A+Documentation%22+">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues in this issue<br>
(If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<p dir="auto">I've not found any documentation on embedding Celery Workers into my Python program. Everything so far has been extending workers to add functionality when the workers start or stop. The closest thing I found was the Py.Test <code class="notranslate">celery/contrib/testing/app.py</code> code but I couldn't figure out how, the setup I've got is similar.</p>
<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h1 dir="auto">Description</h1>

<p dir="auto">I'm trying to embed Celery into a CherryPy server. I'm not looking for a lot of concurrency. So, a single process multi-threaded application is desirable for long running tasks that won't return in a reasonable time for web applications.</p>
<p dir="auto">It seems like both frameworks have good extending documentation with very similar usage patterns. So the reverse could be asked as well, how do I embed a CherryPy server into Celery?</p>
<ul dir="auto">
<li><a href="https://docs.cherrypy.org/en/latest/extend.html" rel="nofollow">https://docs.cherrypy.org/en/latest/extend.html</a></li>
<li><a href="https://docs.celeryproject.org/en/stable/userguide/extending.html" rel="nofollow">https://docs.celeryproject.org/en/stable/userguide/extending.html</a></li>
</ul>
<p dir="auto">Here's the code I have so far.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from threading import Thread
from cherrypy.process import plugins
from celery.bin import worker as celery_worker
from .config import get_config
from .tasks import celery_app

def run_celery_worker():
    &quot;&quot;&quot;Run the main solo worker.&quot;&quot;&quot;
    worker = celery_worker.worker(app=celery_app)
    options = {
        'broker': get_config().get('celery', 'broker_url'),
        'loglevel': 'INFO',
        'traceback': True,
        'pool': 'eventlet'
    }
    return worker.run(**options)

class CeleryThreadPlugin(plugins.SimplePlugin):
    def start(self):
        self.bus.log('Starting up Celery worker')
        self.celery_thread = Thread(target=run_celery_worker)
        self.celery_thread.start()

    def stop(self):
        self.bus.log('Stopping down Celery worker')
        celery_app.loader.shutdown_worker()
        sleep(10)
        # The Celery worker.run() isn't returning
        self.bus.log('Celery worker is {}'.format(self.celery_thread.isAlive()))
        self.celery_thread.join()
        self.bus.log('Celery worker joined')"><pre class="notranslate"><code class="notranslate">from threading import Thread
from cherrypy.process import plugins
from celery.bin import worker as celery_worker
from .config import get_config
from .tasks import celery_app

def run_celery_worker():
    """Run the main solo worker."""
    worker = celery_worker.worker(app=celery_app)
    options = {
        'broker': get_config().get('celery', 'broker_url'),
        'loglevel': 'INFO',
        'traceback': True,
        'pool': 'eventlet'
    }
    return worker.run(**options)

class CeleryThreadPlugin(plugins.SimplePlugin):
    def start(self):
        self.bus.log('Starting up Celery worker')
        self.celery_thread = Thread(target=run_celery_worker)
        self.celery_thread.start()

    def stop(self):
        self.bus.log('Stopping down Celery worker')
        celery_app.loader.shutdown_worker()
        sleep(10)
        # The Celery worker.run() isn't returning
        self.bus.log('Celery worker is {}'.format(self.celery_thread.isAlive()))
        self.celery_thread.join()
        self.bus.log('Celery worker joined')
</code></pre></div>
<p dir="auto">I'm using <code class="notranslate">memory:///</code> for my <code class="notranslate">broker_url</code> similar to the Py.Test integration. The problem is shutdown is not clean. The main script runs and brings up both CherryPy and Celery, when I hit Ctrl-C CherryPy handles the shutdown sequence. The <code class="notranslate">worker.run()</code> method doesn't seem to return to stop the worker thread.</p>
<p dir="auto">I've tried various ways to issue the worker shutdown and none of them seem to make <code class="notranslate">worker.run()</code> return.</p>
<h1 dir="auto">Suggestions</h1>

<p dir="auto">It would be nice to have some good documentation on how to embed Celery properly in another application. CherryPy does have similar <a href="https://docs.cherrypy.org/en/latest/deploy.html#embedding-into-another-wsgi-framework" rel="nofollow">docs</a> and I've followed them to success embedding CherryPy into specialized Python applications.</p>
<p dir="auto">Some things to consider:</p>
<ul dir="auto">
<li>Is it okay for Celery workers to be managed from child Python thread (not main)?</li>
<li>What would the API be for that usage?</li>
</ul>