<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>:</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate"></code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Celery Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate"></code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<p dir="auto">When a task is revoked and/or worker is mingling and receives revoked task data, when adding or updating it's store with the new info, it should always purge it's data (let the purge function do it's job by asking the data to decide which IDs have expired and should be discarded). It doesn't seem clear to me why this isn't the case (rational/history, etc).</p>
<h1 dir="auto">Actual Behavior</h1>

<p dir="auto">When revoking a task (whether by expiration or some other process/control), the documentation for the data structure on each worker holding the revoked task IDs says that when adding a task to it, if the maxlen is reached, or other items have reached expiration, those respective items will be removed from the LimitedSet data structure.</p>
<p dir="auto">Docs for the LimitedSet arguments (specifically on expires): <a href="https://docs.celeryproject.org/en/stable/internals/reference/celery.utils.collections.html#celery.utils.collections.LimitedSet" rel="nofollow">https://docs.celeryproject.org/en/stable/internals/reference/celery.utils.collections.html#celery.utils.collections.LimitedSet</a> say that when an item is added, items that should be expired will be. The add docs only mention maxlen logic being applied, for when purge is internally called. Now, I understand why this check is made, but it also leads to a bug where we're waiting (for who knows how long), for the next task to be revoked...and eventually we hope to reach the max, or another worker will come online and force us to update our set, and that method will do the proper thing, in a <em>small</em>/<em>wishful</em> thinking number of cases. It only works well (on revoke sharing and general revocation) if maxlen is reached.</p>
<p dir="auto">Bug in the docs and in the source code for the logic in add: <a href="https://docs.celeryproject.org/en/stable/internals/reference/celery.utils.collections.html#celery.utils.collections.LimitedSet.add" rel="nofollow">https://docs.celeryproject.org/en/stable/internals/reference/celery.utils.collections.html#celery.utils.collections.LimitedSet.add</a> (the source code for <code class="notranslate">add</code> only purges if maxlen is hit).</p>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="456640934" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/5605" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/5605/hovercard" href="https://github.com/celery/celery/issues/5605">#5605</a> describes the issue as seen when using the Consul backend, though I believe the issue described here applies to all backends.</li>
<li>Also, I believe kombu/transport/consul.py is at least exposed to the same issue in Consul, but I have no idea of its impact.</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>:</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ celery -A paiyroll report

software -&gt; celery:4.3.0 (rhubarb) kombu:4.5.0 py:3.7.3
            billiard:3.6.0.0 py-amqp:2.4.2
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:5.0.0-20-generic imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:consul://localhost:8500/

ABSOLUTE_URL_OVERRIDES: {
 }
ADMINS: []
ALLOWED_HOSTS: ['*']
APPEND_SLASH: True
AUTHENTICATION_BACKENDS: ['django.contrib.auth.backends.ModelBackend']
AUTH_PASSWORD_VALIDATORS: '********'
AUTH_USER_MODEL: 'paiyroll.User'
BASE_DIR: '/main/srhaque/Innovatieltd/source'
CACHES: {
 'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}}
CACHE_MIDDLEWARE_ALIAS: 'default'
CACHE_MIDDLEWARE_KEY_PREFIX: '********'
CACHE_MIDDLEWARE_SECONDS: 600
CELERY_BROKER_HEARTBEAT: None
CELERY_BROKER_URL: 'amqp://guest:********@localhost:5672//'
CELERY_RESULT_BACKEND: 'consul://localhost:8500/'
CSRF_COOKIE_AGE: 31449600
CSRF_COOKIE_DOMAIN: None
CSRF_COOKIE_HTTPONLY: False
CSRF_COOKIE_NAME: 'csrftoken'
CSRF_COOKIE_PATH: '/'
CSRF_COOKIE_SAMESITE: 'Lax'
CSRF_COOKIE_SECURE: False
CSRF_FAILURE_VIEW: 'django.views.csrf.csrf_failure'
CSRF_HEADER_NAME: 'HTTP_X_CSRFTOKEN'
CSRF_TRUSTED_ORIGINS: []
CSRF_USE_SESSIONS: False
DATABASES: {
    'default': {   'ATOMIC_REQUESTS': False,
                   'AUTOCOMMIT': True,
                   'CONN_MAX_AGE': 0,
                   'ENGINE': 'django.db.backends.postgresql',
                   'HOST': 'localhost',
                   'NAME': 'foo',
                   'OPTIONS': {},
                   'PASSWORD': '********',
                   'PORT': '5432',
                   'TEST': {   'CHARSET': None,
                               'COLLATION': None,
                               'MIRROR': None,
                               'NAME': None},
                   'TIME_ZONE': None,
                   'USER': 'dbcoreuser'},
    'fdw': {   'ATOMIC_REQUESTS': False,
               'AUTOCOMMIT': True,
               'CONN_MAX_AGE': 0,
               'ENGINE': 'django.db.backends.postgresql',
               'HOST': 'localhost',
               'NAME': 'foo',
               'OPTIONS': {},
               'PASSWORD': '********',
               'PORT': '5432',
               'TEST': {   'CHARSET': None,
                           'COLLATION': None,
                           'MIRROR': None,
                           'NAME': None},
               'TIME_ZONE': None,
               'USER': 'dbcoreuser'}}
DATABASE_ROUTERS: '********'
DATA_UPLOAD_MAX_MEMORY_SIZE: 2621440
DATA_UPLOAD_MAX_NUMBER_FIELDS: 1000
DATETIME_FORMAT: 'N j, Y, P'
DATETIME_INPUT_FORMATS: ['%Y-%m-%d %H:%M:%S',
 '%Y-%m-%d %H:%M:%S.%f',
 '%Y-%m-%d %H:%M',
 '%Y-%m-%d',
 '%m/%d/%Y %H:%M:%S',
 '%m/%d/%Y %H:%M:%S.%f',
 '%m/%d/%Y %H:%M',
 '%m/%d/%Y',
 '%m/%d/%y %H:%M:%S',
 '%m/%d/%y %H:%M:%S.%f',
 '%m/%d/%y %H:%M',
 '%m/%d/%y']
DATE_FORMAT: 'N j, Y'
DATE_INPUT_FORMATS: ['%Y-%m-%d',
 '%m/%d/%Y',
 '%m/%d/%y',
 '%b %d %Y',
 '%b %d, %Y',
 '%d %b %Y',
 '%d %b, %Y',
 '%B %d %Y',
 '%B %d, %Y',
 '%d %B %Y',
 '%d %B, %Y']
DEBUG: True
DEBUG_PROPAGATE_EXCEPTIONS: False
DECIMAL_SEPARATOR: '.'
DEFAULT_CHARSET: 'utf-8'
DEFAULT_CONTENT_TYPE: 'text/html'
DEFAULT_DB: 'default'
DEFAULT_EXCEPTION_REPORTER_FILTER: 'django.views.debug.SafeExceptionReporterFilter'
DEFAULT_FILE_STORAGE: 'django.core.files.storage.FileSystemStorage'
DEFAULT_FROM_EMAIL: 'webmaster@localhost'
DEFAULT_INDEX_TABLESPACE: ''
DEFAULT_TABLESPACE: ''
DISALLOWED_USER_AGENTS: []
EMAIL_BACKEND: 'django.core.mail.backends.filebased.EmailBackend'
EMAIL_FILE_PATH: '/tmp/email_messages'
EMAIL_HOST: 'smtp.gmail.com'
EMAIL_HOST_PASSWORD: '********'
EMAIL_HOST_USER: 'paiyroll.com@gmail.com'
EMAIL_PORT: 587
EMAIL_SSL_CERTFILE: ''
EMAIL_SSL_KEYFILE: '********'
EMAIL_SUBJECT_PREFIX: '[Django] '
EMAIL_TIMEOUT: None
EMAIL_USE_LOCALTIME: False
EMAIL_USE_SSL: False
EMAIL_USE_TLS: True
FDW_DB: 'fdw'
FILE_CHARSET: 'utf-8'
FILE_UPLOAD_DIRECTORY_PERMISSIONS: None
FILE_UPLOAD_HANDLERS: ['django.core.files.uploadhandler.MemoryFileUploadHandler',
 'django.core.files.uploadhandler.TemporaryFileUploadHandler']
FILE_UPLOAD_MAX_MEMORY_SIZE: 2621440
FILE_UPLOAD_PERMISSIONS: None
FILE_UPLOAD_TEMP_DIR: None
FIRST_DAY_OF_WEEK: 0
FIXTURE_DIRS: []
FORCE_SCRIPT_NAME: None
FORMAT_MODULE_PATH: None
FORM_RENDERER: 'django.forms.renderers.DjangoTemplates'
IGNORABLE_404_URLS: []
INSTALLED_APPS: ['paiyroll.apps.PaiyrollConfig',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.postgres',
 'django_jinja',
 'bootstrapform_jinja',
 'import_export',
 'phonenumber_field',
 'polymorphic',
 'viewflow']
INTERNAL_IPS: []
LANGUAGES: [('af', 'Afrikaans'),
 ('ar', 'Arabic'),
 ('ast', 'Asturian'),
 ('az', 'Azerbaijani'),
 ('bg', 'Bulgarian'),
 ('be', 'Belarusian'),
 ('bn', 'Bengali'),
 ('br', 'Breton'),
 ('bs', 'Bosnian'),
 ('ca', 'Catalan'),
 ('cs', 'Czech'),
 ('cy', 'Welsh'),
 ('da', 'Danish'),
 ('de', 'German'),
 ('dsb', 'Lower Sorbian'),
 ('el', 'Greek'),
 ('en', 'English'),
 ('en-au', 'Australian English'),
 ('en-gb', 'British English'),
 ('eo', 'Esperanto'),
 ('es', 'Spanish'),
 ('es-ar', 'Argentinian Spanish'),
 ('es-co', 'Colombian Spanish'),
 ('es-mx', 'Mexican Spanish'),
 ('es-ni', 'Nicaraguan Spanish'),
 ('es-ve', 'Venezuelan Spanish'),
 ('et', 'Estonian'),
 ('eu', 'Basque'),
 ('fa', 'Persian'),
 ('fi', 'Finnish'),
 ('fr', 'French'),
 ('fy', 'Frisian'),
 ('ga', 'Irish'),
 ('gd', 'Scottish Gaelic'),
 ('gl', 'Galician'),
 ('he', 'Hebrew'),
 ('hi', 'Hindi'),
 ('hr', 'Croatian'),
 ('hsb', 'Upper Sorbian'),
 ('hu', 'Hungarian'),
 ('hy', 'Armenian'),
 ('ia', 'Interlingua'),
 ('id', 'Indonesian'),
 ('io', 'Ido'),
 ('is', 'Icelandic'),
 ('it', 'Italian'),
 ('ja', 'Japanese'),
 ('ka', 'Georgian'),
 ('kab', 'Kabyle'),
 ('kk', 'Kazakh'),
 ('km', 'Khmer'),
 ('kn', 'Kannada'),
 ('ko', 'Korean'),
 ('lb', 'Luxembourgish'),
 ('lt', 'Lithuanian'),
 ('lv', 'Latvian'),
 ('mk', 'Macedonian'),
 ('ml', 'Malayalam'),
 ('mn', 'Mongolian'),
 ('mr', 'Marathi'),
 ('my', 'Burmese'),
 ('nb', 'Norwegian Bokmål'),
 ('ne', 'Nepali'),
 ('nl', 'Dutch'),
 ('nn', 'Norwegian Nynorsk'),
 ('os', 'Ossetic'),
 ('pa', 'Punjabi'),
 ('pl', 'Polish'),
 ('pt', 'Portuguese'),
 ('pt-br', 'Brazilian Portuguese'),
 ('ro', 'Romanian'),
 ('ru', 'Russian'),
 ('sk', 'Slovak'),
 ('sl', 'Slovenian'),
 ('sq', 'Albanian'),
 ('sr', 'Serbian'),
 ('sr-latn', 'Serbian Latin'),
 ('sv', 'Swedish'),
 ('sw', 'Swahili'),
 ('ta', 'Tamil'),
 ('te', 'Telugu'),
 ('th', 'Thai'),
 ('tr', 'Turkish'),
 ('tt', 'Tatar'),
 ('udm', 'Udmurt'),
 ('uk', 'Ukrainian'),
 ('ur', 'Urdu'),
 ('vi', 'Vietnamese'),
 ('zh-hans', 'Simplified Chinese'),
 ('zh-hant', 'Traditional Chinese')]
LANGUAGES_BIDI: ['he', 'ar', 'fa', 'ur']
LANGUAGE_CODE: 'en-us'
LANGUAGE_COOKIE_AGE: None
LANGUAGE_COOKIE_DOMAIN: None
LANGUAGE_COOKIE_NAME: 'django_language'
LANGUAGE_COOKIE_PATH: '/'
LOCALE_PATHS: []
LOGGING: {
    'disable_existing_loggers': False,
    'formatters': {   'standard': {   'format': '%(asctime)s [%(levelname)s] '
                                                '%(name)s: %(message)s'}},
    'handlers': {   'file': {   'backupCount': 10,
                                'class': 'logging.handlers.RotatingFileHandler',
                                'filename': '/main/srhaque/.local/share/paiyroll/django.log',
                                'formatter': 'standard',
                                'level': 'DEBUG',
                                'maxBytes': 15728640}},
    'loggers': {'': {'handlers': ['file'], 'level': 'INFO', 'propagate': True}},
    'version': 1}
LOGGING_CONFIG: 'logging.config.dictConfig'
LOGGING_DIR: '/main/srhaque/.local/share/paiyroll'
LOGIN_REDIRECT_URL: '/accounts/profile/'
LOGIN_URL: '/accounts/login/'
LOGOUT_REDIRECT_URL: None
MANAGERS: []
MEDIA_ROOT: ''
MEDIA_URL: ''
MESSAGE_STORAGE: 'django.contrib.messages.storage.fallback.FallbackStorage'
MIDDLEWARE: ['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'project.middleware.SessionExpiryMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']
MIGRATION_MODULES: {
 }
MONTH_DAY_FORMAT: 'F j'
NUMBER_GROUPING: 0
PAIYROLL_CLIENT_OUTPUT_DIR: '/tmp'
PAIYROLL_DEBUG_ADMIN_ACCESS_ALL_WORKFLOWS: True
PAIYROLL_DEBUG_NOTIFICATIONS: True
PAIYROLL_DEBUG_PAY_WALK: 0
PAIYROLL_DEBUG_PROMPTS: 259
PAIYROLL_DEBUG_SCHEDULING_DATE: '2018-01-01'
PAIYROLL_DEBUG_SCHEDULING_FAST: 10
PAIYROLL_GB_RTI_ALL_TRANSACT: None
PAIYROLL_GB_RTI_OTHERS_TRANSACT_FROM: '2019-04-06'
PAIYROLL_GB_RTI_TEST_COMPANIES: 
    ('HMRC', 'HMRC RTI Recognition')
PAIYROLL_GB_RTI_TEST_COMPANIES_ACTUAL_T: '2019-03-31'
PAIYROLL_INTERNAL_NETWORK: IPv4Network('192.168.1.0/24')
PAIYROLL_NOTIFICATIONS: {
    'email': {   'backend': 'django.core.mail.backends.filebased.EmailBackend',
                 'file_path': '/tmp/email_messages',
                 'host': 'smtp.gmail.com',
                 'password': '********',
                 'port': 587,
                 'ssl_certfile': '',
                 'ssl_keyfile': '********',
                 'timeout': None,
                 'use_ssl': False,
                 'use_tls': True,
                 'username': 'paiyroll.com@gmail.com'},
    'sms': {   'account_sid': 'ACbabf740b7a3d23010cbe381a8a184fe0',
               'from': '+15005550006',
               'password': '********'}}
PAIYROLL_REPORT_SPEC_DIR: '/main/srhaque/Innovatieltd/source/paiyroll/report/JasperReports'
PAIYROLL_REPORT_SPEC_SPREADSHEET: '/main/srhaque/Innovatieltd/source/paiyroll/report/Spreadsheets'
PAIYROLL_SITE_NAME: 'login.paiyroll.com'
PASSWORD_HASHERS: '********'
PASSWORD_RESET_TIMEOUT_DAYS: '********'
PREPEND_WWW: False
ROOT_URLCONF: 'project.urls'
SECRET_KEY: '********'
SECURE_BROWSER_XSS_FILTER: False
SECURE_CONTENT_TYPE_NOSNIFF: False
SECURE_HSTS_INCLUDE_SUBDOMAINS: False
SECURE_HSTS_PRELOAD: False
SECURE_HSTS_SECONDS: 0
SECURE_PROXY_SSL_HEADER: None
SECURE_REDIRECT_EXEMPT: []
SECURE_SSL_HOST: None
SECURE_SSL_REDIRECT: False
SERVER_EMAIL: 'root@localhost'
SESSION_CACHE_ALIAS: 'default'
SESSION_COOKIE_AGE: 5400
SESSION_COOKIE_DOMAIN: None
SESSION_COOKIE_HTTPONLY: True
SESSION_COOKIE_NAME: 'sessionid'
SESSION_COOKIE_PATH: '/'
SESSION_COOKIE_SAMESITE: 'Lax'
SESSION_COOKIE_SECURE: False
SESSION_ENGINE: 'django.contrib.sessions.backends.db'
SESSION_EXPIRE_AT_BROWSER_CLOSE: False
SESSION_FILE_PATH: None
SESSION_SAVE_EVERY_REQUEST: False
SESSION_SERIALIZER: 'django.contrib.sessions.serializers.JSONSerializer'
SETTINGS_MODULE: 'project.settings'
SHORT_DATETIME_FORMAT: 'm/d/Y P'
SHORT_DATE_FORMAT: 'm/d/Y'
SIGNING_BACKEND: 'django.core.signing.TimestampSigner'
SILENCED_SYSTEM_CHECKS: []
STATICFILES_DIRS: []
STATICFILES_FINDERS: ['django.contrib.staticfiles.finders.FileSystemFinder',
 'django.contrib.staticfiles.finders.AppDirectoriesFinder']
STATICFILES_STORAGE: 'django.contrib.staticfiles.storage.StaticFilesStorage'
STATIC_ROOT: '../staticroot'
STATIC_URL: '/static/'
TEMPLATES: [{'APP_DIRS': True,
  'BACKEND': 'django_jinja.backend.Jinja2',
  'DIRS': ['/main/srhaque/Innovatieltd/source/templates'],
  'OPTIONS': {'app_dirname': 'templates',
              'auto_reload': True,
              'autoescape': True,
              'bytecode_cache': {'backend': 'django_jinja.cache.BytecodeCache',
                                 'enabled': False,
                                 'name': 'default'},
              'constants': {},
              'context_processors': ['django.contrib.auth.context_processors.auth',
                                     'django.contrib.messages.context_processors.messages'],
              'extensions': ['jinja2.ext.do',
                             'jinja2.ext.loopcontrols',
                             'jinja2.ext.with_',
                             'jinja2.ext.i18n',
                             'jinja2.ext.autoescape',
                             'django_jinja.builtins.extensions.CsrfExtension',
                             'django_jinja.builtins.extensions.CacheExtension',
                             'django_jinja.builtins.extensions.TimezoneExtension',
                             'django_jinja.builtins.extensions.UrlsExtension',
                             'django_jinja.builtins.extensions.StaticFilesExtension',
                             'django_jinja.builtins.extensions.DjangoFiltersExtension',
                             'jinja_extensions.DebugExtension'],
              'filters': {},
              'globals': {},
              'match_extension': ('.html',
                                  '.jinja'),
              'match_regex': '^(?!admin/).*',
              'newstyle_gettext': True,
              'tests': {},
              'translation_engine': 'django.utils.translation',
              'undefined': None}},
 {'APP_DIRS': True,
  'BACKEND': 'django.template.backends.django.DjangoTemplates',
  'DIRS': ['/main/srhaque/Innovatieltd/source/templates'],
  'OPTIONS': {'context_processors': ['django.template.context_processors.debug',
                                     'django.template.context_processors.request',
                                     'django.contrib.auth.context_processors.auth',
                                     'django.contrib.messages.context_processors.messages']}}]
TEST_NON_SERIALIZED_APPS: []
TEST_RUNNER: 'django.test.runner.DiscoverRunner'
THOUSAND_SEPARATOR: ','
TIME_FORMAT: 'P'
TIME_INPUT_FORMATS: ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M']
TIME_ZONE: 'UTC'
USE_I18N: True
USE_L10N: True
USE_THOUSAND_SEPARATOR: False
USE_TZ: True
USE_X_FORWARDED_HOST: False
USE_X_FORWARDED_PORT: False
WSGI_APPLICATION: 'project.wsgi.application'
X_FRAME_OPTIONS: 'SAMEORIGIN'
YEAR_MONTH_FORMAT: 'F Y'
is_overridden: &lt;bound method Settings.is_overridden of &lt;Settings &quot;project.settings&quot;&gt;&gt;"><pre class="notranslate"><code class="notranslate">$ celery -A paiyroll report

software -&gt; celery:4.3.0 (rhubarb) kombu:4.5.0 py:3.7.3
            billiard:3.6.0.0 py-amqp:2.4.2
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:5.0.0-20-generic imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:consul://localhost:8500/

ABSOLUTE_URL_OVERRIDES: {
 }
ADMINS: []
ALLOWED_HOSTS: ['*']
APPEND_SLASH: True
AUTHENTICATION_BACKENDS: ['django.contrib.auth.backends.ModelBackend']
AUTH_PASSWORD_VALIDATORS: '********'
AUTH_USER_MODEL: 'paiyroll.User'
BASE_DIR: '/main/srhaque/Innovatieltd/source'
CACHES: {
 'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}}
CACHE_MIDDLEWARE_ALIAS: 'default'
CACHE_MIDDLEWARE_KEY_PREFIX: '********'
CACHE_MIDDLEWARE_SECONDS: 600
CELERY_BROKER_HEARTBEAT: None
CELERY_BROKER_URL: 'amqp://guest:********@localhost:5672//'
CELERY_RESULT_BACKEND: 'consul://localhost:8500/'
CSRF_COOKIE_AGE: 31449600
CSRF_COOKIE_DOMAIN: None
CSRF_COOKIE_HTTPONLY: False
CSRF_COOKIE_NAME: 'csrftoken'
CSRF_COOKIE_PATH: '/'
CSRF_COOKIE_SAMESITE: 'Lax'
CSRF_COOKIE_SECURE: False
CSRF_FAILURE_VIEW: 'django.views.csrf.csrf_failure'
CSRF_HEADER_NAME: 'HTTP_X_CSRFTOKEN'
CSRF_TRUSTED_ORIGINS: []
CSRF_USE_SESSIONS: False
DATABASES: {
    'default': {   'ATOMIC_REQUESTS': False,
                   'AUTOCOMMIT': True,
                   'CONN_MAX_AGE': 0,
                   'ENGINE': 'django.db.backends.postgresql',
                   'HOST': 'localhost',
                   'NAME': 'foo',
                   'OPTIONS': {},
                   'PASSWORD': '********',
                   'PORT': '5432',
                   'TEST': {   'CHARSET': None,
                               'COLLATION': None,
                               'MIRROR': None,
                               'NAME': None},
                   'TIME_ZONE': None,
                   'USER': 'dbcoreuser'},
    'fdw': {   'ATOMIC_REQUESTS': False,
               'AUTOCOMMIT': True,
               'CONN_MAX_AGE': 0,
               'ENGINE': 'django.db.backends.postgresql',
               'HOST': 'localhost',
               'NAME': 'foo',
               'OPTIONS': {},
               'PASSWORD': '********',
               'PORT': '5432',
               'TEST': {   'CHARSET': None,
                           'COLLATION': None,
                           'MIRROR': None,
                           'NAME': None},
               'TIME_ZONE': None,
               'USER': 'dbcoreuser'}}
DATABASE_ROUTERS: '********'
DATA_UPLOAD_MAX_MEMORY_SIZE: 2621440
DATA_UPLOAD_MAX_NUMBER_FIELDS: 1000
DATETIME_FORMAT: 'N j, Y, P'
DATETIME_INPUT_FORMATS: ['%Y-%m-%d %H:%M:%S',
 '%Y-%m-%d %H:%M:%S.%f',
 '%Y-%m-%d %H:%M',
 '%Y-%m-%d',
 '%m/%d/%Y %H:%M:%S',
 '%m/%d/%Y %H:%M:%S.%f',
 '%m/%d/%Y %H:%M',
 '%m/%d/%Y',
 '%m/%d/%y %H:%M:%S',
 '%m/%d/%y %H:%M:%S.%f',
 '%m/%d/%y %H:%M',
 '%m/%d/%y']
DATE_FORMAT: 'N j, Y'
DATE_INPUT_FORMATS: ['%Y-%m-%d',
 '%m/%d/%Y',
 '%m/%d/%y',
 '%b %d %Y',
 '%b %d, %Y',
 '%d %b %Y',
 '%d %b, %Y',
 '%B %d %Y',
 '%B %d, %Y',
 '%d %B %Y',
 '%d %B, %Y']
DEBUG: True
DEBUG_PROPAGATE_EXCEPTIONS: False
DECIMAL_SEPARATOR: '.'
DEFAULT_CHARSET: 'utf-8'
DEFAULT_CONTENT_TYPE: 'text/html'
DEFAULT_DB: 'default'
DEFAULT_EXCEPTION_REPORTER_FILTER: 'django.views.debug.SafeExceptionReporterFilter'
DEFAULT_FILE_STORAGE: 'django.core.files.storage.FileSystemStorage'
DEFAULT_FROM_EMAIL: 'webmaster@localhost'
DEFAULT_INDEX_TABLESPACE: ''
DEFAULT_TABLESPACE: ''
DISALLOWED_USER_AGENTS: []
EMAIL_BACKEND: 'django.core.mail.backends.filebased.EmailBackend'
EMAIL_FILE_PATH: '/tmp/email_messages'
EMAIL_HOST: 'smtp.gmail.com'
EMAIL_HOST_PASSWORD: '********'
EMAIL_HOST_USER: 'paiyroll.com@gmail.com'
EMAIL_PORT: 587
EMAIL_SSL_CERTFILE: ''
EMAIL_SSL_KEYFILE: '********'
EMAIL_SUBJECT_PREFIX: '[Django] '
EMAIL_TIMEOUT: None
EMAIL_USE_LOCALTIME: False
EMAIL_USE_SSL: False
EMAIL_USE_TLS: True
FDW_DB: 'fdw'
FILE_CHARSET: 'utf-8'
FILE_UPLOAD_DIRECTORY_PERMISSIONS: None
FILE_UPLOAD_HANDLERS: ['django.core.files.uploadhandler.MemoryFileUploadHandler',
 'django.core.files.uploadhandler.TemporaryFileUploadHandler']
FILE_UPLOAD_MAX_MEMORY_SIZE: 2621440
FILE_UPLOAD_PERMISSIONS: None
FILE_UPLOAD_TEMP_DIR: None
FIRST_DAY_OF_WEEK: 0
FIXTURE_DIRS: []
FORCE_SCRIPT_NAME: None
FORMAT_MODULE_PATH: None
FORM_RENDERER: 'django.forms.renderers.DjangoTemplates'
IGNORABLE_404_URLS: []
INSTALLED_APPS: ['paiyroll.apps.PaiyrollConfig',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.postgres',
 'django_jinja',
 'bootstrapform_jinja',
 'import_export',
 'phonenumber_field',
 'polymorphic',
 'viewflow']
INTERNAL_IPS: []
LANGUAGES: [('af', 'Afrikaans'),
 ('ar', 'Arabic'),
 ('ast', 'Asturian'),
 ('az', 'Azerbaijani'),
 ('bg', 'Bulgarian'),
 ('be', 'Belarusian'),
 ('bn', 'Bengali'),
 ('br', 'Breton'),
 ('bs', 'Bosnian'),
 ('ca', 'Catalan'),
 ('cs', 'Czech'),
 ('cy', 'Welsh'),
 ('da', 'Danish'),
 ('de', 'German'),
 ('dsb', 'Lower Sorbian'),
 ('el', 'Greek'),
 ('en', 'English'),
 ('en-au', 'Australian English'),
 ('en-gb', 'British English'),
 ('eo', 'Esperanto'),
 ('es', 'Spanish'),
 ('es-ar', 'Argentinian Spanish'),
 ('es-co', 'Colombian Spanish'),
 ('es-mx', 'Mexican Spanish'),
 ('es-ni', 'Nicaraguan Spanish'),
 ('es-ve', 'Venezuelan Spanish'),
 ('et', 'Estonian'),
 ('eu', 'Basque'),
 ('fa', 'Persian'),
 ('fi', 'Finnish'),
 ('fr', 'French'),
 ('fy', 'Frisian'),
 ('ga', 'Irish'),
 ('gd', 'Scottish Gaelic'),
 ('gl', 'Galician'),
 ('he', 'Hebrew'),
 ('hi', 'Hindi'),
 ('hr', 'Croatian'),
 ('hsb', 'Upper Sorbian'),
 ('hu', 'Hungarian'),
 ('hy', 'Armenian'),
 ('ia', 'Interlingua'),
 ('id', 'Indonesian'),
 ('io', 'Ido'),
 ('is', 'Icelandic'),
 ('it', 'Italian'),
 ('ja', 'Japanese'),
 ('ka', 'Georgian'),
 ('kab', 'Kabyle'),
 ('kk', 'Kazakh'),
 ('km', 'Khmer'),
 ('kn', 'Kannada'),
 ('ko', 'Korean'),
 ('lb', 'Luxembourgish'),
 ('lt', 'Lithuanian'),
 ('lv', 'Latvian'),
 ('mk', 'Macedonian'),
 ('ml', 'Malayalam'),
 ('mn', 'Mongolian'),
 ('mr', 'Marathi'),
 ('my', 'Burmese'),
 ('nb', 'Norwegian Bokmål'),
 ('ne', 'Nepali'),
 ('nl', 'Dutch'),
 ('nn', 'Norwegian Nynorsk'),
 ('os', 'Ossetic'),
 ('pa', 'Punjabi'),
 ('pl', 'Polish'),
 ('pt', 'Portuguese'),
 ('pt-br', 'Brazilian Portuguese'),
 ('ro', 'Romanian'),
 ('ru', 'Russian'),
 ('sk', 'Slovak'),
 ('sl', 'Slovenian'),
 ('sq', 'Albanian'),
 ('sr', 'Serbian'),
 ('sr-latn', 'Serbian Latin'),
 ('sv', 'Swedish'),
 ('sw', 'Swahili'),
 ('ta', 'Tamil'),
 ('te', 'Telugu'),
 ('th', 'Thai'),
 ('tr', 'Turkish'),
 ('tt', 'Tatar'),
 ('udm', 'Udmurt'),
 ('uk', 'Ukrainian'),
 ('ur', 'Urdu'),
 ('vi', 'Vietnamese'),
 ('zh-hans', 'Simplified Chinese'),
 ('zh-hant', 'Traditional Chinese')]
LANGUAGES_BIDI: ['he', 'ar', 'fa', 'ur']
LANGUAGE_CODE: 'en-us'
LANGUAGE_COOKIE_AGE: None
LANGUAGE_COOKIE_DOMAIN: None
LANGUAGE_COOKIE_NAME: 'django_language'
LANGUAGE_COOKIE_PATH: '/'
LOCALE_PATHS: []
LOGGING: {
    'disable_existing_loggers': False,
    'formatters': {   'standard': {   'format': '%(asctime)s [%(levelname)s] '
                                                '%(name)s: %(message)s'}},
    'handlers': {   'file': {   'backupCount': 10,
                                'class': 'logging.handlers.RotatingFileHandler',
                                'filename': '/main/srhaque/.local/share/paiyroll/django.log',
                                'formatter': 'standard',
                                'level': 'DEBUG',
                                'maxBytes': 15728640}},
    'loggers': {'': {'handlers': ['file'], 'level': 'INFO', 'propagate': True}},
    'version': 1}
LOGGING_CONFIG: 'logging.config.dictConfig'
LOGGING_DIR: '/main/srhaque/.local/share/paiyroll'
LOGIN_REDIRECT_URL: '/accounts/profile/'
LOGIN_URL: '/accounts/login/'
LOGOUT_REDIRECT_URL: None
MANAGERS: []
MEDIA_ROOT: ''
MEDIA_URL: ''
MESSAGE_STORAGE: 'django.contrib.messages.storage.fallback.FallbackStorage'
MIDDLEWARE: ['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'project.middleware.SessionExpiryMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']
MIGRATION_MODULES: {
 }
MONTH_DAY_FORMAT: 'F j'
NUMBER_GROUPING: 0
PAIYROLL_CLIENT_OUTPUT_DIR: '/tmp'
PAIYROLL_DEBUG_ADMIN_ACCESS_ALL_WORKFLOWS: True
PAIYROLL_DEBUG_NOTIFICATIONS: True
PAIYROLL_DEBUG_PAY_WALK: 0
PAIYROLL_DEBUG_PROMPTS: 259
PAIYROLL_DEBUG_SCHEDULING_DATE: '2018-01-01'
PAIYROLL_DEBUG_SCHEDULING_FAST: 10
PAIYROLL_GB_RTI_ALL_TRANSACT: None
PAIYROLL_GB_RTI_OTHERS_TRANSACT_FROM: '2019-04-06'
PAIYROLL_GB_RTI_TEST_COMPANIES: 
    ('HMRC', 'HMRC RTI Recognition')
PAIYROLL_GB_RTI_TEST_COMPANIES_ACTUAL_T: '2019-03-31'
PAIYROLL_INTERNAL_NETWORK: IPv4Network('192.168.1.0/24')
PAIYROLL_NOTIFICATIONS: {
    'email': {   'backend': 'django.core.mail.backends.filebased.EmailBackend',
                 'file_path': '/tmp/email_messages',
                 'host': 'smtp.gmail.com',
                 'password': '********',
                 'port': 587,
                 'ssl_certfile': '',
                 'ssl_keyfile': '********',
                 'timeout': None,
                 'use_ssl': False,
                 'use_tls': True,
                 'username': 'paiyroll.com@gmail.com'},
    'sms': {   'account_sid': 'ACbabf740b7a3d23010cbe381a8a184fe0',
               'from': '+15005550006',
               'password': '********'}}
PAIYROLL_REPORT_SPEC_DIR: '/main/srhaque/Innovatieltd/source/paiyroll/report/JasperReports'
PAIYROLL_REPORT_SPEC_SPREADSHEET: '/main/srhaque/Innovatieltd/source/paiyroll/report/Spreadsheets'
PAIYROLL_SITE_NAME: 'login.paiyroll.com'
PASSWORD_HASHERS: '********'
PASSWORD_RESET_TIMEOUT_DAYS: '********'
PREPEND_WWW: False
ROOT_URLCONF: 'project.urls'
SECRET_KEY: '********'
SECURE_BROWSER_XSS_FILTER: False
SECURE_CONTENT_TYPE_NOSNIFF: False
SECURE_HSTS_INCLUDE_SUBDOMAINS: False
SECURE_HSTS_PRELOAD: False
SECURE_HSTS_SECONDS: 0
SECURE_PROXY_SSL_HEADER: None
SECURE_REDIRECT_EXEMPT: []
SECURE_SSL_HOST: None
SECURE_SSL_REDIRECT: False
SERVER_EMAIL: 'root@localhost'
SESSION_CACHE_ALIAS: 'default'
SESSION_COOKIE_AGE: 5400
SESSION_COOKIE_DOMAIN: None
SESSION_COOKIE_HTTPONLY: True
SESSION_COOKIE_NAME: 'sessionid'
SESSION_COOKIE_PATH: '/'
SESSION_COOKIE_SAMESITE: 'Lax'
SESSION_COOKIE_SECURE: False
SESSION_ENGINE: 'django.contrib.sessions.backends.db'
SESSION_EXPIRE_AT_BROWSER_CLOSE: False
SESSION_FILE_PATH: None
SESSION_SAVE_EVERY_REQUEST: False
SESSION_SERIALIZER: 'django.contrib.sessions.serializers.JSONSerializer'
SETTINGS_MODULE: 'project.settings'
SHORT_DATETIME_FORMAT: 'm/d/Y P'
SHORT_DATE_FORMAT: 'm/d/Y'
SIGNING_BACKEND: 'django.core.signing.TimestampSigner'
SILENCED_SYSTEM_CHECKS: []
STATICFILES_DIRS: []
STATICFILES_FINDERS: ['django.contrib.staticfiles.finders.FileSystemFinder',
 'django.contrib.staticfiles.finders.AppDirectoriesFinder']
STATICFILES_STORAGE: 'django.contrib.staticfiles.storage.StaticFilesStorage'
STATIC_ROOT: '../staticroot'
STATIC_URL: '/static/'
TEMPLATES: [{'APP_DIRS': True,
  'BACKEND': 'django_jinja.backend.Jinja2',
  'DIRS': ['/main/srhaque/Innovatieltd/source/templates'],
  'OPTIONS': {'app_dirname': 'templates',
              'auto_reload': True,
              'autoescape': True,
              'bytecode_cache': {'backend': 'django_jinja.cache.BytecodeCache',
                                 'enabled': False,
                                 'name': 'default'},
              'constants': {},
              'context_processors': ['django.contrib.auth.context_processors.auth',
                                     'django.contrib.messages.context_processors.messages'],
              'extensions': ['jinja2.ext.do',
                             'jinja2.ext.loopcontrols',
                             'jinja2.ext.with_',
                             'jinja2.ext.i18n',
                             'jinja2.ext.autoescape',
                             'django_jinja.builtins.extensions.CsrfExtension',
                             'django_jinja.builtins.extensions.CacheExtension',
                             'django_jinja.builtins.extensions.TimezoneExtension',
                             'django_jinja.builtins.extensions.UrlsExtension',
                             'django_jinja.builtins.extensions.StaticFilesExtension',
                             'django_jinja.builtins.extensions.DjangoFiltersExtension',
                             'jinja_extensions.DebugExtension'],
              'filters': {},
              'globals': {},
              'match_extension': ('.html',
                                  '.jinja'),
              'match_regex': '^(?!admin/).*',
              'newstyle_gettext': True,
              'tests': {},
              'translation_engine': 'django.utils.translation',
              'undefined': None}},
 {'APP_DIRS': True,
  'BACKEND': 'django.template.backends.django.DjangoTemplates',
  'DIRS': ['/main/srhaque/Innovatieltd/source/templates'],
  'OPTIONS': {'context_processors': ['django.template.context_processors.debug',
                                     'django.template.context_processors.request',
                                     'django.contrib.auth.context_processors.auth',
                                     'django.contrib.messages.context_processors.messages']}}]
TEST_NON_SERIALIZED_APPS: []
TEST_RUNNER: 'django.test.runner.DiscoverRunner'
THOUSAND_SEPARATOR: ','
TIME_FORMAT: 'P'
TIME_INPUT_FORMATS: ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M']
TIME_ZONE: 'UTC'
USE_I18N: True
USE_L10N: True
USE_THOUSAND_SEPARATOR: False
USE_TZ: True
USE_X_FORWARDED_HOST: False
USE_X_FORWARDED_PORT: False
WSGI_APPLICATION: 'project.wsgi.application'
X_FRAME_OPTIONS: 'SAMEORIGIN'
YEAR_MONTH_FORMAT: 'F Y'
is_overridden: &lt;bound method Settings.is_overridden of &lt;Settings "project.settings"&gt;&gt;
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: N/A</li>
<li><strong>Minimal Celery Version</strong>: Unknown</li>
<li><strong>Minimal Kombu Version</strong>: N/A</li>
<li><strong>Minimal Broker Version</strong>: N/A</li>
<li><strong>Minimal Result Backend Version</strong>: Any result backend should show the problem</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A</li>
<li><strong>Minimal Broker Client Version</strong>: N/A</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="
$ pip3 freeze
aiohttp==3.5.4
alabaster==0.7.12
alembic==1.0.10
amqp==2.4.2
appdirs==1.4.3
apt-xapian-index==0.47
asn1crypto==0.24.0
async-timeout==3.0.1
atomicwrites==1.3.0
attrs==19.1.0
Automat==0.7.0
aws-shell==0.2.1
awscli==1.16.196
Babel==2.6.0
backcall==0.1.0
backports.csv==1.0.7
bcrypt==3.1.6
beautifulsoup4==4.7.1
billiard==3.6.0.0
blinker==1.4
boto3==1.9.137
botocore==1.12.186
cached-property==1.5.1
calendra==3.0
celery==4.3.0
certifi==2018.8.24
cffi==1.12.3
chardet==3.0.4
chromedriver==2.24.1
Click==7.0
colorama==0.3.9
command-not-found==0.3
configobj==5.0.6
constantly==15.1.0
coverage==4.5.3
cryptography==2.3
cssselect==1.0.3
cupshelpers==1.0
datadiff==2.0.0
decorator==4.4.0
deepdiff==4.0.6
defer==1.0.6
defusedxml==0.6.0
diff-match-patch==20181111
distro-info===0.21ubuntu2
Django==2.2
django-excel==0.0.10
django-extra-views==0.12.0
django-filter==2.1.0
django-import-export==1.2.0
django-jinja==2.4.1
django-jinja-bootstrap-form==4.2.3
django-model-utils==3.1.2
django-phonenumber-field==2.3.1
django-polymorphic==2.0.3
django-viewflow==1.5.3
django-webtest==1.9.4
dnspython==1.16.0
docutils==0.14
entrypoints==0.3
ephem==3.7.6.0
et-xmlfile==1.0.1
filelock==3.0.12
flake8==3.7.8
Flask==1.0.2
Flask-BabelEx==0.9.3
Flask-Gravatar==0.5.0
Flask-HTMLmin==1.5.0
Flask-Login==0.4.1
Flask-Mail==0.9.1
Flask-Migrate==2.4.0
Flask-Paranoid==0.2.0
Flask-Principal==0.4.0
Flask-Security==3.0.0
Flask-SQLAlchemy==2.3.2
Flask-WTF==0.14.2
gunicorn==19.9.0
html5lib==1.0.1
htmlmin==0.1.12
httplib2==0.11.3
hyperlink==19.0.0
idna==2.6
imagesize==1.1.0
importlib-metadata==0.18
incremental==17.5.0
ipython==7.6.1
ipython-genutils==0.2.0
isodate==0.6.0
itsdangerous==1.1.0
jdcal==1.4.1
jedi==0.13.3
Jinja2==2.10.1
jmespath==0.9.4
jpy==0.10.0.dev1
jsonpickle==1.1
keyring==17.1.1
keyrings.alt==3.1.1
kombu==4.5.0
language-selector==0.1
ldap3==2.4.1
lml==0.0.9
lunardate==0.2.0
lxml==4.3.3
Mako==1.0.9
MarkupSafe==1.1.1
mccabe==0.6.1
more-itertools==7.0.0
multicorn===-VERSION-
multidict==4.5.2
netifaces==0.10.4
networkx==2.3
numpy==1.16.4
odfpy==1.4.0
olefile==0.46
openpyxl==2.5.14
ordered-set==3.1.1
packaging==19.0
paiyroll==0.1
paiyroll-fdw==0.1
pandas==0.24.2
paramiko==2.4.2
parso==0.4.0
passlib==1.7.1
pexpect==4.6.0
pgadmin4==4.5
phonenumbers==8.10.10
pickleshare==0.7.5
pika==1.0.1
Pillow==5.4.1
pluggy==0.12.0
prompt-toolkit==1.0.16
psutil==5.5.1
psycopg2==2.8.3
psycopg2-binary==2.8.2
py==1.8.0
pyasn1==0.4.2
pycairo==1.16.2
pyCalverter==1.6.1
pycodestyle==2.5.0
pycparser==2.19
pycrypto==2.6.1
pycups==1.9.73
pyecharts-jupyter-installer==0.0.3
pyexcel==0.5.13
pyexcel-handsontable==0.0.2
pyexcel-io==0.5.11
pyexcel-ods==0.5.6
pyexcel-webio==0.1.4
pyexcel-xls==0.5.8
pyexcel-xlsx==0.5.7
pyflakes==2.1.1
Pygments==2.3.1
PyGObject==3.32.0
PyHamcrest==1.9.0
PyJWT==1.7.1
pyluach==1.0.1
pymacaroons==0.13.0
PyNaCl==1.3.0
pyparsing==2.4.0
PyQt5==5.12.1
pyquery==1.4.0
pyrabbit==1.1.0
PySocks==1.6.8
pytest==4.4.1
pytest-cov==2.7.1
pytest-flake8==1.0.4
python-apt==1.8.4
python-consul==1.1.0.dev0
python-dateutil==2.8.0
python-debian==0.1.34
python-editor==1.0.4
pytz==2018.9
PyYAML==3.13
reportlab==3.5.18
requests==2.22.0
requests-toolbelt==0.9.1
requests-unixsocket==0.1.5
rsa==3.4.2
s3transfer==0.2.0
SecretStorage==2.3.1
selenium==3.141.0
simplejson==3.16.0
six==1.12.0
snowballstemmer==1.9.0
soupsieve==1.8
speaklater==1.3
Sphinx==2.1.2
sphinxcontrib-applehelp==1.0.1
sphinxcontrib-devhelp==1.0.1
sphinxcontrib-htmlhelp==1.0.2
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.2
sphinxcontrib-serializinghtml==1.1.3
SQLAlchemy==1.3.3
sqlparse==0.2.4
ssh-import-id==5.7
sshtunnel==0.1.4
systemd-python==234
tablib==0.13.0
tblib==1.3.2
terminaltables==3.1.0
texttable==1.6.1
toml==0.10.0
tornado==6.0.3
tox==3.13.2
traitlets==4.3.2
twilio==6.26.2
Twisted==19.2.1
ubuntu-advantage-tools==19.2
ubuntu-drivers-common==0.0.0
ufw==0.36
uk-postcode-utils==1.0
unattended-upgrades==0.1
urllib3==1.25.3
vine==1.3.0
virtualenv==16.6.1
waitress==1.3.0
wcwidth==0.1.7
webencodings==0.5.1
WebOb==1.8.5
WebTest==2.0.33
Werkzeug==0.15.2
WTForms==2.2.1
xdg==4.0.0
xkit==0.0.0
xlrd==1.2.0
xlwt==1.3.0
xmltodict==0.12.0
yarl==1.3.0
zeep==3.4.0
zipp==0.5.2
zope.interface==4.6.0"><pre lang="," class="notranslate"><code class="notranslate">
$ pip3 freeze
aiohttp==3.5.4
alabaster==0.7.12
alembic==1.0.10
amqp==2.4.2
appdirs==1.4.3
apt-xapian-index==0.47
asn1crypto==0.24.0
async-timeout==3.0.1
atomicwrites==1.3.0
attrs==19.1.0
Automat==0.7.0
aws-shell==0.2.1
awscli==1.16.196
Babel==2.6.0
backcall==0.1.0
backports.csv==1.0.7
bcrypt==3.1.6
beautifulsoup4==4.7.1
billiard==3.6.0.0
blinker==1.4
boto3==1.9.137
botocore==1.12.186
cached-property==1.5.1
calendra==3.0
celery==4.3.0
certifi==2018.8.24
cffi==1.12.3
chardet==3.0.4
chromedriver==2.24.1
Click==7.0
colorama==0.3.9
command-not-found==0.3
configobj==5.0.6
constantly==15.1.0
coverage==4.5.3
cryptography==2.3
cssselect==1.0.3
cupshelpers==1.0
datadiff==2.0.0
decorator==4.4.0
deepdiff==4.0.6
defer==1.0.6
defusedxml==0.6.0
diff-match-patch==20181111
distro-info===0.21ubuntu2
Django==2.2
django-excel==0.0.10
django-extra-views==0.12.0
django-filter==2.1.0
django-import-export==1.2.0
django-jinja==2.4.1
django-jinja-bootstrap-form==4.2.3
django-model-utils==3.1.2
django-phonenumber-field==2.3.1
django-polymorphic==2.0.3
django-viewflow==1.5.3
django-webtest==1.9.4
dnspython==1.16.0
docutils==0.14
entrypoints==0.3
ephem==3.7.6.0
et-xmlfile==1.0.1
filelock==3.0.12
flake8==3.7.8
Flask==1.0.2
Flask-BabelEx==0.9.3
Flask-Gravatar==0.5.0
Flask-HTMLmin==1.5.0
Flask-Login==0.4.1
Flask-Mail==0.9.1
Flask-Migrate==2.4.0
Flask-Paranoid==0.2.0
Flask-Principal==0.4.0
Flask-Security==3.0.0
Flask-SQLAlchemy==2.3.2
Flask-WTF==0.14.2
gunicorn==19.9.0
html5lib==1.0.1
htmlmin==0.1.12
httplib2==0.11.3
hyperlink==19.0.0
idna==2.6
imagesize==1.1.0
importlib-metadata==0.18
incremental==17.5.0
ipython==7.6.1
ipython-genutils==0.2.0
isodate==0.6.0
itsdangerous==1.1.0
jdcal==1.4.1
jedi==0.13.3
Jinja2==2.10.1
jmespath==0.9.4
jpy==0.10.0.dev1
jsonpickle==1.1
keyring==17.1.1
keyrings.alt==3.1.1
kombu==4.5.0
language-selector==0.1
ldap3==2.4.1
lml==0.0.9
lunardate==0.2.0
lxml==4.3.3
Mako==1.0.9
MarkupSafe==1.1.1
mccabe==0.6.1
more-itertools==7.0.0
multicorn===-VERSION-
multidict==4.5.2
netifaces==0.10.4
networkx==2.3
numpy==1.16.4
odfpy==1.4.0
olefile==0.46
openpyxl==2.5.14
ordered-set==3.1.1
packaging==19.0
paiyroll==0.1
paiyroll-fdw==0.1
pandas==0.24.2
paramiko==2.4.2
parso==0.4.0
passlib==1.7.1
pexpect==4.6.0
pgadmin4==4.5
phonenumbers==8.10.10
pickleshare==0.7.5
pika==1.0.1
Pillow==5.4.1
pluggy==0.12.0
prompt-toolkit==1.0.16
psutil==5.5.1
psycopg2==2.8.3
psycopg2-binary==2.8.2
py==1.8.0
pyasn1==0.4.2
pycairo==1.16.2
pyCalverter==1.6.1
pycodestyle==2.5.0
pycparser==2.19
pycrypto==2.6.1
pycups==1.9.73
pyecharts-jupyter-installer==0.0.3
pyexcel==0.5.13
pyexcel-handsontable==0.0.2
pyexcel-io==0.5.11
pyexcel-ods==0.5.6
pyexcel-webio==0.1.4
pyexcel-xls==0.5.8
pyexcel-xlsx==0.5.7
pyflakes==2.1.1
Pygments==2.3.1
PyGObject==3.32.0
PyHamcrest==1.9.0
PyJWT==1.7.1
pyluach==1.0.1
pymacaroons==0.13.0
PyNaCl==1.3.0
pyparsing==2.4.0
PyQt5==5.12.1
pyquery==1.4.0
pyrabbit==1.1.0
PySocks==1.6.8
pytest==4.4.1
pytest-cov==2.7.1
pytest-flake8==1.0.4
python-apt==1.8.4
python-consul==1.1.0.dev0
python-dateutil==2.8.0
python-debian==0.1.34
python-editor==1.0.4
pytz==2018.9
PyYAML==3.13
reportlab==3.5.18
requests==2.22.0
requests-toolbelt==0.9.1
requests-unixsocket==0.1.5
rsa==3.4.2
s3transfer==0.2.0
SecretStorage==2.3.1
selenium==3.141.0
simplejson==3.16.0
six==1.12.0
snowballstemmer==1.9.0
soupsieve==1.8
speaklater==1.3
Sphinx==2.1.2
sphinxcontrib-applehelp==1.0.1
sphinxcontrib-devhelp==1.0.1
sphinxcontrib-htmlhelp==1.0.2
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.2
sphinxcontrib-serializinghtml==1.1.3
SQLAlchemy==1.3.3
sqlparse==0.2.4
ssh-import-id==5.7
sshtunnel==0.1.4
systemd-python==234
tablib==0.13.0
tblib==1.3.2
terminaltables==3.1.0
texttable==1.6.1
toml==0.10.0
tornado==6.0.3
tox==3.13.2
traitlets==4.3.2
twilio==6.26.2
Twisted==19.2.1
ubuntu-advantage-tools==19.2
ubuntu-drivers-common==0.0.0
ufw==0.36
uk-postcode-utils==1.0
unattended-upgrades==0.1
urllib3==1.25.3
vine==1.3.0
virtualenv==16.6.1
waitress==1.3.0
wcwidth==0.1.7
webencodings==0.5.1
WebOb==1.8.5
WebTest==2.0.33
Werkzeug==0.15.2
WTForms==2.2.1
xdg==4.0.0
xkit==0.0.0
xlrd==1.2.0
xlwt==1.3.0
xmltodict==0.12.0
yarl==1.3.0
zeep==3.4.0
zipp==0.5.2
zope.interface==4.6.0
</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="        #
        # Any exception thrown in the set() method of a result backend should show the issue.
        #
        raise KeyError(&quot;simulated failure&quot;)"><pre class="notranslate">        <span class="pl-c">#</span>
        <span class="pl-c"># Any exception thrown in the set() method of a result backend should show the issue.</span>
        <span class="pl-c">#</span>
        <span class="pl-k">raise</span> <span class="pl-v">KeyError</span>(<span class="pl-s">"simulated failure"</span>)</pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<p dir="auto">A failure in any result backend (for example, as described <a href="https://github.com/celery/celery/issues/5605#issuecomment-510819179" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/5605/hovercard">in #5605</a>), should be reported by the worker and then Celery ought to be able to continue to service further requests. The recovery mechanism is of course free to kill and replace the failing worker as needed.</p>
<h1 dir="auto">Actual Behavior</h1>

<p dir="auto">In my case, I am using the Consul backend, though I believe this issue affects all backends. The Consul backend uses the python-consul package which can throw an exception while performing network operations. In Celery, this is reported like this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2019-07-12 08:48:51,191: WARNING/ForkPoolWorker-80] /usr/local/lib/python3.7/dist-packages/celery/app/trace.py:568: RuntimeWarning: Exception raised outside body: TypeError(&quot;'bool' object is not subscriptable&quot;):
Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.7/dist-packages/celery/app/trace.py&quot;, line 449, in trace_task
    uuid, retval, task_request, publish_result,
  File &quot;/usr/local/lib/python3.7/dist-packages/celery/backends/base.py&quot;, line 149, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File &quot;/usr/local/lib/python3.7/dist-packages/celery/backends/base.py&quot;, line 342, in store_result
    request=request, **kwargs)
  File &quot;/usr/local/lib/python3.7/dist-packages/celery/backends/base.py&quot;, line 714, in _store_result
    self.set(self.get_key_for_task(task_id), self.encode(meta))
  File &quot;/usr/local/lib/python3.7/dist-packages/celery/backends/consul.py&quot;, line 92, in set
    ttl=self.expires)
  File &quot;/usr/local/lib/python3.7/dist-packages/consul/base.py&quot;, line 1781, in create
    data=data)
  File &quot;/usr/local/lib/python3.7/dist-packages/consul/std.py&quot;, line 33, in put
    self.session.request('PUT', uri, body=data, headers=JSON_HEADER)))
  File &quot;/usr/local/lib/python3.7/dist-packages/consul/base.py&quot;, line 234, in cb
    data = data['ID']
TypeError: 'bool' object is not subscriptable

  exc, exc_info.traceback)))"><pre class="notranslate"><code class="notranslate">[2019-07-12 08:48:51,191: WARNING/ForkPoolWorker-80] /usr/local/lib/python3.7/dist-packages/celery/app/trace.py:568: RuntimeWarning: Exception raised outside body: TypeError("'bool' object is not subscriptable"):
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 449, in trace_task
    uuid, retval, task_request, publish_result,
  File "/usr/local/lib/python3.7/dist-packages/celery/backends/base.py", line 149, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File "/usr/local/lib/python3.7/dist-packages/celery/backends/base.py", line 342, in store_result
    request=request, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/celery/backends/base.py", line 714, in _store_result
    self.set(self.get_key_for_task(task_id), self.encode(meta))
  File "/usr/local/lib/python3.7/dist-packages/celery/backends/consul.py", line 92, in set
    ttl=self.expires)
  File "/usr/local/lib/python3.7/dist-packages/consul/base.py", line 1781, in create
    data=data)
  File "/usr/local/lib/python3.7/dist-packages/consul/std.py", line 33, in put
    self.session.request('PUT', uri, body=data, headers=JSON_HEADER)))
  File "/usr/local/lib/python3.7/dist-packages/consul/base.py", line 234, in cb
    data = data['ID']
TypeError: 'bool' object is not subscriptable

  exc, exc_info.traceback)))
</code></pre></div>
<p dir="auto">This is fine. However, it seems that the affected worker then get "stuck" in that it is permanently visible in the output of "celery inspect active":</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ celery inspect active
-&gt; celery@freenas: OK
    * {'id': '2996120d-87d2-4c9b-8774-b7a3b64e1332', 'name': 'paiyroll.tasks.function_run', 'args': &quot;['paiyroll.report_run', 'report_preflight_validate_async', 1921]&quot;, 'kwargs': '{}', 'type': 'paiyroll.tasks.function_run', 'hostname': 'celery@freenas', 'time_start': 1562921376.9528346, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}, 'worker_pid': 7111}
.
.
."><pre class="notranslate"><code class="notranslate">$ celery inspect active
-&gt; celery@freenas: OK
    * {'id': '2996120d-87d2-4c9b-8774-b7a3b64e1332', 'name': 'paiyroll.tasks.function_run', 'args': "['paiyroll.report_run', 'report_preflight_validate_async', 1921]", 'kwargs': '{}', 'type': 'paiyroll.tasks.function_run', 'hostname': 'celery@freenas', 'time_start': 1562921376.9528346, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}, 'worker_pid': 7111}
.
.
.
</code></pre></div>
<p dir="auto">This happens to each worker as it hits the error, until Celery is unable to progress any work. On MacOS, though <em>usually</em> not on Linux, the workers are not then stopped using "celery control shutdown" but must be killed by hand.</p>