<h1 dir="auto">Checklist</h1>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>
<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>
<p dir="auto"><strong>Celery version</strong>: 5.0.5 (singularity)</p>
<details>
<summary><b><code class="notranslate">celery report</code> Outputsoftware -&gt; celery:5.0.5 (singularity) kombu:5.0.2 py:3.7.7
            billiard:3.6.3.0 py-amqp:5.0.2
platform -&gt; system:Darwin arch:64bit
            kernel version:20.3.0 imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:******
</b><p dir="auto"><b>ABSOLUTE_URL_OVERRIDES: {<br>
}<br>
ADMINS: []<br>
ALLOWED_HOSTS: ['<em>']<br>
APPEND_SLASH: False<br>
AUTHENTICATION_BACKENDS: {'auth.auth0backend.Auth0',<br>
'django.contrib.auth.backends.ModelBackend'}<br>
AUTH_PASSWORD_VALIDATORS: '********'<br>
AUTH_USER_MODEL: 'users.CustomUser'<br>
BASE_DIR: '</em><strong>'<br>
BROKER_ADDRESS: None<br>
BROKER_TYPE: None<br>
CACHES: {<br>
'default': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'}}<br>
CACHE_MIDDLEWARE_ALIAS: 'default'<br>
CACHE_MIDDLEWARE_KEY_PREFIX: ''<br>
CACHE_MIDDLEWARE_SECONDS: 600<br>
CACHE_TIMEOUT: 345600<br>
CELERY_ACCEPT_CONTENT: ['pickle', 'json']<br>
CELERY_BROKER_TRANSPORT_OPTIONS: {<br>
'visibility_timeout': 172800}<br>
CELERY_IMPORTS: ['.orchestrator.tasks',<br>
'.positioning_pipeline.pipeline_tasks',<br>
'</strong><strong>.positioning_pipeline.tasks_consumers',<br>
'</strong><strong>.services_settings.celery_commands']<br>
CELERY_RESULT_BACKEND: '</strong><strong>'<br>
CELERY_RESULT_CHORD_RETRY_INTERVAL: 0.5<br>
CELERY_RESULT_EXTENDED: True<br>
CELERY_RESULT_SERIALIZER: 'json'<br>
CELERY_TASK_QUEUES: [&lt;unbound Queue celery -&gt; &lt;unbound Exchange ''(direct)&gt; -&gt; &gt;,<br>
&lt;unbound Queue renderer_tasks -&gt; &lt;unbound Exchange ''(direct)&gt; -&gt; &gt;,<br>
&lt;unbound Queue small_tasks -&gt; &lt;unbound Exchange ''(direct)&gt; -&gt; &gt;,<br>
&lt;unbound Queue medium_tasks -&gt; &lt;unbound Exchange ''(direct)&gt; -&gt; &gt;,<br>
&lt;unbound Queue big_tasks -&gt; &lt;unbound Exchange ''(direct)&gt; -&gt; &gt;,<br>
&lt;unbound Queue users_tasks -&gt; &lt;unbound Exchange ''(direct)&gt; -&gt; &gt;]<br>
CELERY_TASK_ROUTES: [&lt;bound method Router.route_task of &lt;class '</strong><em><em>.services_settings.celery_routing.Router'&gt;&gt;]<br>
CELERY_TASK_SERIALIZER: 'pickle'<br>
CELERY_TASK_TRACK_STARTED: True<br>
CELERY_WORKER_CONCURRENCY: 1<br>
CELERY_WORKER_LOG_FORMAT: '%(message)s'<br>
CELERY_WORKER_MAX_MEMORY_PER_CHILD: 3145728<br>
CELERY_WORKER_PREFETCH_MULTIPLIER: 1<br>
CORS_ALLOW_CREDENTIALS: True<br>
CORS_ORIGIN_ALLOW_ALL: True<br>
CORS_URLS_REGEX: '/api/.</em>/.</em>-image/.<em>'<br>
CSRF_COOKIE_AGE: 31449600<br>
CSRF_COOKIE_DOMAIN: None<br>
CSRF_COOKIE_HTTPONLY: False<br>
CSRF_COOKIE_NAME: 'csrftoken'<br>
CSRF_COOKIE_PATH: '/'<br>
CSRF_COOKIE_SAMESITE: 'Lax'<br>
CSRF_COOKIE_SECURE: False<br>
CSRF_FAILURE_VIEW: 'django.views.csrf.csrf_failure'<br>
CSRF_HEADER_NAME: 'HTTP_X_CSRFTOKEN'<br>
CSRF_TRUSTED_ORIGINS: []<br>
CSRF_USE_SESSIONS: False<br>
DATABASES: {<br>
'default': {   'ATOMIC_REQUESTS': False,<br>
'AUTOCOMMIT': True,<br>
'CONN_MAX_AGE': 0,<br>
'ENGINE': 'django.db.backends.sqlite3',<br>
'HOST': '',<br>
'NAME': '</em><em><strong>',<br>
'OPTIONS': {},<br>
'PASSWORD': '</strong></em><em><strong>',<br>
'PORT': '',<br>
'TEST': {   'CHARSET': None,<br>
'COLLATION': None,<br>
'MIGRATE': True,<br>
'MIRROR': None,<br>
'NAME': None},<br>
'TIME_ZONE': None,<br>
'USER': ''}}<br>
DATABASE_INFO_TO_USE: {<br>
'ATOMIC_REQUESTS': False,<br>
'AUTOCOMMIT': True,<br>
'CONN_MAX_AGE': 0,<br>
'ENGINE': 'django.db.backends.sqlite3',<br>
'HOST': '',<br>
'NAME': '</strong></em><em><strong>',<br>
'OPTIONS': {},<br>
'PASSWORD': '</strong></em><em><strong>',<br>
'PORT': '',<br>
'TEST': {   'CHARSET': None,<br>
'COLLATION': None,<br>
'MIGRATE': True,<br>
'MIRROR': None,<br>
'NAME': None},<br>
'TIME_ZONE': None,<br>
'USER': ''}<br>
DATABASE_ROUTERS: '</strong></em><em><strong>'<br>
DATA_UPLOAD_MAX_MEMORY_SIZE: 2621440<br>
DATA_UPLOAD_MAX_NUMBER_FIELDS: 1000<br>
DATETIME_FORMAT: 'N j, Y, P'<br>
DATETIME_INPUT_FORMATS: ['%Y-%m-%d %H:%M:%S',<br>
'%Y-%m-%d %H:%M:%S.%f',<br>
'%Y-%m-%d %H:%M',<br>
'%m/%d/%Y %H:%M:%S',<br>
'%m/%d/%Y %H:%M:%S.%f',<br>
'%m/%d/%Y %H:%M',<br>
'%m/%d/%y %H:%M:%S',<br>
'%m/%d/%y %H:%M:%S.%f',<br>
'%m/%d/%y %H:%M']<br>
DATE_FORMAT: 'N j, Y'<br>
DATE_INPUT_FORMATS: ['%Y-%m-%d',<br>
'%m/%d/%Y',<br>
'%m/%d/%y',<br>
'%b %d %Y',<br>
'%b %d, %Y',<br>
'%d %b %Y',<br>
'%d %b, %Y',<br>
'%B %d %Y',<br>
'%B %d, %Y',<br>
'%d %B %Y',<br>
'%d %B, %Y']<br>
DEBUG: True<br>
DEBUG_PROPAGATE_EXCEPTIONS: False<br>
DEBUG_TOOLBAR_CONFIG: {<br>
'SHOW_TOOLBAR_CALLBACK': '</strong></em><em><strong>.settings.show_toolbar'}<br>
DECIMAL_SEPARATOR: '.'<br>
DEFAULT_CHARSET: 'utf-8'<br>
DEFAULT_EXCEPTION_REPORTER: 'django.views.debug.ExceptionReporter'<br>
DEFAULT_EXCEPTION_REPORTER_FILTER: 'django.views.debug.SafeExceptionReporterFilter'<br>
DEFAULT_FILE_STORAGE: 'django.core.files.storage.FileSystemStorage'<br>
DEFAULT_FROM_EMAIL: 'webmaster@localhost'<br>
DEFAULT_HASHING_ALGORITHM: 'sha256'<br>
DEFAULT_INDEX_TABLESPACE: ''<br>
DEFAULT_TABLESPACE: ''<br>
DISALLOWED_USER_AGENTS: []<br>
EMAIL_BACKEND: 'django.core.mail.backends.smtp.EmailBackend'<br>
EMAIL_HOST: 'localhost'<br>
EMAIL_HOST_PASSWORD: '</strong></em><em><strong>'<br>
EMAIL_HOST_USER: ''<br>
EMAIL_PORT: 25<br>
EMAIL_SSL_CERTFILE: None<br>
EMAIL_SSL_KEYFILE: '</strong></em><em><strong>'<br>
EMAIL_SUBJECT_PREFIX: '[Django] '<br>
EMAIL_TIMEOUT: None<br>
EMAIL_USE_LOCALTIME: False<br>
EMAIL_USE_SSL: False<br>
EMAIL_USE_TLS: False<br>
FILE_UPLOAD_DIRECTORY_PERMISSIONS: None<br>
FILE_UPLOAD_HANDLERS: ['django.core.files.uploadhandler.MemoryFileUploadHandler',<br>
'django.core.files.uploadhandler.TemporaryFileUploadHandler']<br>
FILE_UPLOAD_MAX_MEMORY_SIZE: 2621440<br>
FILE_UPLOAD_PERMISSIONS: 420<br>
FILE_UPLOAD_TEMP_DIR: None<br>
FIRST_DAY_OF_WEEK: 0<br>
FIXTURE_DIRS: []<br>
FORCE_SCRIPT_NAME: None<br>
FORMAT_MODULE_PATH: None<br>
FORM_RENDERER: 'django.forms.renderers.DjangoTemplates'<br>
IGNORABLE_404_URLS: []<br>
INSTALLED_APPS: ['social_django',<br>
'</strong></em><em><strong>',<br>
'</strong></em><em><strong>',<br>
'django.contrib.auth',<br>
'django.contrib.contenttypes',<br>
'django.contrib.sessions',<br>
'django.contrib.messages',<br>
'django.contrib.staticfiles',<br>
'corsheaders',<br>
'rest_framework',<br>
'debug_toolbar',<br>
'</strong></em><em><strong>.django_celery_results']<br>
INTERNAL_IPS: ['127.0.0.1']<br>
IS_PRODUCTION_OR_DEMO: False<br>
IS_REAL_PRODUCTION: False<br>
JWT_AUTH: {<br>
'JWT_ALGORITHM': 'RS256',<br>
'JWT_AUDIENCE': '</strong></em><em><strong>',<br>
'JWT_AUTH_HEADER_PREFIX': 'Bearer',<br>
'JWT_ISSUER': '</strong></em><em><strong>',<br>
'JWT_PAYLOAD_GET_USERNAME_HANDLER': 'auth.auth0user.jwt_get_username_from_payload_handler',<br>
'JWT_PUBLIC_KEY': '</strong></em><em><strong>'}<br>
LANGUAGES: [('af', 'Afrikaans'),<br>
('ar', 'Arabic'),<br>
('ar-dz', 'Algerian Arabic'),<br>
('ast', 'Asturian'),<br>
('az', 'Azerbaijani'),<br>
('bg', 'Bulgarian'),<br>
('be', 'Belarusian'),<br>
('bn', 'Bengali'),<br>
('br', 'Breton'),<br>
('bs', 'Bosnian'),<br>
('ca', 'Catalan'),<br>
('cs', 'Czech'),<br>
('cy', 'Welsh'),<br>
('da', 'Danish'),<br>
('de', 'German'),<br>
('dsb', 'Lower Sorbian'),<br>
('el', 'Greek'),<br>
('en', 'English'),<br>
('en-au', 'Australian English'),<br>
('en-gb', 'British English'),<br>
('eo', 'Esperanto'),<br>
('es', 'Spanish'),<br>
('es-ar', 'Argentinian Spanish'),<br>
('es-co', 'Colombian Spanish'),<br>
('es-mx', 'Mexican Spanish'),<br>
('es-ni', 'Nicaraguan Spanish'),<br>
('es-ve', 'Venezuelan Spanish'),<br>
('et', 'Estonian'),<br>
('eu', 'Basque'),<br>
('fa', 'Persian'),<br>
('fi', 'Finnish'),<br>
('fr', 'French'),<br>
('fy', 'Frisian'),<br>
('ga', 'Irish'),<br>
('gd', 'Scottish Gaelic'),<br>
('gl', 'Galician'),<br>
('he', 'Hebrew'),<br>
('hi', 'Hindi'),<br>
('hr', 'Croatian'),<br>
('hsb', 'Upper Sorbian'),<br>
('hu', 'Hungarian'),<br>
('hy', 'Armenian'),<br>
('ia', 'Interlingua'),<br>
('id', 'Indonesian'),<br>
('ig', 'Igbo'),<br>
('io', 'Ido'),<br>
('is', 'Icelandic'),<br>
('it', 'Italian'),<br>
('ja', 'Japanese'),<br>
('ka', 'Georgian'),<br>
('kab', 'Kabyle'),<br>
('kk', 'Kazakh'),<br>
('km', 'Khmer'),<br>
('kn', 'Kannada'),<br>
('ko', 'Korean'),<br>
('ky', 'Kyrgyz'),<br>
('lb', 'Luxembourgish'),<br>
('lt', 'Lithuanian'),<br>
('lv', 'Latvian'),<br>
('mk', 'Macedonian'),<br>
('ml', 'Malayalam'),<br>
('mn', 'Mongolian'),<br>
('mr', 'Marathi'),<br>
('my', 'Burmese'),<br>
('nb', 'Norwegian Bokmål'),<br>
('ne', 'Nepali'),<br>
('nl', 'Dutch'),<br>
('nn', 'Norwegian Nynorsk'),<br>
('os', 'Ossetic'),<br>
('pa', 'Punjabi'),<br>
('pl', 'Polish'),<br>
('pt', 'Portuguese'),<br>
('pt-br', 'Brazilian Portuguese'),<br>
('ro', 'Romanian'),<br>
('ru', 'Russian'),<br>
('sk', 'Slovak'),<br>
('sl', 'Slovenian'),<br>
('sq', 'Albanian'),<br>
('sr', 'Serbian'),<br>
('sr-latn', 'Serbian Latin'),<br>
('sv', 'Swedish'),<br>
('sw', 'Swahili'),<br>
('ta', 'Tamil'),<br>
('te', 'Telugu'),<br>
('tg', 'Tajik'),<br>
('th', 'Thai'),<br>
('tk', 'Turkmen'),<br>
('tr', 'Turkish'),<br>
('tt', 'Tatar'),<br>
('udm', 'Udmurt'),<br>
('uk', 'Ukrainian'),<br>
('ur', 'Urdu'),<br>
('uz', 'Uzbek'),<br>
('vi', 'Vietnamese'),<br>
('zh-hans', 'Simplified Chinese'),<br>
('zh-hant', 'Traditional Chinese')]<br>
LANGUAGES_BIDI: ['he', 'ar', 'ar-dz', 'fa', 'ur']<br>
LANGUAGE_CODE: 'en-us'<br>
LANGUAGE_COOKIE_AGE: None<br>
LANGUAGE_COOKIE_DOMAIN: None<br>
LANGUAGE_COOKIE_HTTPONLY: False<br>
LANGUAGE_COOKIE_NAME: 'django_language'<br>
LANGUAGE_COOKIE_PATH: '/'<br>
LANGUAGE_COOKIE_SAMESITE: None<br>
LANGUAGE_COOKIE_SECURE: False<br>
LOCALE_PATHS: []<br>
LOGGING: {<br>
}<br>
LOGGING_CONFIG: 'logging.config.dictConfig'<br>
LOGIN_REDIRECT_URL: '/'<br>
LOGIN_URL: '/login/auth0'<br>
LOGOUT_REDIRECT_URL: '</strong></em><em>'<br>
MANAGERS: []<br>
MAX_AGE_FOR_DB_CONNECTION_SECONDS: 300<br>
MAX_CACHE_ENTRIES: 1000<br>
MEDIA_ROOT: ''<br>
MEDIA_URL: '/'<br>
MESSAGE_STORAGE: 'django.contrib.messages.storage.fallback.FallbackStorage'<br>
MIDDLEWARE: ['data.middleware.</em><em><strong>',<br>
'data.routers.RouterMiddleware',<br>
'data.middleware.log_performance_middleware',<br>
'corsheaders.middleware.CorsMiddleware',<br>
'django.contrib.sessions.middleware.SessionMiddleware',<br>
'django.middleware.csrf.CsrfViewMiddleware',<br>
'django.contrib.auth.middleware.AuthenticationMiddleware',<br>
'data.middleware.bind_site_username_middleware',<br>
'django.contrib.messages.middleware.MessageMiddleware',<br>
'django.middleware.clickjacking.XFrameOptionsMiddleware',<br>
'debug_toolbar.middleware.DebugToolbarMiddleware',<br>
'data.middleware.</strong></em><em>']<br>
MIGRATION_MODULES: {<br>
}<br>
MONTH_DAY_FORMAT: 'F j'<br>
NUMBER_GROUPING: 0<br>
OPTIONAL_CACHES: {<br>
'dummycache': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'},<br>
'loccache': {   'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',<br>
'OPTIONS': {'MAX_ENTRIES': 1000},<br>
'TIMEOUT': 345600},<br>
'memcached': {   'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',<br>
'LOCATION': None,<br>
'OPTIONS': {'server_max_value_length': 33554432},<br>
'TIMEOUT': 345600}}<br>
OPTIONAL_DATABASES: {<br>
'sqlite': {   'ATOMIC_REQUESTS': False,<br>
'AUTOCOMMIT': True,<br>
'CONN_MAX_AGE': 0,<br>
'ENGINE': 'django.db.backends.sqlite3',<br>
'HOST': '',<br>
'NAME': '</em><em><strong>',<br>
'OPTIONS': {},<br>
'PASSWORD': '</strong></em><em><strong>',<br>
'PORT': '',<br>
'TEST': {   'CHARSET': None,<br>
'COLLATION': None,<br>
'MIGRATE': True,<br>
'MIRROR': None,<br>
'NAME': None},<br>
'TIME_ZONE': None,<br>
'USER': ''},<br>
'sqlite-tests': {   'ENGINE': 'django.db.backends.sqlite3',<br>
'NAME': ':memory:'}}<br>
PASSWORD_HASHERS: '</strong></em><em><strong>'<br>
PASSWORD_RESET_TIMEOUT: '</strong></em><em><strong>'<br>
PASSWORD_RESET_TIMEOUT_DAYS: '</strong></em><em><strong>'<br>
PREPEND_WWW: False<br>
PROJECT_DIR: '</strong></em><em>'<br>
REST_FRAMEWORK: {<br>
'DEFAULT_AUTHENTICATION_CLASSES': (   'rest_framework_jwt.authentication.JSONWebTokenAuthentication',<br>
'rest_framework.authentication.SessionAuthentication',<br>
'rest_framework.authentication.BasicAuthentication'),<br>
'DEFAULT_PERMISSION_CLASSES': (   'data.</em><em><strong>.utils.views_utils.ProjectViewerEndpoint',),<br>
'DEFAULT_RENDERER_CLASSES': [   'rest_framework.renderers.JSONRenderer',<br>
'rest_framework.renderers.BrowsableAPIRenderer'],<br>
'DEFAULT_VERSIONING_CLASS': 'rest_framework.versioning.URLPathVersioning'}<br>
ROOT_URLCONF: '</strong></em><em>.urls'<br>
SECRET_KEY: '</em><em><strong>'<br>
SECURE_BROWSER_XSS_FILTER: True<br>
SECURE_CONTENT_TYPE_NOSNIFF: False<br>
SECURE_HSTS_INCLUDE_SUBDOMAINS: False<br>
SECURE_HSTS_PRELOAD: False<br>
SECURE_HSTS_SECONDS: 0<br>
SECURE_PROXY_SSL_HEADER: None<br>
SECURE_REDIRECT_EXEMPT: []<br>
SECURE_REFERRER_POLICY: 'same-origin'<br>
SECURE_SSL_HOST: None<br>
SECURE_SSL_REDIRECT: False<br>
SELF_ADDRESS: '<a href="http://localhost:7000" rel="nofollow">http://localhost:7000</a>'<br>
SERVER_EMAIL: 'root@localhost'<br>
SESSION_CACHE_ALIAS: 'default'<br>
SESSION_COOKIE_AGE: 1209600<br>
SESSION_COOKIE_DOMAIN: None<br>
SESSION_COOKIE_HTTPONLY: True<br>
SESSION_COOKIE_NAME: '</strong>'<br>
SESSION_COOKIE_PATH: '/'<br>
SESSION_COOKIE_SAMESITE: 'Lax'<br>
SESSION_COOKIE_SECURE: False<br>
SESSION_ENGINE: 'django.contrib.sessions.backends.db'<br>
SESSION_EXPIRE_AT_BROWSER_CLOSE: False<br>
SESSION_FILE_PATH: None<br>
SESSION_SAVE_EVERY_REQUEST: False<br>
SESSION_SERIALIZER: 'django.contrib.sessions.serializers.JSONSerializer'<br>
SETTINGS_MODULE: '</em><em><strong>.settings'<br>
SHORT_DATETIME_FORMAT: 'm/d/Y P'<br>
SHORT_DATE_FORMAT: 'm/d/Y'<br>
SHOULD_PROXY_AUTH_TABLES: False<br>
SIGNING_BACKEND: 'django.core.signing.TimestampSigner'<br>
SILENCED_SYSTEM_CHECKS: []<br>
SOCIAL_AUTH_AUTH0_AUDIENCE: '</strong></em><em>'<br>
SOCIAL_AUTH_AUTH0_DOMAIN: '</em><em><strong>'<br>
SOCIAL_AUTH_AUTH0_JWKS: ('{"keys": [{"alg": "RS256", "kty": "RSA", '<br>
'"use": "sig", "n": '<br>
'"</strong></em><em>", '<br>
'"e": "AQAB", "kid": '<br>
'"</em><em><strong>", '<br>
'"x5t": '<br>
'"</strong></em><em>", '<br>
'"x5c": '<br>
'["</em><em><strong>"]}, '<br>
'{"alg": "RS256", "kty": "RSA", "use": "sig", '<br>
'"n": '<br>
'"</strong></em><em>", '<br>
'"e": "AQAB", "kid": "kd5L2gFzHj_oIGFj3dAQb", '<br>
'"x5t": "</em><em><strong>", "x5c": '<br>
'["</strong></em><em>"]}]}')<br>
SOCIAL_AUTH_AUTH0_KEY: '</em><em><strong>'<br>
SOCIAL_AUTH_AUTH0_SCOPE: ['openid', 'profile']<br>
SOCIAL_AUTH_AUTH0_SECRET: '</strong></em><em>'<br>
SOCIAL_AUTH_PIPELINE:<br>
('social_core.pipeline.social_auth.social_details',<br>
'social_core.pipeline.social_auth.social_uid',<br>
'social_core.pipeline.social_auth.social_user',<br>
'social_core.pipeline.user.get_username',<br>
'social_core.pipeline.social_auth.associate_by_email',<br>
'social_core.pipeline.user.create_user',<br>
'social_core.pipeline.social_auth.associate_user',<br>
'social_core.pipeline.social_auth.load_extra_data',<br>
'social_core.pipeline.user.user_details')<br>
SOCIAL_AUTH_REDIRECT_IS_HTTPS: False<br>
<em><strong>: '</strong></em></em>'<br>
STATICFILES_DIRS: []<br>
STATICFILES_FINDERS: ['django.contrib.staticfiles.finders.FileSystemFinder',<br>
'django.contrib.staticfiles.finders.AppDirectoriesFinder']<br>
STATICFILES_STORAGE: 'django.contrib.staticfiles.storage.StaticFilesStorage'<br>
STATIC_ROOT: '<strong>'<br>
STATIC_URL: '/backend_static/'<br>
TEMPLATES: [{'APP_DIRS': True,<br>
'BACKEND': 'django.template.backends.django.DjangoTemplates',<br>
'DIRS': [],<br>
'OPTIONS': {'context_processors': ['django.template.context_processors.debug',<br>
'django.template.context_processors.request',<br>
'django.contrib.auth.context_processors.auth',<br>
'django.contrib.messages.context_processors.messages']}}]<br>
TEST_NON_SERIALIZED_APPS: []<br>
TEST_RUNNER: 'django.test.runner.DiscoverRunner'<br>
THOUSAND_SEPARATOR: ','<br>
TIME_FORMAT: 'P'<br>
TIME_INPUT_FORMATS: ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M']<br>
TIME_ZONE: 'UTC'<br>
USE_I18N: False<br>
USE_L10N: False<br>
USE_THOUSAND_SEPARATOR: False<br>
USE_TZ: True<br>
USE_X_FORWARDED_HOST: True<br>
USE_X_FORWARDED_PORT: True<br>
WSGI_APPLICATION: '</strong>.wsgi.application'<br>
X_FRAME_OPTIONS: 'DENY'<br>
YEAR_MONTH_FORMAT: 'F Y'<br>
is_overridden: &lt;bound method Settings.is_overridden of &lt;Settings "******.settings"&gt;&gt;<br>
deprecated_settings: None<br>
:</b></p></summary><p dir="auto"></p>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate"></code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>
<ul dir="auto">
<li><strong>Minimal Python Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Celery Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>
<details>
<summary><b><code class="notranslate">pip freeze</code> Output:amqp==5.0.6
celery==5.0.5
pickleshare==0.7.5</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate"></code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>
<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>
<details>
<p dir="auto">
import pickle
from celery.canvas import chord
from io import BytesIO
</p><p dir="auto">dict(pickle.load(BytesIO(pickle.dumps(chord([])))))</p>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">I expect that after pickle load I will get the same object as I dumped</p>
<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">dumped:<br>
{'task': 'celery.chord',<br>
'args': (),<br>
'kwargs': {'kwargs': {}, 'header': [], 'body': None},<br>
'options': {},<br>
'subtask_type': 'chord',<br>
'immutable': False,<br>
'chord_size': None}</p>
<p dir="auto">loaded:<br>
{'task': 'celery.chord',<br>
'args': (),<br>
'kwargs': {'kwargs': {'kwargs': {}}, 'header': [], 'body': None},<br>
'options': {'chord_size': None},<br>
'subtask_type': 'chord',<br>
'immutable': False,<br>
'chord_size': None}</p>
<p dir="auto">kwargs added multiple times.</p>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>
<p dir="auto"><strong>Celery version</strong>:</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b>
</summary>
<p dir="auto">
software -&gt; celery:5.1.0 (sun-harmonics) kombu:5.1.0 py:3.9.2
            billiard:3.6.4.0 redis:3.5.3
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:4.4.0-198-generic imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:redis results:redis://localhost/1
</p><p dir="auto">broker_url: 'redis://localhost:6379/1'<br>
result_backend: 'redis://localhost/1'<br>
result_serializer: 'pickle'<br>
deprecated_settings: None<br>
task_serializer: 'pickle'<br>
accept_content: ['pickle', 'json']<br>
result_expires: 3600<br>
task_default_queue: 'zabbix'</p>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<p dir="auto">Schedule a chord with beat via a task.</p>
<p dir="auto">I created a task that takes a chain/chord as parameter and executes it, since beat does not allow for scheduling anything but a simple task directly:<br>
@app.task()<br>
def execute_chain(ch):<br>
ch.apply_async(ignore_result=True)</p>
<p dir="auto">This is added to the beat schedule via<br>
ch = group(*zbxitems) | func.s() | publish_mqtt.s()<br>
sender.add_periodic_task(interval, execute_chain.s(ch), name=name)</p>
<h2 dir="auto">Required Dependencies</h2>

<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
amqp==5.0.6
billiard==3.6.4.0
celery==5.1.0
certifi==2021.5.30
chardet==4.0.0
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.2.0
colorama==0.4.4
idna==2.10
kombu==5.1.0
paho-mqtt==1.5.1
prompt-toolkit==3.0.18
pytz==2021.1
PyYAML==5.4.1
pyzabbix==0.8.2
redis==3.5.3
requests==2.25.1
six==1.16.0
urllib3==1.26.5
vine==5.0.0
wcwidth==0.2.5
(venv) root@vmt
</p>
</details>
<h3 dir="auto">Other Dependencies</h3>
<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"></pre></div>
<p dir="auto">version 3.9.5</p>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">This should continue to work, executing the chain again and again.</p>
<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">I works for several hours till it exits with:<br>
2021-06-11 06:25:54,253:CRITICAL:worker.start:line 207:Unrecoverable error: RecursionError('maximum recursion depth exceeded while pickling an object')<br>
Traceback (most recent call last):<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/worker.py", line 203, in start<br>
self.blueprint.start(self)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start<br>
step.start(parent)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/bootsteps.py", line 365, in start<br>
return self.obj.start()<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 326, in start<br>
blueprint.start(self)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start<br>
step.start(parent)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 618, in start<br>
c.loop(*c.loop_args())<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/loops.py", line 81, in asynloop<br>
next(loop)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/asynchronous/hub.py", line 361, in create_loop<br>
cb(*cbargs)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 1128, in on_readable<br>
self.cycle.on_readable(fileno)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 398, in on_readable<br>
chan.handlers<a href="">type</a><br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 789, in _brpop_read<br>
self.connection._deliver(loads(bytes_to_str(item)), dest)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/transport/virtual/base.py", line 980, in _deliver<br>
callback(message)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/transport/virtual/base.py", line 630, in _callback<br>
return callback(message)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/messaging.py", line 626, in _receive_callback<br>
return on_m(message) if on_m else self.receive(decoded, message)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 586, in on_task_received<br>
strategy(<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/strategy.py", line 201, in task_message_handler<br>
handle(req)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/worker.py", line 221, in _process_task_sem<br>
return self._quick_acquire(self._process_task, req)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/asynchronous/semaphore.py", line 57, in acquire<br>
callback(*partial_args, **partial_kwargs)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/worker.py", line 226, in _process_task<br>
req.execute_using_pool(self.pool)<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/request.py", line 691, in execute_using_pool<br>
result = apply_async(<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/concurrency/base.py", line 152, in apply_async<br>
return self.on_apply(target, args, kwargs,<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/billiard/pool.py", line 1530, in apply_async<br>
self._quick_put((TASK, (result._job, None, func, args, kwds)))<br>
File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/concurrency/asynpool.py", line 859, in send_job<br>
body = dumps(tup, protocol=protocol)<br>
RecursionError: maximum recursion depth exceeded while pickling an object</p>
<p dir="auto">When json dumping the task, I see this after a few iterations (mind the kwargs):<br>
2021-06-11 16:44:16,370:DEBUG:beat.execute_chain:line 90:{"task": "celery.chord", "args": [], "kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {<br>
"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}, "header": [{"task": "tasks.zabbix.get<br>
_zabbix_items_value", "args": ["Memory Free", "rd servers"], "kwargs": {}, "options": {}, "subtask_type": null, "immutable": true}, {"task": "tasks.zabbix.get_zabbix_items_value", "args": ["Available memory", "rd servers"], "kwargs": {}, "options": {}, "subtask_type": n<br>
ull, "immutable": true}], "body": {"task": "celery.chain", "args": [], "kwargs": {"tasks": [{"task": "tasks.aggregate.percentage_per_item", "args": [], "kwargs": {}, "options": {}, "subtask_type": null, "immutable": false}, {"task": "tasks.mqtt.publish_mqtt", "args": ["<br>
vmmqtt01", "zabbix/"], "kwargs": {}, "options": {}, "subtask_type": null, "immutable": false}]}, "options": {}, "subtask_type": "chain", "immutable": false}}, "options": {}, "subtask_type": "chord", "immutable": false}</p>
<p dir="auto">I tried also by cloning the chord first in the scheduled task and executing the clone, but same result.</p>