|
======================== |
|
CODE SNIPPETS |
|
======================== |
|
TITLE: Install Celery from a source tarball |
|
DESCRIPTION: Provides instructions for downloading, extracting, building, and installing Celery directly from a compressed source archive. This method is typically used for specific versions or when PyPI is not an option. The final installation step may require superuser privileges if not in a virtual environment. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/includes/installation.txt#_snippet_2 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ tar xvfz celery-0.0.0.tar.gz |
|
$ cd celery-0.0.0 |
|
$ python setup.py build |
|
# python setup.py install |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Celery Python: Example Configuration File |
|
DESCRIPTION: This snippet provides a basic Celery configuration example, including broker settings, module imports, and result backend setup. It demonstrates how to configure Celery using `celeryconfig.py` for a simple setup. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/configuration.rst#_snippet_0 |
|
|
|
LANGUAGE: python |
|
CODE: |
|
``` |
|
## Broker settings. |
|
broker_url = 'amqp://guest:guest@localhost:5672//' |
|
|
|
# List of modules to import when the Celery worker starts. |
|
imports = ('myapp.tasks,') |
|
|
|
## Using the database to store task state and results. |
|
result_backend = 'db+sqlite:///results.db' |
|
|
|
task_annotations = {'tasks.add': {'rate_limit': '10/s'}} |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install Celery development version from GitHub using pip |
|
DESCRIPTION: Installs the latest development snapshots of Celery and its core dependencies (billiard, amqp, kombu, vine) directly from their respective GitHub repositories. This is suitable for testing new features or contributing to the project. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/includes/installation.txt#_snippet_3 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install https://github.com/celery/celery/zipball/main#egg=celery |
|
``` |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install https://github.com/celery/billiard/zipball/main#egg=billiard |
|
``` |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install https://github.com/celery/py-amqp/zipball/main#egg=amqp |
|
``` |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install https://github.com/celery/kombu/zipball/main#egg=kombu |
|
``` |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install https://github.com/celery/vine/zipball/main#egg=vine |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Start Celery Worker Consuming Specific Queues (CLI) |
|
DESCRIPTION: Shows how to start a Celery worker instance and specify a comma-separated list of queues to consume from using the -Q option. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/workers.rst#_snippet_41 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ celery -A proj worker -l INFO -Q foo,bar,baz |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install Celery with Gevent Pool Dependencies |
|
DESCRIPTION: This command installs the necessary Python packages: `gevent` for the asynchronous pool, `celery` for the distributed task queue, and `pybloom-live`. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/examples/gevent/README.rst#_snippet_0 |
|
|
|
LANGUAGE: bash |
|
CODE: |
|
``` |
|
$ python -m pip install gevent celery pybloom-live |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Start Celery Workers with Dedicated PID and Log Directories |
|
DESCRIPTION: Demonstrates how to start Celery workers using "celery multi" while specifying dedicated directories for PID and log files to prevent conflicts and organize output. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/next-steps.rst#_snippet_8 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ mkdir -p /var/run/celery |
|
$ mkdir -p /var/log/celery |
|
$ celery multi start w1 -A proj -l INFO --pidfile=/var/run/celery/%n.pid --logfile=/var/log/celery/%n%I.log |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install Celery via pip |
|
DESCRIPTION: Installs the Celery distributed task queue library from the Python Package Index (PyPI) using pip, ensuring the package is updated to its latest stable version. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/includes/installation.txt#_snippet_0 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install -U Celery |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install and Configure Celery Consul K/V Store Backend |
|
DESCRIPTION: Details the installation of the `python-consul2` library and provides examples for configuring Celery to use Consul as its result backend. It also explains the full URL syntax and its components, including the `one_client` option. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/configuration.rst#_snippet_95 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install python-consul2 |
|
``` |
|
|
|
LANGUAGE: Python |
|
CODE: |
|
``` |
|
CELERY_RESULT_BACKEND = 'consul://localhost:8500/' |
|
|
|
or:: |
|
|
|
result_backend = 'consul://localhost:8500/' |
|
``` |
|
|
|
LANGUAGE: text |
|
CODE: |
|
``` |
|
consul://host:port[?one_client=1] |
|
``` |
|
|
|
LANGUAGE: APIDOC |
|
CODE: |
|
``` |
|
Consul URL Components: |
|
host: Host name of the Consul server. |
|
port: The port the Consul server is listening to. |
|
one_client: By default, for correctness, the backend uses a separate client connection per operation. In cases of extreme load, the rate of creation of new connections can cause HTTP 429 "too many connections" error responses from the Consul server when under load. The recommended way to handle this is to enable retries in python-consul2 using the patch at https://github.com/poppyred/python-consul2/pull/31. Alternatively, if one_client is set, a single client connection will be used for all operations instead. This should eliminate the HTTP 429 errors, but the storage of results in the backend can become unreliable. |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Start Multiple Celery Workers with Custom Arguments using celery multi |
|
DESCRIPTION: Illustrates starting multiple Celery workers with "celery multi", applying different queue and log level configurations to specific worker instances using advanced command-line syntax. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/next-steps.rst#_snippet_9 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ celery multi start 10 -A proj -l INFO -Q:1-3 images,video -Q:4,5 data |
|
-Q default -L:4,5 debug |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Example Celery Configuration Module |
|
DESCRIPTION: Provides an example of a `celeryconfig.py` file, which defines configuration variables for a Celery application. This module can be loaded by `app.config_from_object()` to apply settings like `enable_utc` and `timezone`. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/application.rst#_snippet_9 |
|
|
|
LANGUAGE: python |
|
CODE: |
|
``` |
|
enable_utc = True |
|
timezone = 'Europe/London' |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install Celery with feature bundles via pip |
|
DESCRIPTION: Installs Celery along with specific feature dependencies, known as bundles, using pip. This allows users to include support for various serializers, concurrency models, or transport/backend systems. Multiple bundles can be specified by separating them with commas inside the brackets. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/includes/installation.txt#_snippet_1 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install "celery[librabbitmq]" |
|
``` |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install "celery[librabbitmq,redis,auth,msgpack]" |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install Celery and Eventlet dependencies |
|
DESCRIPTION: This command installs the necessary Python packages, `eventlet`, `celery`, and `pybloom-live`, required to run the Celery application with the Eventlet pool. It uses `pip` for package management. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/examples/eventlet/README.rst#_snippet_0 |
|
|
|
LANGUAGE: bash |
|
CODE: |
|
``` |
|
$ python -m pip install eventlet celery pybloom-live |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install Celery with Consul result backend extension |
|
DESCRIPTION: This command installs Celery along with the necessary dependencies to use Consul as a result backend, simplifying the setup process. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/history/whatsnew-4.0.rst#_snippet_34 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install celery[consul] |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Run Celery Event Capture via CLI |
|
DESCRIPTION: This command line interface (CLI) example demonstrates how to start a Celery event consumer to capture events using a custom event handler. It specifies the Celery application, the event consumer class, and the frequency for event processing. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/monitoring.rst#_snippet_36 |
|
|
|
LANGUAGE: bash |
|
CODE: |
|
``` |
|
celery -A proj events -c myapp.DumpCam --frequency=2.0 |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Run RabbitMQ with Docker |
|
DESCRIPTION: This Docker command starts a RabbitMQ container, mapping port 5672. It provides a quick way to get a RabbitMQ broker running for development or testing purposes. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/first-steps-with-celery.rst#_snippet_1 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ docker run -d -p 5672:5672 rabbitmq |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install setproctitle for Enhanced Celery Process Visibility |
|
DESCRIPTION: This console command demonstrates how to install the `setproctitle` Python package. Installing this library allows Celery worker processes to display more descriptive names in `ps` listings, making it easier to identify different process types for debugging and management. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/faq.rst#_snippet_19 |
|
|
|
LANGUAGE: Console |
|
CODE: |
|
``` |
|
$ pip install setproctitle |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install Celery Flower Web Monitor |
|
DESCRIPTION: This command uses pip, Python's package installer, to install the Flower web-based monitoring tool for Celery. It's a prerequisite for running Flower. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/monitoring.rst#_snippet_19 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install flower |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Define a Simple Celery Application and Task |
|
DESCRIPTION: This Python code demonstrates the most basic setup for a Celery application. It initializes a Celery instance named 'hello', configured to connect to an AMQP broker at localhost. A simple task, `hello`, is defined using the `@app.task` decorator, which returns the string 'hello world'. This example showcases Celery's ease of use and minimal configuration requirements. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/introduction.rst#_snippet_0 |
|
|
|
LANGUAGE: Python |
|
CODE: |
|
``` |
|
from celery import Celery |
|
|
|
app = Celery('hello', broker='amqp://guest@localhost//') |
|
|
|
@app.task |
|
def hello(): |
|
return 'hello world' |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Run Celery HTTP Gateway Service |
|
DESCRIPTION: Commands to start the Celery HTTP gateway service. The `syncdb` command is optional and only needed if using a database backend for Celery results. The `runserver` command starts the Django development server. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/examples/celery_http_gateway/README.rst#_snippet_0 |
|
|
|
LANGUAGE: Bash |
|
CODE: |
|
``` |
|
$ python manage.py syncdb |
|
``` |
|
|
|
LANGUAGE: Bash |
|
CODE: |
|
``` |
|
$ python manage.py runserver |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Start Celery Flower with Custom Broker URL |
|
DESCRIPTION: These commands demonstrate how to start the Flower web server while providing a custom broker URL. This is essential when Celery is configured to use a broker other than the default, such as RabbitMQ (AMQP) or Redis. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/monitoring.rst#_snippet_22 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ celery --broker=amqp://guest:guest@localhost:5672// flower |
|
or |
|
$ celery --broker=redis://guest:guest@localhost:6379/0 flower |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install Celery with Amazon SQS Support |
|
DESCRIPTION: Installs Celery along with its Amazon SQS dependencies using pip, leveraging the `celery[sqs]` bundle for a complete setup. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/backends-and-brokers/sqs.rst#_snippet_0 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install "celery[sqs]" |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Start Celery Workers in Background with celery multi |
|
DESCRIPTION: Demonstrates how to start one or more Celery workers in the background using the "celery multi" command, specifying the application module and log level. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/next-steps.rst#_snippet_4 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ celery multi start w1 -A proj -l INFO |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Starting Celery Worker |
|
DESCRIPTION: Command to start the Celery worker process. The `-A` flag specifies the Celery application instance to load, `worker` indicates the worker mode, and `-l INFO` sets the logging level to INFO for detailed output. The worker processes tasks from configured queues. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/next-steps.rst#_snippet_2 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
celery -A proj worker -l INFO |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Start Celery Worker with Gevent Pool |
|
DESCRIPTION: This command starts the Celery worker in the `examples/gevent` directory, setting the log level to INFO, concurrency to 500, and specifying the `gevent` pool. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/examples/gevent/README.rst#_snippet_1 |
|
|
|
LANGUAGE: bash |
|
CODE: |
|
``` |
|
$ cd examples/gevent |
|
$ celery worker -l INFO --concurrency=500 --pool=gevent |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install RabbitMQ Server on Ubuntu/Debian |
|
DESCRIPTION: This command installs the RabbitMQ message broker server on Ubuntu or Debian systems using `apt-get`. RabbitMQ is a stable and feature-complete choice for production environments. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/first-steps-with-celery.rst#_snippet_0 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ sudo apt-get install rabbitmq-server |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install and Configure Celery CouchDB Backend |
|
DESCRIPTION: Provides instructions for installing the necessary Python library for the CouchDB backend and an example of how to configure Celery to use CouchDB as its result backend via a URL. It also details the components of the CouchDB connection URL. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/configuration.rst#_snippet_93 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install celery[couchdb] |
|
``` |
|
|
|
LANGUAGE: Python |
|
CODE: |
|
``` |
|
result_backend = 'couchdb://username:password@host:port/container' |
|
``` |
|
|
|
LANGUAGE: APIDOC |
|
CODE: |
|
``` |
|
CouchDB URL Components: |
|
username: User name to authenticate to the CouchDB server as (optional). |
|
password: Password to authenticate to the CouchDB server (optional). |
|
host: Host name of the CouchDB server. Defaults to localhost. |
|
port: The port the CouchDB server is listening to. Defaults to 8091. |
|
container: The default container the CouchDB server is writing to. Defaults to default. |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Link Celery tasks sequentially using chain |
|
DESCRIPTION: Illustrates how to use `celery.chain` to link tasks together, so that the output of one task becomes the input of the next. Provides an example of a simple arithmetic chain. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/next-steps.rst#_snippet_38 |
|
|
|
LANGUAGE: Python |
|
CODE: |
|
``` |
|
from celery import chain |
|
from proj.tasks import add, mul |
|
|
|
# (4 + 4) * 8 |
|
chain(add.s(4, 4) | mul.s(8))().get() |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install Celery via pip |
|
DESCRIPTION: This command installs the Celery library from PyPI using pip, the standard Python package installer. Celery is a Python-based task queue. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/first-steps-with-celery.rst#_snippet_3 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install celery |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install django-celery-results Library |
|
DESCRIPTION: This command installs the `django-celery-results` library, which provides result backends for Celery using Django's ORM or cache framework. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/django/first-steps-with-django.rst#_snippet_9 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install django-celery-results |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Start Celery Events Curses Interface |
|
DESCRIPTION: Launches a curses-based interface for interactive monitoring of Celery events. This provides a more structured and real-time view of worker and task activities. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/next-steps.rst#_snippet_54 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ celery -A proj events |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Start Celery Flower Web Server (Custom Port) |
|
DESCRIPTION: This command starts the Flower web server, explicitly specifying the port it should listen on using the `--port` argument. This is useful for avoiding port conflicts or running multiple instances. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/monitoring.rst#_snippet_21 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ celery -A proj flower --port=5555 |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Get Celery CLI Help |
|
DESCRIPTION: Displays a list of all available commands for the Celery command-line interface. This is useful for discovering new commands or recalling command syntax. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/monitoring.rst#_snippet_0 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ celery --help |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Installing Python Requirements for Celery Project |
|
DESCRIPTION: This snippet provides the command to install all necessary Python dependencies for the Celery-Django project from the `requirements.txt` file. It assumes a local RabbitMQ server is running for message brokering. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/examples/django/README.rst#_snippet_0 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install -r requirements.txt |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Get Specific Celery Command Help |
|
DESCRIPTION: Provides detailed help and usage information for a specific Celery command. Replace `<command>` with the name of the command you want to learn more about. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/monitoring.rst#_snippet_1 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ celery <command> --help |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Typical Celery Task State Progression |
|
DESCRIPTION: Outlines the common state transitions for a typical Celery task. The 'STARTED' state is only recorded if `task_track_started` is enabled. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/next-steps.rst#_snippet_26 |
|
|
|
LANGUAGE: Text |
|
CODE: |
|
``` |
|
PENDING -> STARTED -> SUCCESS |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Start Celery Events with Snapshot Camera |
|
DESCRIPTION: This command starts the `celery events` monitor configured to use a specific snapshot camera class and a defined frequency for capturing events. Snapshot cameras are used for persistent event logging. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/monitoring.rst#_snippet_26 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ celery -A proj events --camera=<camera-class> --frequency=1.0 |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install and Configure Celery Couchbase Backend |
|
DESCRIPTION: Instructions for installing the `couchbase` library for Celery and configuring the `result_backend` with a Couchbase connection URL. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/configuration.rst#_snippet_85 |
|
|
|
LANGUAGE: Shell |
|
CODE: |
|
``` |
|
$ pip install celery[couchbase] |
|
``` |
|
|
|
LANGUAGE: Python |
|
CODE: |
|
``` |
|
result_backend = 'couchbase://username:password@host:port/bucket' |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install Celery with Redis support |
|
DESCRIPTION: This command installs Celery along with the necessary dependencies for Redis support, using the `celery[redis]` bundle via pip. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/backends-and-brokers/redis.rst#_snippet_0 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ pip install -U "celery[redis]" |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Execute Celery `urlopen` Task for Single URL |
|
DESCRIPTION: This snippet demonstrates how to navigate to the example directory and execute the `urlopen` task from an interactive Python session to fetch a URL and get its response body size. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/examples/gevent/README.rst#_snippet_2 |
|
|
|
LANGUAGE: bash |
|
CODE: |
|
``` |
|
$ cd examples/gevent |
|
$ python |
|
``` |
|
|
|
LANGUAGE: python |
|
CODE: |
|
``` |
|
>>> from tasks import urlopen |
|
>>> urlopen.delay('https://www.google.com/').get() |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Example Celerybeat Configuration File |
|
DESCRIPTION: Provides an example configuration for `/etc/default/celerybeat` for a Python project, specifying the Celery binary path, app instance, working directory, and additional beat options. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/daemonizing.rst#_snippet_5 |
|
|
|
LANGUAGE: bash |
|
CODE: |
|
``` |
|
# Absolute or relative path to the 'celery' command: |
|
CELERY_BIN="/usr/local/bin/celery" |
|
#CELERY_BIN="/virtualenvs/def/bin/celery" |
|
|
|
# App instance to use |
|
# comment out this line if you don't use an app |
|
CELERY_APP="proj" |
|
# or fully qualified: |
|
#CELERY_APP="proj.tasks:app" |
|
|
|
# Where to chdir at start. |
|
CELERYBEAT_CHDIR="/opt/Myproject/" |
|
|
|
# Extra arguments to celerybeat |
|
CELERYBEAT_OPTS="--schedule=/var/run/celery/celerybeat-schedule" |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Start Celery Multi-Node with Specific Concurrency per Index |
|
DESCRIPTION: This Bash command demonstrates how to use "celery multi start" to launch multiple named nodes (A, B, C, D) and assign specific concurrency levels based on their index in the argument list. For example, node A (index 1) gets 4 processes, and nodes B, C, D (indices 2-4) each get 8 processes. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/history/changelog-3.1.rst#_snippet_28 |
|
|
|
LANGUAGE: bash |
|
CODE: |
|
``` |
|
celery multi start A B C D -c:1 4 -c:2-4 8 |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Run Celery Worker with Main Module Task |
|
DESCRIPTION: Provides an example of a `tasks.py` file that defines a Celery task and includes logic to start a worker when the module is executed directly. This demonstrates how tasks are named with `__main__` when the module is run as a program. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/application.rst#_snippet_2 |
|
|
|
LANGUAGE: python |
|
CODE: |
|
``` |
|
from celery import Celery |
|
app = Celery() |
|
|
|
@app.task |
|
def add(x, y): return x + y |
|
|
|
if __name__ == '__main__': |
|
args = ['worker', '--loglevel=INFO'] |
|
app.worker_main(argv=args) |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install Celery with Google Pub/Sub Support |
|
DESCRIPTION: Install Celery and its Google Pub/Sub dependencies using pip. This command ensures all necessary packages are available for broker functionality. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/backends-and-brokers/gcpubsub.rst#_snippet_0 |
|
|
|
LANGUAGE: Shell |
|
CODE: |
|
``` |
|
pip install "celery[gcpubsub]" |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Define Custom Celery Inspect Command to Get Prefetch Count |
|
DESCRIPTION: Demonstrates how to create a custom inspect command using the `@inspect_command` decorator. This example defines `current_prefetch_count`, which retrieves and returns the current task prefetch count from the worker's consumer state. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/workers.rst#_snippet_38 |
|
|
|
LANGUAGE: python |
|
CODE: |
|
``` |
|
from celery.worker.control import inspect_command |
|
|
|
@inspect_command() |
|
def current_prefetch_count(state): |
|
return {'prefetch_count': state.consumer.qos.value} |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Restart Celery Worker using celery multi |
|
DESCRIPTION: Demonstrates how to start and restart a Celery worker instance using the `celery multi` command, suitable for development environments. It shows starting a worker with specific app, log level, and PID file, then restarting it. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/workers.rst#_snippet_8 |
|
|
|
LANGUAGE: console |
|
CODE: |
|
``` |
|
$ celery multi start 1 -A proj -l INFO -c4 --pidfile=/var/run/celery/%n.pid |
|
$ celery multi restart 1 --pidfile=/var/run/celery/%n.pid |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Celery Task State: STARTED |
|
DESCRIPTION: Describes the STARTED state for Celery tasks, indicating a task has begun execution. This state is not reported by default and requires enabling via `Task.track_started`. Includes metadata like process ID and hostname. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/tasks.rst#_snippet_51 |
|
|
|
LANGUAGE: APIDOC |
|
CODE: |
|
``` |
|
State: STARTED |
|
Description: Task has been started. |
|
Not reported by default, to enable please see @Task.track_started. |
|
Meta-data: pid and hostname of the worker process executing the task. |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Install Celery from a downloaded source tarball |
|
DESCRIPTION: Provides instructions for installing Celery by downloading and extracting its source code. This method involves building the package and then installing it, which may require privileged access if not using a virtual environment. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/README.rst#_snippet_5 |
|
|
|
LANGUAGE: Shell |
|
CODE: |
|
``` |
|
$ tar xvfz celery-0.0.0.tar.gz |
|
$ cd celery-0.0.0 |
|
$ python setup.py build |
|
# python setup.py install |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Example Celery Configuration File (`celeryconfig.py`) (Python) |
|
DESCRIPTION: Provides a comprehensive example of a `celeryconfig.py` file, defining essential Celery settings such as broker URL, result backend, task/result serialization, accepted content types, timezone, and UTC enablement. This file serves as a centralized configuration source. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/getting-started/first-steps-with-celery.rst#_snippet_18 |
|
|
|
LANGUAGE: python |
|
CODE: |
|
``` |
|
broker_url = 'pyamqp://' |
|
result_backend = 'rpc://' |
|
|
|
task_serializer = 'json' |
|
result_serializer = 'json' |
|
accept_content = ['json'] |
|
timezone = 'Europe/Oslo' |
|
enable_utc = True |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Celery Systemd Service Unit File Example |
|
DESCRIPTION: This comprehensive example provides a `systemd` unit file (`celery.service`) for managing Celery as a background service. It defines the service's description, dependencies, execution type, user/group, working directory, and commands for starting, stopping, reloading, and restarting Celery workers, ensuring automatic restarts on failure. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/userguide/daemonizing.rst#_snippet_10 |
|
|
|
LANGUAGE: bash |
|
CODE: |
|
``` |
|
[Unit] |
|
Description=Celery Service |
|
After=network.target |
|
|
|
[Service] |
|
Type=forking |
|
User=celery |
|
Group=celery |
|
EnvironmentFile=/etc/conf.d/celery |
|
WorkingDirectory=/opt/celery |
|
ExecStart=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi start $CELERYD_NODES \ |
|
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \ |
|
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS' |
|
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait $CELERYD_NODES \ |
|
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \ |
|
--loglevel="${CELERYD_LOG_LEVEL}"' |
|
ExecReload=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi restart $CELERYD_NODES \ |
|
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \ |
|
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS' |
|
Restart=always |
|
|
|
[Install] |
|
WantedBy=multi-user.target |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Python Celery Single-Mode API Usage Example |
|
DESCRIPTION: Provides an example of using Celery in its 'single-mode' API, which was prevalent before the introduction of the 'app' concept. This mode typically involves direct imports from Celery sub-modules and demonstrates a basic task definition. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/internals/guide.rst#_snippet_4 |
|
|
|
LANGUAGE: python |
|
CODE: |
|
``` |
|
from celery import task |
|
from celery.task.control import inspect |
|
|
|
from .models import CeleryStats |
|
|
|
@task |
|
``` |
|
|
|
---------------------------------------- |
|
|
|
TITLE: Create a Basic Celery Application |
|
DESCRIPTION: This snippet demonstrates how to initialize a minimal Celery application. It defines a Celery app instance, connects to a message broker (RabbitMQ in this example), and registers a simple task that returns 'hello world'. |
|
|
|
SOURCE: https://github.com/celery/celery/blob/main/docs/includes/introduction.txt#_snippet_0 |
|
|
|
LANGUAGE: Python |
|
CODE: |
|
``` |
|
from celery import Celery |
|
|
|
app = Celery('hello', broker='amqp://guest@localhost//') |
|
|
|
@app.task |
|
def hello(): |
|
return 'hello world' |
|
``` |