celery python

Distributed Task Queue

4 years after

================================= celery - Distributed Task Queue

.. image:: http://cloud.github.com/downloads/ask/celery/celery_favicon_128.png

:Version: 2.1.0a1 :Web: http://celeryproject.org/ :Download: http://pypi.python.org/pypi/celery/ :Source: http://github.com/ask/celery/ :Keywords: task queue, job queue, asynchronous, rabbitmq, amqp, redis, python, webhooks, queue, distributed

--

Celery is an open source asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more worker servers. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

Celery is already used in production to process millions of tasks a day.

Celery is written in Python, but the protocol can be implemented in any language. It can also operate with other languages using webhooks_.

The recommended message broker is RabbitMQ, but support for Redis and databases (SQLAlchemy_) is also available.

You may also be pleased to know that full Django integration exists, delivered by the django-celery_ package.

.. RabbitMQ: http://www.rabbitmq.com/ .. Redis: http://code.google.com/p/redis/ .. SQLAlchemy: http://www.sqlalchemy.org/ .. django-celery: http://pypi.python.org/pypi/django-celery .. _operate with other languages using webhooks: http://ask.github.com/celery/userguide/remote-tasks.html

.. contents:: :local:

Overview

This is a high level overview of the architecture.

.. image:: http://cloud.github.com/downloads/ask/celery/Celery-Overview-v4.jpg

The broker pushes tasks to the worker servers. A worker server is a networked machine running celeryd. This can be one or more machines depending on the workload.

The result of the task can be stored for later retrieval (called its "tombstone").

Example

You probably want to see some code by now, so here's an example task adding two numbers: ::

from celery.decorators import task

@task
def add(x, y):
    return x + y

You can execute the task in the background, or wait for it to finish::

>>> result = add.delay(4, 4)
>>> result.wait() # wait for and return the result
8

Simple!

Features

+-----------------+----------------------------------------------------+
| Messaging       | Supported brokers include `RabbitMQ`_, `Stomp`_,   |
|                 | `Redis`_, and most common SQL databases.           |
+-----------------+----------------------------------------------------+
| Robust          | Using `RabbitMQ`, celery survives most error       |
|                 | scenarios, and your tasks will never be lost.      |
+-----------------+----------------------------------------------------+
| Distributed     | Runs on one or more machines. Supports             |
|                 | `clustering`_ when used in combination with        |
|                 | `RabbitMQ`_. You can set up new workers without    |
|                 | central configuration (e.g. use your dads laptop   |
|                 | while the queue is temporarily overloaded).        |
+-----------------+----------------------------------------------------+
| Concurrency     | Tasks are executed in parallel using the           |
|                 | ``multiprocessing`` module.                        |
+-----------------+----------------------------------------------------+
| Scheduling      | Supports recurring tasks like cron, or specifying  |
|                 | an exact date or countdown for when after the task |
|                 | should be executed.                                |
+-----------------+----------------------------------------------------+
| Performance     | Able to execute tasks while the user waits.        |
+-----------------+----------------------------------------------------+
| Return Values   | Task return values can be saved to the selected    |
|                 | result store backend. You can wait for the result, |
|                 | retrieve it later, or ignore it.                   |
+-----------------+----------------------------------------------------+
| Result Stores   | Database, `MongoDB`_, `Redis`_, `Tokyo Tyrant`,    |
|                 | `AMQP`_ (high performance).                        |
+-----------------+----------------------------------------------------+
| Webhooks        | Your tasks can also be HTTP callbacks, enabling    |
|                 | cross-language communication.                      |
+-----------------+----------------------------------------------------+
| Rate limiting   | Supports rate limiting by using the token bucket   |
|                 | algorithm, which accounts for bursts of traffic.   |
|                 | Rate limits can be set for each task type, or      |
|                 | globally for all.                                  |
+-----------------+----------------------------------------------------+
| Routing         | Using AMQP you can route tasks arbitrarily to      |
|                 | different workers.                                 |
+-----------------+----------------------------------------------------+
| Remote-control  | You can rate limit and delete (revoke) tasks       |
|                 | remotely.                                          |
+-----------------+----------------------------------------------------+
| Monitoring      | You can capture everything happening with the      |
|                 | workers in real-time by subscribing to events.     |
|                 | A real-time web monitor is in development.         |
+-----------------+----------------------------------------------------+
| Serialization   | Supports Pickle, JSON, YAML, or easily defined     |
|                 | custom schemes. One task invocation can have a     |
|                 | different scheme than another.                     |
+-----------------+----------------------------------------------------+
| Tracebacks      | Errors and tracebacks are stored and can be        |
|                 | investigated after the fact.                       |
+-----------------+----------------------------------------------------+
| UUID            | Every task has an UUID (Universally Unique         |
|                 | Identifier), which is the task id used to query    |
|                 | task status and return value.                      |
+-----------------+----------------------------------------------------+
| Retries         | Tasks can be retried if they fail, with            |
|                 | configurable maximum number of retries, and delays |
|                 | between each retry.                                |
+-----------------+----------------------------------------------------+
| Task Sets       | A Task set is a task consisting of several         |
|                 | sub-tasks. You can find out how many, or if all    |
|                 | of the sub-tasks has been executed, and even       |
|                 | retrieve the results in order. Progress bars,      |
|                 | anyone?                                            |
+-----------------+----------------------------------------------------+
| Made for Web    | You can query status and results via URLs,         |
|                 | enabling the ability to poll task status using     |
|                 | Ajax.                                              |
+-----------------+----------------------------------------------------+
| Error e-mails   | Can be configured to send e-mails to the           |
|                 | administrators when tasks fails.                   |
+-----------------+----------------------------------------------------+
| Supervised      | Pool workers are supervised and automatically      |
|                 | replaced if they crash.                            |
+-----------------+----------------------------------------------------+

.. clustering: http://www.rabbitmq.com/clustering.html .. AMQP: http://www.amqp.org/ .. Stomp: http://stomp.codehaus.org/ .. MongoDB: http://www.mongodb.org/ .. _Tokyo Tyrant: http://tokyocabinet.sourceforge.net/

Documentation

The latest documentation_ with user guides, tutorials and API reference is hosted at Github.

.. _latest documentation: http://ask.github.com/celery/

Installation

You can install celery either via the Python Package Index (PyPI) or from source.

To install using pip,::

$ pip install celery

To install using easy_install,::

$ easy_install celery

Downloading and installing from source

Download the latest version of celery from http://pypi.python.org/pypi/celery/

You can install it by doing the following,::

$ tar xvfz celery-0.0.0.tar.gz
$ cd celery-0.0.0
$ python setup.py build
# python setup.py install # as root

Using the development version

You can clone the repository by doing the following::

$ git clone git://github.com/ask/celery.git

Getting Help

Mailing list

For discussions about the usage, development, and future of celery, please join the celery-users_ mailing list.

.. _celery-users: http://groups.google.com/group/celery-users/

IRC

Come chat with us on IRC. The #celery channel is located at the Freenode network.

.. #celery: irc://irc.freenode.net/celery .. Freenode: http://freenode.net

Bug tracker

If you have any suggestions, bug reports or annoyances please report them to our issue tracker at http://github.com/ask/celery/issues/

Wiki

http://wiki.github.com/ask/celery/

Contributing

Development of celery happens at Github: http://github.com/ask/celery

You are highly encouraged to participate in the development of celery. If you don't like Github (for some reason) you're welcome to send regular patches.

License

This software is licensed under the New BSD License. See the LICENSE file in the top distribution directory for the full license text.

.. # vim: syntax=rst expandtab tabstop=4 shiftwidth=4 shiftround

Related Repositories

django-celery

django-celery

Old Celery integration project for Django ...

django-celery

django-celery

Old Celery integration project for Django ...

django-celery

django-celery

Celery integration for Django ...

django-celery

django-celery

Celery integration for Django ...

django-celery

django-celery

Celery integration for Django ...


Top Contributors

nvie brosner vbabiy fredj gthb gcachet stovenator markhellewell heyman rlotun kmike jasonbaker paltman abecciu bradjasper fberger jonashaag HonzaKral westurner winhamwr andrewwatts mitsuhiko aaronelliotross jonozzz jbalogh miracle2k fladi dottedmag bfirsh

Releases

-   v2.0.2 zip tar
-   v2.0.1 zip tar
-   v2.0.0 zip tar
-   v1.1.3 zip tar
-   v1.1.2 zip tar
-   v1.1.1 zip tar
-   v1.0.6 zip tar
-   v1.0.5 zip tar
-   v1.0.4 zip tar
-   v1.0.3 zip tar
-   v1.0.3-pre2 zip tar
-   v1.0.3-pre1 zip tar
-   v1.0.2 zip tar
-   v1.0.1 zip tar
-   v1.0.1-pre2 zip tar
-   v1.0.0 zip tar
-   v1.0.0-pre4 zip tar
-   v1.0.0-pre3 zip tar
-   v1.0.0-pre2 zip tar
-   v0.9.7 zip tar
-   v0.9.6 zip tar
-   v0.9.5 zip tar
-   v0.9.4 zip tar
-   v0.9.3 zip tar
-   v0.9.2 zip tar
-   v0.8.4 zip tar
-   v0.8.3 zip tar
-   v0.8.2 zip tar
-   v0.8.0 zip tar
-   v0.8.0-pre1 zip tar