Rowan Bohde

Async Workers in Django with Celery

Posted by Rowan Bohde

Celery, the distributed task queue framework, recently introduced asynchronous workers in the 2.2 release. I found that when I tried to use the eventlet worker using Celery's Django integration, it started to hang after making some database calls, at least when using the psycopg2 database backend.

I found that I could get around this by applying eventlet's monkey patch before Django starts loading. I did this by writing a separate manage.py, like the following.

Now I'm able to run ./async_manage.py celeryd -P eventlet -c 100, and not have it hang on database calls.

I've not tried this method with gevent, but a similar approach should work.