This is the #2 of my Django Celery tutorial series, and you can find the first one here How to setup Celery with Django
Many newbie developers does not know Celery would not auto-reload worker like Django dev server so this might cause some weird problem when then learn Celery.
In this Django Celery tutorial, I would talk about how to auto-reload Celery worker on code change.
Here I will provide you two solutions to solve this problem.
Solution 1 (WatchDog)
We can use
WatchDog to help us monitor the code files and rerun the command if necessary.
First, let's install the package, please note that
argh also need to be installed as well.
$ pip install watchdog==0.8.3 $ pip install PyYAML==3.12 $ pip install argh==0.26.2
Let's assume you run your Celery worker using command below
$ celery worker -A django_celery_example --loglevel=info
Now you can run the command in this way
$ watchmedo auto-restart -d django_celery_example/ -p '*.py' -- celery worker -A django_celery_example --loglevel=info
watchmedoto watch files under
pyfiles (so if you change js or scss files, the worker would not restart)
Another thing I want to say here is that if you press
Ctrl + C twice to terminate above command, sometimes the Celery worker child process would not be closed, this might cause some weird problem in some cases.
So I prefer method below. Let's keep reading how it works.
Solution 2 (Django Autoreload)
As you know, if you change files in Djnago project, the dev server would restart because Django supports
So here I would show you how to use Django
autoreload code to help us solve this problem.
First, let's create a Django command
. ├── db.sqlite3 ├── django_celery_example │ ├── __init__.py │ ├── celery.py │ ├── management │ │ ├── __init__.py │ │ └── commands │ │ ├── __init__.py │ │ └── celery_worker.py │ ├── settings.py │ ├── urls.py │ └── wsgi.py ├── manage.py ├── polls │ ├── __init__.py │ ├── admin.py │ ├── apps.py │ ├── migrations │ │ └── __init__.py │ ├── models.py │ ├── tests.py │ └── views.py └── requirements.txt
As you can see, I created a Django command
import shlex import subprocess from django.core.management.base import BaseCommand from django.utils import autoreload def restart_celery(): cmd = 'pkill -f "celery worker"' subprocess.call(shlex.split(cmd)) cmd = 'celery worker -A django_celery_example --loglevel=info' subprocess.call(shlex.split(cmd)) class Command(BaseCommand): def handle(self, *args, **options): print('Starting celery worker with autoreload...') # For Django>=2.2 autoreload.run_with_reloader(restart_celery) # For django<2.1 # autoreload.main(restart_celery)
Here we use Django
autlreload function to run Celery worker for us, which is very clean and simple.
In this Django Celery tutorial, I provide you 2 ways to solve the Celery worker autoreload problem.
If you have any question, please feel free to contact us.