How to auto-reload Celery worker on Code Change

Table of Contents

Please check my course: The Definitive Guide to Celery and Django on testdriven.io

Introduction

This is #2 of my Django Celery Tutorial Series

Many newbie developers does not know Celery would not auto-reload worker like Django dev server so this might cause some weird problem when then learn Celery.

In this Django Celery tutorial, I would talk about how to auto-reload Celery worker on code change.

Here I will provide you two solutions to solve this problem.

Solution 1 (WatchDog)

We can use WatchDog to help us monitor the code files and rerun the command if necessary.

First, let's install the package, please note that PyYAML and argh also need to be installed as well.

$ pip install watchdog

Let's assume you run your Celery worker using command below

$ celery worker -A django_celery_example --loglevel=info

Now you can run the command in this way

$ watchmedo auto-restart -d django_celery_example/ -p '*.py' -- celery worker -A django_celery_example --loglevel=info
  1. -d django_celery_example told watchmedo to watch files under django_celery_example directory

  2. -p '*.py' told watchmedo only watch py files (so if you change js or scss files, the worker would not restart)

Another thing I want to say here is that if you press Ctrl + C twice to terminate above command, sometimes the Celery worker child process would not be closed, this might cause some weird problem in some cases.

So I prefer method below. Let's keep reading how it works.

Solution 2 (Django Autoreload)

As you know, if you change files in Djnago project, the dev server would restart because Django supports autoreload.

So here I would show you how to use Django autoreload code to help us solve this problem.

First, let's create a Django command

.
├── db.sqlite3
├── django_celery_example
│   ├── __init__.py
│   ├── celery.py
│   ├── management
│   │   ├── __init__.py
│   │   └── commands
│   │       ├── __init__.py
│   │       └── celery_worker.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── manage.py
├── polls
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
└── requirements.txt

As you can see, I created a Django command celery_worker

import shlex
import subprocess

from django.core.management.base import BaseCommand
from django.utils import autoreload


def restart_celery():
    cmd = 'pkill -f "celery worker"'
    subprocess.call(shlex.split(cmd))
    cmd = 'celery worker -A django_celery_example --loglevel=info'
    subprocess.call(shlex.split(cmd))


class Command(BaseCommand):

    def handle(self, *args, **options):
        print('Starting celery worker with autoreload...')

        # For Django>=2.2
        autoreload.run_with_reloader(restart_celery)

        # For django<2.1
        # autoreload.main(restart_celery)

Here we use Django autlreload function to run Celery worker for us, which is very clean and simple.

Conslution

In this Django Celery tutorial, I provide you 2 ways to solve the Celery worker autoreload problem.

Please check my course: The Definitive Guide to Celery and Django on testdriven.io

Launch Products Faster with Django

SaaS Hammer helps you launch products in faster way. It contains all the foundations you need so you can focus on your product.

Michael Yin

Michael is a Full Stack Developer from China who loves writing code, tutorials about Django, and modern frontend tech.

He has published some ebooks on leanpub and tech course on testdriven.io.

He is also the founder of the AccordBox which provides the web development services.

Django SaaS Template

It aims to save your time and money building your product

Learn More

Hotwire is the default frontend solution shipped in Rails, this book will teach you how to make it work with Django, you will learn building modern web applications without using much JavaScript.

Read More
© 2018 - 2024 AccordBox