How to do A/B Testing in Wagtail CMS

Last updated on Feb 27 2019 by Michael Yin

Introduction

Split Testing, also know as A/B testing, is a very important way to optimize the site conversion rate.

In this Wagtail tutorial, I will show you how to do Split Testing in Wagtail CMS. After reading this tutorial, you will:

  1. Learn how to use wagtail-experiments to setup A/B test in Wagtail.

  2. Understand how wagtail-experiments work.

  3. Get some tips and useful code snippets to work with wagtail-experiments

Step 1: Install wagtail-experiments

Note: The source code used in this tutorial is located in wagtail-experiments-tutorial. You can help other people by giving it a star.

First, let's install wagtail-experiments in our Wagtail project.

pip install wagtail-experiments

And then edit the settings/base.py, add the packages to the INSTALLED_APPS, modeladmin is also needed here.

INSTALLED_APPS = [
    'wagtail.contrib.modeladmin',
    'experiments',
]

After config is done, please migrate the database schema.

python manage.py migrate
python nmanage.py runserver

Step 2: Setup A/B test in Wagtail admin

Let's assume you want to do some modification on your existing blog post, because you wish more people subscribe to your blog.

Here I will teach you how to do it in Wagtail admin.

Now you have one post welcome, we call it Page A, then you copied the page and make the new page(Page B) as a child of the Page A. Here to make you better distinguish, the Page B have title Welcome-alternative and This is the alternative version in page body.

The only difference between Page A and Page B is the body content. You should not test many things at once because this might lead to the wrong conclusion.

So the page structure in your Wagtail project would seem like this.

Blog Home
    Welcome Page (Page A)
        Welcome-alternative (Page B)
    Contact Page

Let's assume if the reader visit Contact page after visiting Page A or Page B, we treat it a conversion.

Now let's setup A/B test in Wagtail admin.

Go to settings/experiments,

  1. Set Control page to Welcome Page, when reader visit this page url, the wagtail-experiments would handle the request.

  2. Set Alternatives to Welcome-alternative, the wagtail-experiments would pick one from this list to render it.

  3. Set Goal to Contact, if the reader visits the welcome page and then contact page, the wagtail would record the conversion.

  4. Change the status to live to make this experiment take effect.

Step 3: Check A/B test to see if it work

Now try to visit http://127.0.0.1:8000/blog/welcome/ in incognito window.

Here you might see the Page A or Page B, if you can not see the This is the alternative version in page content, just clear cookies and try again.

After you see the alternative page, click the contact link in the top menu to do a conversion.

Now go to wagtail admin settings/experiments/show report to check. Then you will see your conversion has been recorded.

Step 4: How wagtail experiments work

Now you already have a basic understanding about how a/b test work in Wagtail, so let me give you more detail about how it works.

  1. When you visit the http://127.0.0.1:8000/blog/welcome/, wagtail-experiments would detect if there are any a/b test's control page is that page.

  2. If it found a record, it would check which page to render based on the django session. That is why you can clear the cookie to make it start over again.

  3. When you visit the contact page, wagtail-experiments would detect if there are any a/b test's goal page is that page.

  4. If it found a record and record does not conversion yet, it would set it conversion. Here you should know even you do conversion in one django session many times it would only record one conversion

Step 5: How to make js work with wagtail-experiments

Sometimes, you want to record conversion using javascript, wagtail-experiments also support this feature.

Add code below to Django urls.py

from experiments import views as experiment_views

urlpatterns = [
    # ...
    url(r'^experiments/complete/([^/]+)/$', experiment_views.record_completion, name="experiments_complete"),
    # ...
]
<script>
  $(function() {
    $('form').on('submit', function (event) {
      event.preventDefault();
      var form = this;
      $.ajax({
        type: "GET",
        url: "{% url 'experiments_complete' experiment_slug %}",
        complete: function (obj) {
          form.submit();
        },
      });
    });
  });
</script>

You can use the js code above in your wagtail page template. experiment_slug should be config as page variable. When user submit form in that page, an ajax request would sent to wagtail experiments to record and then the form would be submitted. You can modify the code to make that work in your project.

Step 5: How to a/b test page design

Above method can help you find the copywriting which have higher conversion, but we should know sometimes design is also a very important part for a good product. So how to test different design in the wagtail project?

The key here is the template variable. We can make Page A and Page B have different template variables. Then test which page have better conversion rate.

class AlternateTemplateMixin(models.Model):
    alternate_template = models.CharField(max_length=100, blank=True)
    experiment_slug = models.CharField(max_length=50, blank=True)

    settings_panels = [
        FieldPanel('alternate_template'),
        FieldPanel('experiment_slug'),
    ]

    class Meta:
        abstract = True

    def get_template(self, request, *args, **kwargs):
        if self.alternate_template:
            return self.alternate_template
        else:
            return super().get_template(request)

So you can use in this way

class PostPage(AlternateTemplateMixin, Page):
    pass

Step 6: Something you should know

There is no silver bullet in this world and this solution might also have some limitations.

  1. wagtail-experiments is built based on db so it might not work very well on large site. You can solve this issue by building custom backend.

  2. wagtail-experiments is built based on Django session so the statistics data might not be very accurate in some cases.

Conclusion

In this wagtail tutorial, we learned how to setup a/b tests in wagtail project.

You can get all source code of this article here wagtail-experiments-tutorial

If you have any question with your Wagtail project, just feel free to contact us.

Michael is a passionate Python developer from China and love writing code, articles about Django and Wagtail CMS.

Michael Yin

Full Stack Developer

Need any Help in your Project?

Sign up to our newsletter