Sensor API with Django

Disclaimer: This post is not a Django tutorial. The Django documentation has a really good one.

About 9 years ago we build an API to save particulates sensor data using Django and PostgreSQL. I was asked to build something similar, but this time with a few learnings from the previous iteration. Back then we never thought this will get that big, so the database table structure couldn't cope with the load later. Today I would chose a way more free data scheme with a lot less joins needed. This may require (async) postprocessing for displaying on a map, but removes joins on the database. For authentication the ESP internal id together with an (optional) pin was used. This limits to Espressif devices and may not be unique when using other manufacturers or microcontrollers. I plan on generating a ulid for every Sensor and not Node level. For authentication I will generate a random string per Sensor. There will be multiple secrets allowed and a comment field to make it easier to replace the secret later.

But on the other hand I would do a lot of things the same way: Django is still the most boring choice. PostgreSQL (or maybe timescaledb) is still my choice for the database. The hosting will be on fly.io and the PostgreSQL instance managed by supabase. This should be enough for quite a while and gives always the option to scale either with fly/supabase or move everthing to a Hetzner VPS and pay with your spare time instead of money.

This (probably long) post will describe the initial setup of the Django service at fly.io with using Supabase for PostgreSQL and Mailgun for sending emails (i.e. password reset; or later password-less login).

First step is to setup a Django project. This is already well documented for example in the fly.io docs I followed this steps and deployed a first version using SQLite as database. Without a volume this database will be replaced on every deployment, so this clearly needs to be solved. For this I don't want to add a volume and start with SQLite, because I don't want to migrate the SQLite database to PostgreSQL later, so the next step is adding Supabase.

The SECRET_KEY in settings should never be in the repository, so we deploy a new one with fly secrets set SECERT_KEY=insert-some-secret-value-here and use it in settings.py. For static files (i.e. the css/js of the admin) I added whitenoise. We use a start.sh script as CMD at the end of the Dockerfile to start gunicorn. Before this we add the collectstatic command to prepare the static files.

To get a PostgreSQL database we create a new project in the Supabase dashboard. The password set on create is the password used to log into the database, so choose a long one! Additionally we need the HOST and the USER, which can be found in the api settings, i.e. https://supabase.com/dashboard/project/YOUR_PROJECT_ID/settings/database. On this database settings page we additionally need to download the client certificate and enable SSL enforcing. Before we can use the Host, Password and User we need to give them to fly.io by setting the secrets, i.e. fly secrets set SUPABASE_HOST=aws-0-eu-central-1.pooler.supabase.com. The same for SUPABASE_PASSWORD and SUPABASE_USER.

In the Django settings file we set the database like this:

DATABASES = {
    'default': {
        'ENGINE' : 'django.db.backends.postgresql',
        'NAME' : 'postgres',
        'HOST' : os.environ.get('SUPABASE_HOST'),
        'PASSWORD': os.environ.get('SUPABASE_PASSWORD'),
        'PORT': 5432,
        'USER': os.environ.get('SUPABASE_USER'),
        # from supabase: database/settings
        'CERT' : 'config.prod-ca-2021.crt',
    }
}

After adding the migrate command to the start.sh, the same way we did for collectstatic, the database at Supabase looks like this:

img1

FIXME:

datamodel:

  • sensor: owner, ulid, created, lat, lon, meta (json free to retreive for configuration)

  • sensor_secret: sensor, secret, comment, created

one sensor example

{
   "id": "01HR2V9YM3W8GR67YE4JRGXHX3",
   "owner": "<user_id>",
   "description": "...",
   "meta": {
     "hardware": {
       "name": "adafruit ultimate gps breakout",
       "antenna": {
         "name": "na",
         "active": true
        }
      },
      "serial": {
        "port": "/dev/ttyAMA0",
        "speed": 9600
      }
   },
   "lat": 90.0,
   "lon": 360.0,
   "alt": 123.4
 }

blog post TODOs:

  • add a lot of links, i.e. fly/supabase/mailgun/django/...

  • add a screenshot