FastApi with PGMQ on Fly.io

Out of curiosity I want to explore alternatives to RabbitMQ. Todays alternative is a message queue as extension to PostgreSQL: PGMQ. To test this I deployed PGMQ to fly.io and connected a FastAPI Python app with psycopg3.

First the deployment of PGMQ. Because we want to store the data on a volume, we need a custom Dockerfile:

FROM quay.io/tembo/pgmq-pg
ENV PGDATA=/data/pgdata

and of course a fly app with a volume, the fly.toml for me looks like this:

app = "empty-dream-2474"
primary_region = "ams"

[[services]]
  internal_port = 5432
  protocol = "tcp"
  auto_stop_machines = false
  auto_start_machines = true
  min_machines_running = 1

[[mounts]]
  source = "pgmq_data"
  destination = "/data"
  processes = ["app"]

The app is on purpose not stopped when idle, because it wasn't waking up when connecting. Something so solve another day.

Finally the app needs a secret for the password, i.e.

fly secrets set POSTGRES_PASSWORD=FIXME

When deployed this app will provide a PostgreSQL service on port 5432 to be used from another app in the same account. So I deployed a second fly.io app with FastAPI connecting to the PostgreSQL instance. All apps in the same account can connect to each other internally by their region and name, i.e. the address of my PostgreSQL instance is ams.empty-dream-2474.internal.

The full code of the FastAPI + psycopg3 on fly.io example: https://gist.github.com/mfa/1977381ef7f57c7884098c7dee233295.

Some remarks to the code:

  • Using parameters from a rest api, plugging them together to json and adding them to the psycopg3-SQL is messy.

  • The init of the extension and creation of queues for PGMQ needs to run once. This should probably happen when initializing the PostgreSQL app.

  • The two rest function are send and pop to add a message to the queue and pop the oldest one. For more possibities see the PGMQ documentation

The send call returns the current length of the queue:

/images/pgmq-fastapi-send.png

The pop call returns the oldest message from the queue (I called already twice before, to get to this message):

/images/pgmq-fastapi-pop.png

As a result of this PGMQ exploration, I will probably not use this for an actual (hobby) project. I probably don't need the performance that is possible with this. And running a custom PostgreSQL with its own Dockerfile feels like too much future maintenance.

Both, database and FastAPI app, are not deployed anymore.