Emacs Literate Config

A few months ago I declared Emacs bankruptcy (again) and started a new Emacs config setup. This time using org files to document/structure the config. I adapted the version from Jeroen Faijdherbe which was a very good starting point.

Of course I started the new setup by using Emacs 29 using the ArchLinux aur for emacs29-git. Yesterday Emacs 29.1 was released, so I switched to the official emacs package in ArchLinux (actually emacs-wayland).

Why use Emacs 29+ only? Especially the use-package syntax to install and update packages is so much better than everything before. Another feature I started to use a lot is eglot. Installing this for older Emacs versions was of course possible but with Emacs 29 it's included.

My very opinionated setup of Emacs: https://github.com/mfa/emacs-config.

Calculate personal count for a bicycle counter

The bicycle counter at the "cycle highway" between Stuttgart-Rohr and Böblingen reached over 1 million cyclists counted. The counting started at 2019-03-30 and I am interested how many times I was counted when cycling there.

First we need the polygon covering the track near the counter (in GeoJSON format):

{"type": "Feature",
  "properties": {},
  "geometry": {
    "type": "Polygon",
    "coordinates": [[
        [9.081402, 48.711934],
        [9.082003, 48.711683],
        [9.081429, 48.711134],
        [9.080785, 48.711339],
        [9.081402, 48.711934]
      ]]}}

The polygon will be loaded into SQLite and used to calculate if my tours intersected with it. To simplify the problem every tour only counts once, even if I crossed the counter more than one time. The following code loads all "fit" files in the folder "tours" (which contains my tours from 2019-05-30 till today); chunks them in lists of 100 dataframes and intersects them with the polygon using Spatialite. The SQLite database is only used to calculate the intersection, so it is only in memory. The libraries required are sqlite-utils, shapely and fitdecode.

import json
import warnings
from itertools import zip_longest
from pathlib import Path
from typing import Iterator

import fitdecode
from shapely.geometry import shape
from sqlite_utils.db import Database


def get_data(fn: Path) -> Iterator[dict]:
    with fitdecode.FitReader(
        fn, processor=fitdecode.StandardUnitsDataProcessor()
    ) as fit:
        for frame in fit:
            if frame.frame_type == fitdecode.FIT_FRAME_DATA:
                data = {i.name: i.value for i in frame.fields}
                # only data frames with lat/lon
                if "position_lat" in data:
                    yield data


def process_file(db: Database, fn: Path) -> Iterator[int]:
    # process 100 dataframes at once to speed up
    for chunk in zip_longest(*[get_data(fn)] * 100):
        # get lat/lon and filter empty values
        coords = [
            [c["position_long"], c["position_lat"]]
            for c in chunk
            if c and c["position_long"]
        ]
        # only use coords with a few values
        if len(coords) > 2:
            r = db.query(
                "select Intersects(GeomFromText(?), Envelope(calc.geometry)) as x from calc",
                [shape({"type": "LineString", "coordinates": coords}).wkt],
            )
            # [{'x': 1}] -> 1; [{'x': 0}] -> 0
            yield list(r)[0]["x"]


db = Database(memory=True)
db.init_spatialite("/usr/lib/mod_spatialite.so")  # path to library file

# table to calculate the intersections
table = db["calc"]
table.insert(
    {
        "id": 1,
        "geojson": {
            "type": "Polygon",
            "coordinates": [
                [
                    [9.081402, 48.711934],
                    [9.082003, 48.711683],
                    [9.081429, 48.711134],
                    [9.080785, 48.711339],
                    [9.081402, 48.711934],
                ]
            ],
        },
    },
    pk="id",
)

# add a geometry column and fill with geojson polygon
table.add_geometry_column(f"geometry", "POLYGON")
for row in table.rows:
    db.execute(
        f"update calc set geometry = GeomFromText(?, 4326) where id=?",
        [shape(json.loads(row["geojson"])).wkt, row["id"]],
    )

# disable warnings from fitdecode
warnings.simplefilter("ignore")

counter = 0
files = sorted(Path("tours").glob("*.fit"))
for index, fn in enumerate(files, start=1):
    result = sum(process_file(db, fn))
    # only count once, even if sum() > 1
    counter += 1 if result else 0
    print(
        f"{index:>4}/{len(files):>4} -- {fn.name:>50} -- {result} -- {counter:>3}"
    )

print(f"crossed the counter {counter} times")

The last lines of the output:

1579/1581 --       2023-07-29-081454-ELEMNT BOLT 105D-708-0.fit -- 0 -- 366
1580/1581 --       2023-07-29-151400-ELEMNT BOLT 105D-709-0.fit -- 1 -- 367
1581/1581 --       2023-07-30-083422-ELEMNT BOLT 105D-710-0.fit -- 1 -- 368
crossed the counter 368 times

Of the over 1 million cyclists counted, I contributed 368 counts.

Decoding FIT files

My bicycle computer (a 1st gen Wahoo ELEMNT Bolt) is recording FIT files. I was interested how much is stored to maybe do something with it. The easiest way I found to export the data is by using the fitdecode Python library. The library already has a ready-to-use CLI command which does everything I need:

fitjson --pretty -o 20190219_094908.json 2019-02-19-094908-ELEMNT\ BOLT\ 8284-15-0.fit

A lot of meta information is in there, but I am mainly interested in the record data frames, i.e.:

{
  "frame_type": "data_message",
  "name": "record",
  "header": {
    "local_mesg_num": 0,
    "time_offset": null,
    "is_developer_data": false
  },
  "fields": [
    {
      "name": "timestamp",
      "value": "2019-02-19T09:53:34+00:00",
      "units": "",
      "def_num": 253,
      "raw_value": 919504414
    },
    {
      "name": "position_lat",
      "value": 48.733650958165526,
      "units": "deg",
      "def_num": 0,
      "raw_value": 581415103
    },
    {
      "name": "position_long",
      "value": 9.120587892830372,
      "units": "deg",
      "def_num": 1,
      "raw_value": 108812852
    },
    {
      "name": "gps_accuracy",
      "value": 2,
      "units": "m",
      "def_num": 31,
      "raw_value": 2
    },
    {
      "name": "enhanced_altitude",
      "value": 410.20000000000005,
      "units": "m",
      "def_num": 78,
      "raw_value": 410.20000000000005
    },
    {
      "name": "altitude",
      "value": 410.20000000000005,
      "units": "m",
      "def_num": 2,
      "raw_value": 4551
    },
    {
      "name": "grade",
      "value": -5.26,
      "units": "%",
      "def_num": 9,
      "raw_value": -526
    },
    {
      "name": "distance",
      "value": 1.5201300000000002,
      "units": "km",
      "def_num": 5,
      "raw_value": 152013
    },
    {
      "name": "heart_rate",
      "value": 106,
      "units": "bpm",
      "def_num": 3,
      "raw_value": 106
    },
    {
      "name": "cadence",
      "value": 0,
      "units": "rpm",
      "def_num": 4,
      "raw_value": 0
    },
    {
      "name": "enhanced_speed",
      "value": 51.9912,
      "units": "km/h",
      "def_num": 73,
      "raw_value": 14.442
    },
    {
      "name": "speed",
      "value": 51.9912,
      "units": "km/h",
      "def_num": 6,
      "raw_value": 14442
    },
    {
      "name": "temperature",
      "value": 12,
      "units": "C",
      "def_num": 13,
      "raw_value": 12
    }
  ],
  "chunk": {
    "index": 346,
    "offset": 8054,
    "size": 27
  }
},

The position is the road downhill from Vaihingen to Kaltental and was probably on my way to work that day. The speed with 51km/h is at that point realistic and 12°C for January sounds good too. Overall this is pretty nice and easy to use.

Every time the data message format changes (i.e. values are added or removed) there is a "definition message". For example there is a "definition message" announcing that the next message is about temperature and battery. Or a following message is about the device (the Wahoo) with serial number, software version and the battery charing state. Or a message about the heartrate sensor device information. Always first a "definition message" and then one or more "data message"s. A bit verbose with a lot of information, but the format is in binary format and seems pretty optimized (35K fit file vs. 2.2M (prettified) json file).

Overall this looks like less trouble than parsing XML based GPX files.