Shrinking a lot of JPGs

The images from the previous post are not using any JPEG compression. I fiddled a bit with the compression values of raspistill but the scale seem to be not aligned with the JPEG standards. So I use raspistill without any compression and compress the images later with ImageMagick.

First iteration

This script converts all images into a new folder by keeping my folder structure (year/month):

#!/bin/bash

# example input
# images/2017/12/1512699901.jpg

# example output
# images.smaller/2017/12/1512699901.jpg

# input files
files=$(find images -name \*jpg)

for fn in $files;
do
    echo "$fn"
    # target folder name
    target=images.smaller/$(dirname "$fn" | cut -d"/" -f2,3)
    # generate folder if needed
    mkdir -p $target
    # actual image processing; shrinking the image
    magick -quality 90 $fn $target/$(basename $fn)
# add progressbar; counting echos until length of files list processed
done | pv -l -s "${#files}" > /dev/null

Because I have more than 50k images a progressbar using pv was added. This takes quite some time. Single threaded about 10 images per second are converted. Clearly not fast enough.

Second iteration

Using GNU parallel.

The default is to run the given bash function with as many threads as cores in your computer. I wanted to limit this, so I set it to 8. There is a lot of magic possible for filename replacement and processing, but because I needed a bash function for the mkdir, I didn't care to read up on a more parallel native solution. Everything is happening in the bash function.

#!/bin/bash

convert_func() {
    target=images.smaller/$(dirname "$1" | cut -d"/" -f2,3)
    mkdir -p $target
    magick -quality 90 $1 $target/$(basename $1)
}
export -f convert_func

find images -name \*jpg | parallel --progress -j 8 convert_func

This runs a lot faster than the first iteration by maintaining 8 concurrent magick calls at the same time. The result is that this needed about 20 minutes to process all 50k images.