Shrink a lot of JPGs
The images from the previous post are not using any JPEG compression.
I fiddled a bit with the compression values of raspistill
but the scale seem to be not aligned with the JPEG standards.
So I use raspistill without any compression and compress the images later with ImageMagick.
First iteration
This script converts all images into a new folder by keeping my folder structure (year/month):
#!/bin/bash # example input # images/2017/12/1512699901.jpg # example output # images.smaller/2017/12/1512699901.jpg # input files files=$(find images -name \*jpg) for fn in $files; do echo "$fn" # target folder name target=images.smaller/$(dirname "$fn" | cut -d"/" -f2,3) # generate folder if needed mkdir -p $target # actual image processing; shrinking the image magick -quality 90 $fn $target/$(basename $fn) # add progressbar; counting echos until length of files list processed done | pv -l -s "${#files}" > /dev/null
Because I have more than 50k images a progressbar using pv
was added.
This takes quite some time, because it processes only 10 images per second.
Clearly not fast enough.
Second iteration
Using GNU parallel.
The default is to run the given bash function with as many threads as cores in your computer.
I wanted to limit this, so I set it to 8
.
There is a lot of magic possible for filename replacement and processing, but because I needed a bash function for the mkdir
I didn't care to read up on a more parallel native solution.
#!/bin/bash convert_func() { target=images.smaller/$(dirname "$1" | cut -d"/" -f2,3) mkdir -p $target magick -quality 90 $1 $target/$(basename $1) } export -f convert_func find images -name \*jpg | parallel --progress -j 8 convert_func
This runs a lot faster than the first iteration and took about 5 minutes to process all 50k images.