Progressbars in Bash

In a bash script I needed a progressbar for a "for"-loop.

The best solution was to use pv.

photo_count=`find . -name \*JPG | wc -l`
for photo in *.JPG; do
    do_some_stuff $photo
    echo -n "."
done | pv -pte -i0.1 -s${photo_count} > /dev/null
The count is necessary to get a real progressbar and an ETA.
How does it work: pv is counting the characters (here '.') until size (-s) is reached.

The results looks like:

0:00:03 [==========>                              ] 30% ETA 0:00:10

Run AllenNLP on AWS P2 with ArchLinux

Quest: Run AllenNLP with GPU support on an AWS p2/p3 instances with ArchLinux.

Steps:

  1. Go to https://www.uplinklabs.net/projects/arch-linux-on-ec2/
  2. Start a new EC2 instance by clicking on the "ebs hvm x86_64 lts"-link for your prefered region.
  3. Choose p2.xlarge as instance type (all other p2 and p3 will work the same)
  4. Configure the stuff you need (allow ssh, storage, ...)
  5. Launch instance (save the pem-certificate!)
  6. Get the IP address from the ec2 instances view
  7. Connect to server
ssh -i certificate.pem root@<IP-ADDRESS>

Yay! We have a shell on archlinux.

  1. First you may have to repopulate the archlinux-keyring
pacman-key --init
  1. Next update the already installed packages
pacman --noconfirm -Syu
  1. Install a few packages (we need nvidia-dkms and not nvidia because the kernel is a custom one and not default archlinux)
pacman --noconfirm -S git cudnn cuda python-virtualenvwrapper nvidia-dkms nvidia-utils htop tmux
  1. Reboot server to load new kernel with nvidia module
reboot
  1. After the reboot, connect to server again
ssh -i certificate.pem root@<IP-ADDRESS>
  1. Test if nvidia module is loaded
nvidia-smi
  1. Add virtualenvwrapper to your shell
source /usr/bin/virtualenvwrapper.sh
  1. And create a virtualenv
mkvirtualenv testing
  1. Install allennlp 
pip install allennlp
  1. And test if PyTorch/AllenNLP finds the gpu
python -c "import torch; print('cuda:', torch.cuda.is_available())"
  1. (Optional) run the AllenNLP testsuite
allennlp test-install

Now you can use AllenNLP with GPU on AWS!

AllenNLP: Machine Translation using configuration

I got inspired by this blogpost: http://www.realworldnlpbook.com/blog/building-seq2seq-machine-translation-models-using-allennlp.html

My goal was to train my own model with the language pair English to German. Additionally I wanted to use AllenNLP json configurations.

Data

First, fetch the language pair you want. See the realworldnlpbook.com blog post linked at the top. For this example I fetched ENG--DEU. The files should be stored in data.

Configuration

This configuration is pretty close to the python code from the original blogpost:

{
  "dataset_reader": {
    "type": "seq2seq",
    "source_tokenizer": {
      "type": "word"
    },
    "target_tokenizer": {
      "type": "character"
    },
    "source_token_indexers": {
      "tokens": {
        "type": "single_id"
      }
    },
    "target_token_indexers": {
      "tokens": {
        "type": "single_id",
        "namespace": "target_tokens"
      }
    }
  },
  "train_data_path": "data/tatoeba.eng_deu.train.tsv",
  "validation_data_path": "data/tatoeba.eng_deu.dev.tsv",
  "test_data_path": "data/tatoeba.eng_deu.test.tsv",
  "evaluate_on_test": true,
  "model": {
    "type": "simple_seq2seq",
    "source_embedder": {
      "type": "basic",
      "token_embedders": {
        "tokens": {
          "type": "embedding",
          "embedding_dim": 256
        }
      }
    },
    "encoder": {
      "type": "stacked_self_attention",
      "input_dim": 256,
      "hidden_dim": 256,
      "projection_dim": 128,
      "feedforward_hidden_dim": 128,
      "num_layers": 1,
      "num_attention_heads": 8
    },
    "max_decoding_steps": 20,
    "attention": {
      "type": "dot_product"
    },
    "beam_size": 8,
    "target_namespace": "target_tokens",
    "target_embedding_dim": 256
  },
  "iterator": {
    "type": "bucket",
    "batch_size": 32,
    "sorting_keys": [["source_tokens", "num_tokens"]]
  },
  "trainer": {
    "optimizer": {
      "type": "adam"
    },
    "patience": 10,
    "num_epochs": 100,
    "cuda_device": 0
  }
}

Change cuda_device to -1 if you have no GPU. But beware that the training will take a lot longer without a GPU.

The changes for another attention (as described in the original blog post) are for example:

"attention": {
    "type": "linear",
    "tensor_1_dim": 256,
    "tensor_2_dim": 256,
    "activation": "tanh"
},

instead of

"attention": {
    "type": "dot_product"
},

How to train

allennlp train mt_eng_deu.json -s output

At the end of the training there will be a model.tar.gz in the output folder

How to evaluate

run evaluate with a trained model:

allennlp evaluate model.tar.gz data/tatoeba.eng_deu.test.tsv

Predict one sentence

generate one sentence to predict:

cat <<EOF > inputs.txt
{"source": "Let's try something."}
EOF

run predict with a trained model:

allennlp predict model.tar.gz inputs.txt

expected (as single characters!): "Lass uns etwas versuchen!"