GPU performance is about 50% slower than benchmarks
My guess is that the reduction in performance is due to differences in versions PyTorch. The published benchmark uses torch==0.4.1 (even though the README states something different). You are using PyTorch 1.0.1.

PyTorch 1.0 speed is sometimes slower than lower than 0.4. PyTorch 1.0 can be speed-up by adding torch.backends.cudnn.benchmark = True and compiling from source.

Often times published benchmark are heavily tuned from the "off-the-shelf" version.