Contextual filter by taxonomy term not using language
Temporary solution for was to create patch file to override views module functionality a little bit. Patch can be found under issue on drupal.org
https://www.drupal.org/node/2466001#comment-9795191.