Need help finding keysize for Vigenere cipher
For the explanation to why this approach works I found an excellent answer here: repeating-key xor and hamming distance
And I managed to finally find the correct keysize using the following pseudocode:
dictionary = () // dict or hashmap which stores &lt;Keysize, Score&gt; as &lt;K,V&gt; pair
for keysize in 2..max_keysize:
    n = 0, score = 0
    while True:
         if 2*keysize &gt;= bytes.len():
             break
         a = bytes[:keysize]
         b = bytes[keysize:2*keysize]
         score += (hamming_distance(a,b) / (8*keysize))
         n += 1
    averaged_score = score / n
    dictionary.add(keysize, averaged_score)

And finally you select keyisze with lowest averaged_score