'Splitting by "<" instead of " " seems to make more sense:
* Lower run time
* Lower fail rate
tokenized_set fail rate: 0.115763546798
Total time: 2.94191193581s
Analyzed samples: 1218
There is a reason for this... "<" is less common than " " in the
response bodies:
[(\' \', 21827575),
(\'e\', 7560435),
(\'a\', 7394690),
(\'t\', 7115405),
(\'i\', 6285118),
(\'s\', 5746931),
(\'"\', 5272311),
(\'o\', 5011207),
(\'l\', 4831219),
(\'n\', 4732762),
(\'r\', 4701036),
(\'/\', 4417755),
(\'c\', 4003959),
(\'d\', 3420064),
(\'p\', 3269927),
(\'>\', 3258488),
(\'<\', 3255475),
(\'=\', 2858727),
(\' \', 2849400),
(\'h\', 2766599)]
So when we split by a less common char, we\'re "moving towards" the
trivial string equal comparison. As an example, splitting by "<dnfslsk"
would return 0% fail rate.'

def test_tokenized_set_split_tag(self):

    def tokenized_set_split_tag(str_a, str_b, ratio):
        set_a = set(str_a.split('<dnfslsk'))
        set_b = set(str_b.split('<dnfslsk'))
        maxl = max(len(set_a), len(set_b))
        return ((len(set_a.intersection(set_b)) / maxl) > ratio)
    self.generic_fuzzy_string_diff_runner_against_404(tokenized_set_split_tag, IS_EQUAL_RATIO)
