2025-02-14 16:08:03,171 ocrqa_create_bloom_filter.py:425 INFO: Namespace(input_files=['lex/fr/realword_ocr_errors.nw.txt', 'lex/fr/ocr_errors.nw.txt', 'lex/fr/old_spelling.rw.txt', 'lex/fr/modern_spelling.rw.txt', 'lex/fr/morph.rw.txt.bz2', 'lex/fr/single_char.rw.txt.bz2', 'lex/fr/frwiki.unigram.freq.tsv.bz2'], bloom_path='build.d/fp_prob_0.00001/ocrqa-wp_v1.0.6-fr.bloom', fp_probability=1e-05, log_level='INFO', log_file='build.d/fp_prob_0.00001/ocrqa-wp_v1.0.6-fr.bloom.log', config=None, min_frequency=2, single_char_min_frequency=20, diagnose_bloom=True)
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:226 INFO: Starting Bloom Filter creation...
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:178 INFO: Processing nonword file: lex/fr/realword_ocr_errors.nw.txt
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['gazelle']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:190 INFO: Excluded 1 words that should never be added
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:178 INFO: Processing nonword file: lex/fr/ocr_errors.nw.txt
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['bàie']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['chepin']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['dftce']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['ì']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['locamo']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['mmmi']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['mmml']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['mmmv']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['neuchat']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['neuchàtel']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['ölten']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['ponr']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['pribourg']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['remercîments']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['sadresser']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['zuberbuhler']
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:190 INFO: Excluded 17 words that should never be added
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:196 INFO: Processing real-word file: lex/fr/old_spelling.rw.txt
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:196 INFO: Processing real-word file: lex/fr/modern_spelling.rw.txt
2025-02-14 16:08:03,172 ocrqa_create_bloom_filter.py:196 INFO: Processing real-word file: lex/fr/morph.rw.txt.bz2
2025-02-14 16:08:03,856 ocrqa_create_bloom_filter.py:196 INFO: Processing real-word file: lex/fr/single_char.rw.txt.bz2
2025-02-14 16:08:03,857 ocrqa_create_bloom_filter.py:135 INFO: Processing frequency file: lex/fr/frwiki.unigram.freq.tsv.bz2
2025-02-14 16:08:08,932 ocrqa_create_bloom_filter.py:240 INFO: low_freq_excluded before removing parts from high-frequency words: 1548627
2025-02-14 16:08:09,031 ocrqa_create_bloom_filter.py:248 INFO: low_freq_excluded after removing parts from high-frequency words: 1224019
2025-02-14 16:08:09,031 ocrqa_create_bloom_filter.py:252 INFO: Lexical processing complete.
2025-02-14 16:08:09,031 ocrqa_create_bloom_filter.py:256 INFO:   - nonwords_read: 17
2025-02-14 16:08:09,031 ocrqa_create_bloom_filter.py:256 INFO:   - nonwords_count: 17
2025-02-14 16:08:09,031 ocrqa_create_bloom_filter.py:256 INFO:   - realwords_read: 723486
2025-02-14 16:08:09,031 ocrqa_create_bloom_filter.py:256 INFO:   - realwords_accepted: 737153
2025-02-14 16:08:09,031 ocrqa_create_bloom_filter.py:256 INFO:   - realwords_nonwords_filtered: 1
2025-02-14 16:08:09,031 ocrqa_create_bloom_filter.py:256 INFO:   - freq_words_read: 4047539
2025-02-14 16:08:09,031 ocrqa_create_bloom_filter.py:256 INFO:   - words_accepted: 1992185
2025-02-14 16:08:09,032 ocrqa_create_bloom_filter.py:256 INFO:   - subwords_accepted: 2385659
2025-02-14 16:08:09,032 ocrqa_create_bloom_filter.py:256 INFO:   - subwords_filtered: 9
2025-02-14 16:08:09,032 ocrqa_create_bloom_filter.py:256 INFO:   - low_freq_excluded: 1224019
2025-02-14 16:08:09,032 ocrqa_create_bloom_filter.py:256 INFO:   - single_char_words_filtered: 6597
2025-02-14 16:08:09,032 ocrqa_create_bloom_filter.py:256 INFO:   - freq_words_filtered: 2048757
2025-02-14 16:08:09,032 ocrqa_create_bloom_filter.py:259 INFO: Estimated word count: 1937683
2025-02-14 16:08:09,562 ocrqa_create_bloom_filter.py:263 INFO: Bloom Filter created and saved to build.d/fp_prob_0.00001/ocrqa-wp_v1.0.6-fr.bloom
2025-02-14 16:08:09,924 ocrqa_create_bloom_filter.py:285 INFO: Diagnosis Results:
2025-02-14 16:08:09,924 ocrqa_create_bloom_filter.py:286 INFO:   - Excluded words in bloom filter: 0
2025-02-14 16:08:09,924 ocrqa_create_bloom_filter.py:287 INFO:   - Known words not in bloom filter: 0
2025-02-14 16:08:10,101 ocrqa_create_bloom_filter.py:294 INFO:   - Low-frequency words in bloom filter: 15
2025-02-14 16:08:10,101 ocrqa_create_bloom_filter.py:300 INFO:   - Proportion of excluded words in bloom filter: 0.00000000
2025-02-14 16:08:10,101 ocrqa_create_bloom_filter.py:306 INFO:   - Proportion of known words not in bloom filter: 0.00000000
2025-02-14 16:08:10,101 ocrqa_create_bloom_filter.py:314 INFO:   - Proportion of low-frequency words in bloom filter: 0.00001225