# Unknown words

The default translation mode allows the model to produce the symbol when it is not sure of the specific target word.

Often times symbols will correspond to proper names that can be directly transposed between languages. The -replace_unk option will substitute with source words that have the highest attention weight. The -replace_unk_tagged option will do the same, but wrap the token in a ｟unk:xxxxx｠ tag.

## Phrase table¶

Alternatively, advanced users may prefer to provide a pre-constructed phrase table from an external aligner (such as fast_align) using the -phrase_table option to allow for non-identity replacement.

source|||target


## Workarounds¶

Several techniques exist to minimize the out-of-vocabulary issue:

• sub-tokenization like BPE or "wordpiece" to simulate open vocabularies
• mixed word/characters model as described in Wu et al. (2016)
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 1, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.9002346396446228, "perplexity": 4589.1880066499125}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2018-26/segments/1529267865098.25/warc/CC-MAIN-20180623152108-20180623172108-00480.warc.gz"}