title: BERT tokenize URLs
body: <p>I want to classify a bunch of tweets and therefore I'm using the huggingface implementation of BERT. However I noticed that the deafult BertTokenizer does not use special tokens for urls.</p>
  <pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; from transformers import BertTokenizer
  &gt;&gt;&gt; tokenizer = BertTokenizer.from_pretrained(&quot;bert-base-cased&quot;)
  &gt;&gt;&gt; tokenizer.tokenize(&quot;https://stackoverflow.com/questions/ask&quot;)
  ['https', ':', '/', '/', 'stack', '##over', '##flow', '.', 'com', '/', 'questions', '/', 'ask']
  </code></pre>
  <p>This seems quite inefficent to me. What would be the best way, to encode URLs?</p>