class_docstr: Wikitext-103, token-level, with examples up to 4,096 tokens long.
code: def generate_samples(self, data_dir, tmp_dir, dataset_split):
      samples_by_line = super(
          LanguagemodelWikitext103L4k, self
      ).generate_samples(data_dir, tmp_dir, dataset_split)
  
      def _generate_samples():
          tokens = []
          for sample in samples_by_line:
              sample_tokens = sample["targets"].split()
              if (
                  len(tokens) + len(sample_tokens)
                  < self.sequence_length
              ):
                  tokens.extend(sample_tokens)
              else:
                  yield {"targets": " ".join(tokens)}
                  tokens = sample_tokens
  
      return _generate_samples()
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import collections
  import os
  import zipfile
  from tensor2tensor.data_generators import generator_utils
  from tensor2tensor.data_generators import problem
  from tensor2tensor.data_generators import text_encoder
  from tensor2tensor.data_generators import text_problems
  from tensor2tensor.utils import registry
  import tensorflow.compat.v1
label_desc: Use image node_id 1 for calling the LanguagemodelWikitext103L4k obj's underlying member method code with example usage: obj.generate_samples(data_dir, tmp_dir, dataset_split) and returns: _generate_samples