title: Append a list until a size limit is reached then start a new list
body: <p>I have a list of texts and I need to split each text in multiple chunks each shorter than a limit of 5000 bytes.</p>
  
  <p>The idea is to split each text in sentences and then add them back one by one until the limit of 5000 is reached.</p>
  
  <p>This is how far I got (see the code). I'm def doing something wrong. But I spent too much time debugging it so will really appreciate a pair of fresh eyes. Thanks!</p>
  
  <p>To test it you need can use any text larger than 10k bytes.</p>
  
  <pre><code>import nltk
  nltk.download('punkt')
  from nltk.tokenize import sent_tokenize
  
  def split_text(text, limit):
      sentences = sent_tokenize(text)
      def get_chunk(sentences, limit):
          results = []
          counter = 0
          while counter &lt; limit:
              for s in sentences:
                  counter += len(s.encode('utf-8'))
                  results.append(s)
                  sentences.remove(s)
          return results, sentences
  
      out = []
      while len(' '.join(sentences).encode('utf-8')) &gt; limit:
          results, sentences = get_chunk(sentences, limit)
          out.append(results)
      else:
          out.append(sentences)
      text_out = [' '.join(sentences) for sentences in out]
      return text_out
  </code></pre>