My fellow Americans.  
Last year COVID-19 kept us apart.
This year we are finally together again.
Tonight, we meet as Democrats Republicans and Independents.
But most importantly as Americans.
With a duty to one another to the American people to the Constitution. 
And with an unwavering resolve that freedom will always triumph over tyranny.
Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.
But he badly miscalculated.
He thought he could roll into Ukraine and the world would roll over.
Instead he met a wall of strength he never imagined.
He met the Ukrainian people.
From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.
Groups of citizens blocking tanks with their bodies.
Token Text Splitter#
How the text is split: by tiktoken tokens
How the chunk size is measured: by tiktoken tokens
from langchain.text_splitter import TokenTextSplitter
text_splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=0)
texts = text_splitter.split_text(state_of_the_union)
print(texts[0])
Madam Speaker, Madam Vice President, our
previous
Hypothetical Document Embeddings
next
VectorStores
 Contents
  
Generic Recursive Text Splitting
Markdown Text Splitter
Latex Text Splitter
Python Code Text Splitter
Character Text Splitting
HuggingFace Length Function
tiktoken (OpenAI) Length Function
NLTK Text Splitter
Spacy Text Splitter
Token Text Splitter
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on Mar 22, 2023.