Text Chunker for RAG
Split long documents into chunks for embeddings and retrieval-augmented generation: by size in tokens or characters, with overlap, and along paragraph, sentence or Markdown heading boundaries. The result is a JSON array of chunks with their sizes.
Frequently asked questions
Which splitting modes are there?
Recursive splits at paragraphs and, for long paragraphs, at sentences. Sentences keeps whole sentences together. Headings starts new sections at Markdown headings. Fixed cuts the text into equal pieces regardless of structure.
How is the size measured?
In tokens with the o200k_base tokenizer used by current OpenAI models, or in characters. Pieces longer than the size are cut at the limit.
What does overlap do?
Each chunk repeats the end of the previous one, up to the overlap size, so context that spans a boundary is not lost. 10% to 20% of the chunk size is a common choice.
Which chunk size should I use?
It depends on the embedding model and the questions. 200 to 800 tokens is a common range: smaller chunks retrieve precise passages, larger ones keep more context.
What does the output look like?
A JSON array where each chunk has its index, its size in the chosen unit and its text, ready to send to an embedding API.
How long can the text be?
Up to 2 million characters, and at most 5,000 chunks per run.