Skip to content
Owais KhanSoftware Reviews

Markdown Text Chunker for RAG

Paste any Markdown document and this tool splits it into token-bounded, overlap-aware chunks ready for vector embeddings. Each chunk carries its full heading breadcrumb so your retrieval model always knows where in the document the text came from — the core idea behind aheading aware markdown chunker.

Everything runs locally: what you paste never leaves your browser.

Chunk your Markdown for RAG

Chunks5 chunks
  1. Chunk 1~43 tokens

    Introduction to Vector Databases

    Introduction to Vector Databases

Vector databases store high-dimensional embeddings and enable semantic search at scale. They are a core component of modern RAG pipelines.
  2. Chunk 2~60 tokens

    Introduction to Vector Databases > How Embeddings Work

    Introduction to Vector Databases > How Embeddings Work

An embedding model converts text into a dense numeric vector. Similar texts produce vectors that are close together in the embedding space, which is what makes semantic search possible.
  3. Chunk 3~64 tokens

    Introduction to Vector Databases > How Embeddings Work > Chunking Strategy

    Introduction to Vector Databases > How Embeddings Work > Chunking Strategy

Before embedding, long documents must be split into smaller chunks. Each chunk should be semantically coherent and carry enough context for the retrieval model to rank it correctly.
  4. Chunk 4~62 tokens

    Introduction to Vector Databases > Retrieval-Augmented Generation

    Introduction to Vector Databases > Retrieval-Augmented Generation

RAG combines a retrieval step with a generative model. The retriever fetches the most relevant chunks from the vector store, and the generator conditions its output on those chunks.
  5. Chunk 5~66 tokens

    Introduction to Vector Databases > Retrieval-Augmented Generation > Why Chunk Size Matters

    Introduction to Vector Databases > Retrieval-Augmented Generation > Why Chunk Size Matters

Chunks that are too large dilute the embedding signal. Chunks that are too small lose context. A target of 256–512 tokens balances retrieval precision with context density.

Document chunking for RAG: why structure matters

Naive text splitters cut on character count alone, which means a chunk can start mid-sentence inside a subsection with no indication of where it came from. When that chunk is retrieved, the language model has no anchor — it cannot tell whether "chunking strategy" refers to database compression, video encoding, or vector search. This tool implementsdocument chunking for RAG the structural way: it parses every heading up to your chosen depth, builds a breadcrumb likeIntroduction > How Embeddings Work > Chunking Strategy, and prepends it to every chunk that falls under that heading.

The result is that each chunk is self-describing. An embedding model encodes both the local prose and its document position, so a retriever can distinguish two sections with similar wording but different roles. This is the key insight behind markdown chunking langchain-style pipelines, where MarkdownHeaderTextSplitter does the same job programmatically. This tool gives you the same output interactively, without writing any code.

How the chunker works

The pipeline has three stages. First, the Markdown is parsed into heading-bounded sections. Any heading at or above your chosen depth becomes a split point; deeper headings are treated as body text. Each section inherits the breadcrumb of all its ancestor headings.

Second, each section's body is split into sentences on punctuation boundaries. Sentence splitting is the right unit for a semantic markdown chunker online because it avoids cutting mid-thought: a sentence is the smallest unit that carries a complete idea, which is also the smallest unit an embedding model can represent faithfully.

Third, sentences are packed greedily into token-bounded chunks. The token budget uses the cl100k_base approximation of four characters per token — the same heuristic OpenAI documents for GPT-4. When a chunk is full, the next one starts, optionally re-appending the last few sentences of the previous chunk as overlap. Overlap gives embedding models continuity across boundaries, which reduces the retrieval gap that would otherwise appear at every chunk edge.

Using the output as a markdown text splitter for embeddings

The JSON export follows the OpenAI Embeddings batch format: an array of objects each with atext field and a metadata object. You can pipe that array directly into openai.embeddings.create or into any vector store that accepts pre-chunked documents. The CSV export is useful for inspection in a spreadsheet or for ingestion into pipelines that prefer tabular input. Both formats include the headers breadcrumb and the estimated tokenCount so downstream code can filter or re-chunk without re-parsing the Markdown.

If you are building a markdown text splitter for embeddings in Python, the JSON output can be loaded with json.load and passed to any embeddings client. The breadcrumb in metadata.headers is ready to store as a filterable attribute in Pinecone, Weaviate, Qdrant, or Chroma.

Frequently asked questions

How does heading hierarchy retention improve RAG retrieval accuracy?
When a chunk carries its full breadcrumb — for example "Introduction > How Embeddings Work > Chunking Strategy" — the embedding model encodes both the local content and its document position. Retrievers can then distinguish two sections with similar wording but different roles, reducing false positives. Without the breadcrumb, a chunk about "chunking strategy" in a database guide looks identical to one in a compression guide, and the wrong document surfaces. This heading-aware markdown chunker prepends the breadcrumb to every chunk so that context travels with the text through the entire pipeline.
What token counter tokenizer is used to calculate chunk limits (e.g., cl100k_base / GPT-4)?
The tool uses the standard cl100k_base approximation: Math.round(characters / 4). OpenAI documents this as the rule of thumb for English prose with the tokenizer used by GPT-3.5-Turbo and GPT-4. It is an estimate rather than an exact count, but it is accurate enough for chunk-size budgeting and avoids shipping a full tokenizer vocabulary to the browser. If you need exact counts you can supply a custom countTokens function via the JavaScript API.
Can I export the chunked Markdown into vector-ready JSON, JSONL, or CSV format with metadata?
Yes. The tool offers two export formats. JSON output follows the OpenAI Embeddings format: an array of objects each with a "text" field and a "metadata" object containing "headers" (the breadcrumb) and "tokenCount". CSV output has three columns — text, headers, tokenCount — and is RFC 4180 compliant with double-quote escaping. Both formats are generated client-side and can be copied or downloaded. JSONL can be produced by splitting the JSON array one object per line, which most vector-store ingestion scripts accept directly.
Is my Markdown text safe and kept entirely private in the browser?
Yes. The entire chunking pipeline runs as inline JavaScript inside this page. Your Markdown is parsed, split, and packed without any network activity. No analytics script, no telemetry endpoint, and no background worker sends data anywhere. The site's automated test suite scans the built HTML for every browser API capable of transmitting data and fails the build if it finds one, so the privacy guarantee is enforced structurally rather than just promised in copy.
Is anything I paste uploaded or processed on a server?
Nothing is uploaded. The page ships the complete logic as a minified inline script, so chunking happens entirely inside your browser tab. There is no API call, no form submission, and no server-side processing step. You can disconnect from the internet after the page loads and the tool continues to work identically. This design also means the tool works with confidential internal documentation that must not leave your organisation's devices.