Typorithm · medium

Tokenization (BPE)

tokenizationtransformersbpevocab

Saved on this device

Full lesson · 8 learning steps Open any exercise for focused practice

Step 1 of 8 · intro card

Learn Tokenization (BPE)

Tokenization & Embeddings

Byte-pair encoding (BPE) splits text into subword tokens drawn from a fixed vocabulary. Common words become a single token while rare words split into pieces, so any string is representable, and each token maps to an integer id that indexes the embedding table.

A transformer cannot read raw text — it works with integers. Tokenization converts a string into a sequence of token ids before anything else happens.

Byte-pair encoding (BPE) builds its vocabulary by starting from individual bytes or characters and repeatedly merging the most frequent adjacent pairs. The result is a fixed set of subword tokens.

Common words like "the" become a single token, while a rarer word like unhappiness splits into pieces such as un, happ, and iness. Because the pieces compose, any input — even unseen words — can be represented, and decode reverses the process exactly.

BPE merges frequent pairs into subword tokens.

Common words are one token; rare words split.

Each token maps to an id indexing the embedding table.

// byte-pair encoding merges frequent pairs
const tokens = tokenizer.encode("unhappiness");
// -> ["un", "happ", "iness"] -> ids
const text = tokenizer.decode(tokens);
const vocabSize = tokenizer.vocab.size; // ~100k

Read the implementation

Connect each part of the code to the concept before you type.

1. Text to tokens

encode splits a string into subword tokens and returns their integer ids.

const tokens = tokenizer.encode("unhappiness");

2. Subword pieces

A rare word like unhappiness breaks into reusable subword pieces, each mapped to an id.

// -> ["un", "happ", "iness"] -> ids

3. Tokens to text

decode is the inverse: it turns ids back into the original string.

const text = tokenizer.decode(tokens);

4. Vocabulary size

The vocabulary is a fixed set of tokens, often around 100k entries for modern LLMs.

const vocabSize = tokenizer.vocab.size;

5. How BPE is built

BPE builds its vocabulary by repeatedly merging the most frequent adjacent symbol pairs.

// byte-pair encoding merges frequent pairs

A complete learning loop

Understand

Read the concept map and follow the code breakdown.

Reconstruct

Type the implementation, repair mistakes and work through every recall exercise.

Verify

Use the final check without hints; revisit weak concepts in short practice sessions.