Typorithm · medium

Embedding Layer

embedding-layertransformersembeddingd-modellookup

Saved on this device

Full lesson · 8 learning steps Open any exercise for focused practice

Step 1 of 8 · intro card

Learn Embedding Layer

Tokenization & Embeddings

The embedding layer is a lookup table that maps each token id to a dense vector of size d_model. Positional information is added to that vector, and the whole table is a set of parameters learned during training.

Once text is token ids, the embedding layer turns each id into a vector. It is simply a big matrix with one row per vocabulary token; the id selects a row.

Each row is a dense vector of length d_model — the model's hidden width, often around 4096 for large models. This vector is the model's learned representation of that token.

Before the first transformer block, positional information is added to the token embedding so the model knows where each token sits. The embedding matrix itself is parameters, tuned by gradient descent during training.

Embedding = lookup table indexed by token id.

Each vector has length d_model (e.g. ~4096).

The matrix is learned during training.

// each token id maps to a learned vector
const embedding = embeddingMatrix[tokenId]; // [d_model]
const x = embedding.add(positionalEncoding[pos]);
// d_model ~ 4096, learned during training

Read the implementation

Connect each part of the code to the concept before you type.

1. Table lookup

The token id is a row index into the embedding matrix, returning that token's vector.

const embedding = embeddingMatrix[tokenId];

2. Vector size

Each embedding is a dense vector of length d_model, the model's hidden width.

// [d_model]

3. Add position

Positional information is added to the token embedding to form the input to the first layer.

const x = embedding.add(positionalEncoding[pos]);

4. Learned params

The embedding matrix is a parameter learned by gradient descent, not hand-set.

// d_model ~ 4096, learned during training

5. What it does

The embedding layer is just a lookup that turns each id into a learned vector.

// each token id maps to a learned vector

A complete learning loop

Understand

Read the concept map and follow the code breakdown.

Reconstruct

Type the implementation, repair mistakes and work through every recall exercise.

Verify

Use the final check without hints; revisit weak concepts in short practice sessions.