Fundamentals of Embeddings, Encoder-Decoders

token embeddings = 1 vector to represent a token text embeddings = 1 vector to represents text (sentence(s)) process to get text embeddings: (token embeddings -> transformer -> contextual embeddings -> pooling -> text embedding)

GPT LLM model = decoder = attention to everything before Embedding model = encoder = attention to everything before AND after input, understand, output = encoder-decoder (like translation)

training an embedding model:

• pretraining on transformer (learn the language)
  • masked token prediction (common for encoder models) -> encoder-based embeddings
  • or next-token predicition (common for decoder models) -> decoder-based embeddings
  • or encoder-decoder model
  • or another large-scale language objective
• contrastive embedding training (learn similarities)
  • train on pairs or groups
  • query, positive, negative

during LLM (decoder architecture) pretraining:

• Also trains an embeddings model implicitly (but not specialized for similarity)
• Every Transformer layer produces hidden vectors for the tokens
  • Those vectors contain information about the whole preceding context

• Yes, many dedicated embedding models are trained with contrastive learning.

  • Similar text pairs are pulled closer in vector space.
  • Unrelated text pairs are pushed farther apart.
  • Example:
    • “How do I reset my password?”
    • “Password reset instructions”
    • These should get nearby vectors.

• Embedding models are usually separate from GPT-style LLMs.

  • GPT model:
    • Main goal: predict the next token.
    • Usually decoder-only.
    • Output: text tokens.
  • Embedding model:
    • Main goal: produce a useful vector for similarity/search.
    • Often encoder-only, though other architectures can work.
    • Output: one vector for the input text.

• But embedding models can still benefit from LLM-style pretraining.

  • Common process:
    1. Pretrain a Transformer on lots of text.
    2. Then train or fine-tune it with a contrastive embedding objective.
  • So it can first learn language generally, then learn:
    • which sentences mean similar things,
    • which queries match which documents,
    • how to organize meaning in vector space.

• It does not have to be trained from a GPT model specifically.

  • GPT is decoder-only.
  • Many embedding models start from encoder models, such as BERT-like architectures.
  • Some modern systems also use decoder-only LLMs and adapt their hidden states into embeddings.
  • The important part is the final embedding-specific training objective, not whether it started as GPT.

Encoder vs decoder

• Encoder
  • Reads the input text.
  • Produces contextual vectors.
  • Usually can look at tokens on both sides of each token.
  • Good for:
    • embeddings,
    • classification,
    • search,
    • understanding text.

text → encoder → vectors

• Decoder
  • Generates tokens one at a time.
  • Looks at previous tokens and predicts the next one.
  • Good for:
    • writing,
    • answering questions,
    • generating code.

previous tokens → decoder → next token

• Encoder–decoder
  • The encoder reads and understands the input.
  • The decoder generates a new output based on the encoder’s representations.
  • Common for:
    • translation,
    • summarization,
    • text-to-text tasks.

English sentence
      ↓
   encoder
      ↓
internal representations
      ↓
   decoder
      ↓
French sentence

Example:

Encoder reads:
"Translate 'hello' to French"

Decoder generates:
"bonjour"

Token embeddings vs text embeddings

• Token embedding
  • Represents one token.
  • A token may be:
    • a whole word,
    • part of a word,
    • punctuation,
    • a few characters.
  • Example:

"unbelievable"

might become:
["un", "believ", "able"]

Each token gets its own starting vector.

• Text embedding
  • Represents the meaning of the entire input.
  • It may summarize:
    • one sentence,
    • multiple sentences,
    • a paragraph,
    • a document chunk.

"The dog is running through the park."
        ↓
one text embedding vector

How they are related

• Yes, they are related parts of the same processing pipeline.

Text
 ↓
Tokenizer
 ↓
Token IDs
 ↓
Token embedding lookup
 ↓
One starting vector per token
 ↓
Transformer layers
 ↓
One contextual vector per token
 ↓
Pooling / projection
 ↓
One text embedding

• Step by step:
  • The tokenizer converts text into token IDs.
  • The token embedding table converts each token ID into a starting vector.
  • The Transformer modifies those vectors using context.
  • The final token vectors now contain contextual meaning.
  • The model combines those token vectors into one text embedding.

Example:

"The bank approved my loan"

Initial token vectors:

"The"  → vector
"bank" → vector
"approved" → vector
"my" → vector
"loan" → vector

After the Transformer, the vector for “bank” understands that it means a financial institution.

Then the model combines all the contextual token vectors into one vector representing the full sentence.

The simplest distinction

• Token embedding: starting vector for one token.
• Contextual token representation: that token’s vector after Transformer processing.
• Text embedding: one combined vector representing the whole input.
• GPT output: probabilities over possible next tokens.

So the full relationship is:

token embeddings are inputs to the Transformer

contextual token vectors are outputs of Transformer layers

text embeddings ar

e created by combining those contextual token vectors