Rebuilt the architecture behind GPT from scratch, trained it on English→Italian translation, and learned more from the bugs than the math.

Last week I stopped using transformers.

Not entirely, but I stopped using the comfortable version of them.

No Hugging Face pipelines.

No pre-trained checkpoints.

No high-level abstractions.

I wanted to understand what actually sits underneath GPT, Claude, BERT, and nearly every modern LLM.

So I built a transformer from scratch, trained it on English→Italian translation, and ran it on a GPU.

Three days later I had:

  • A working encoder-decoder transformer

  • A CUDA out-of-memory crash

  • A broken dataset loader

  • Several hours spent tracing tensor shapes

  • And a model that could finally translate real sentences

Here's what I learned.

The surprising thing about transformers

Before transformers, language models were largely built around RNNs and LSTMs.

The limitation was simple:

To understand word #100, the model had to process words 1→99 first.

Everything happened sequentially.

Transformers removed that bottleneck.

Instead of processing words one at a time, every word can directly look at every other word through a mechanism called attention.

That one idea changed everything.

It's the reason modern LLMs became possible.

The entire architecture in one diagram

"I love Paris"
      │
      ▼
Tokenize
      │
      ▼
Embeddings
      │
      ▼
Positional Encoding
      │
      ▼
Encoder Stack (×6)
      │
      ▼
Contextual Understanding
      │
      ▼
Decoder Stack (×6)
      │
      ▼
"Amo Parigi"

The encoder reads the sentence.

The decoder generates the output one token at a time.

That's the whole system at a high level.

The mental model that made everything click

The breakthrough for me came from realizing I'd seen this architecture before.

If you've worked with DETR (the transformer-based object detector), most of the architecture is identical.

DETR                                Translation
─────────────────────────────────────────────────────
CNN feature tokens            →    Words
Encoder self-attention        →    Encoder self-attention
Object queries                →    Growing token sequence
Query self-attention          →    Decoder self-attention
Cross-attention               →    Cross-attention
Class + bbox heads            →    Vocabulary projection

The core transformer machinery barely changes.

The only major difference is generation.

Object detection predicts many objects in parallel.

Translation can't.

Word #3 depends on words #1 and #2, so the decoder generates sequentially.

Once that clicked, the architecture felt dramatically simpler.

Building block #1: turning words into vectors

The model can't work directly with text.

Every token gets converted into a dense vector.

class InputEmbeddings(nn.Module):
    def __init__(self, d_model, vocab_size):
        self.embedding = nn.Embedding(vocab_size, d_model)

    def forward(self, x):
        return self.embedding(x) * math.sqrt(self.d_model)

A token becomes a vector of hundreds of numbers.

Initially those numbers are random.

During training, they gradually learn semantic meaning.

Building block #2: giving the model a sense of order

Attention alone has no idea which word comes first.

Without additional information:

I love Paris
Paris love I

look identical.

That's why transformers add positional encodings.

pe[:, 0::2] = torch.sin(position * div_term)
pe[:, 1::2] = torch.cos(position * div_term)

Each position receives a unique mathematical fingerprint.

Now the model knows both:

  • What the word is

  • Where it appears

Building block #3: attention

This is the entire innovation that launched modern LLMs.

scores = (Q @ K.transpose(-2, -1)) / sqrt(d_k)
scores = softmax(scores)
output = scores @ V

Three lines.

That's it.

Conceptually, each word asks:

❝

Which other words matter for understanding me?

The model calculates relevance scores, converts them into probabilities, and then combines information from the most relevant tokens.

Instead of reading words sequentially, every word can immediately access information from the entire sentence.

Self-attention and cross-attention are the same thing

One detail that confused me initially:

There isn't a special implementation for cross-attention.

It's literally the same attention block.

# Self-attention
attention(x, x, x)

# Cross-attention
attention(
    decoder_tokens,
    encoder_output,
    encoder_output
)

The formula never changes.

Only the source of Q, K, and V changes.

That realization removed a lot of unnecessary complexity from my mental model.

Following a real sentence through the model

Most tutorials stop with random tensors.

Here's what actually happens with a real input.

"I love Paris"

↓ Tokenization

[SOS, 40, 12, 8, EOS, PAD...]

↓ Embeddings

(350, 512)

↓ Positional Encoding

(350, 512)

↓ Encoder

(350, 512)

At this point every token contains contextual information about the entire sentence.

Then decoding begins.

[SOS]
  ↓
"Amo"

[SOS, Amo]
  ↓
"Parigi"

[SOS, Amo, Parigi]
  ↓
EOS

One token at a time.

Each new token can see:

  1. Previous generated tokens

  2. The encoder's output

That's how translation happens.

What broke when I actually tried to run it

This is the part most tutorials skip.

Bug #1 — a dead dependency

The training script imported torchtext.

Modern Python environments don't play nicely with it anymore.

After digging through the code, I realized it wasn't even being used.

Deleted it.

Problem solved.

Bug #2 — the dataset moved

The original code loaded:

load_dataset("opus_books", "en-it")

That no longer works.

The dataset path changed.

Updating it to the correct Hugging Face dataset fixed the issue.

Bug #3 — instant CUDA out-of-memory

I increased the batch size to better utilize a Colab T4 GPU.

The result:

torch.OutOfMemoryError:
CUDA out of memory

Immediately.

The reason is attention's memory cost.

Attention matrices scale roughly with:

batch × heads × seq_len²

With a sequence length of 350, those matrices become large very quickly.

And they exist across:

  • Encoder self-attention

  • Decoder self-attention

  • Decoder cross-attention

Across every layer.

Reducing the batch size solved the problem.

The lesson wasn't about transformers.

It was about resource management.

What surprised me most

The biggest surprise wasn't the math.

It was how small the core architecture actually is.

Once you strip away:

  • Massive datasets

  • RLHF

  • Distributed training

  • Billions of parameters

The fundamental transformer is remarkably compact.

A handful of reusable blocks.

One attention formula.

Some residual connections.

A lot of matrix multiplication.

That's it.

The real challenge isn't understanding the architecture.

It's tracing tensor shapes, debugging training runs, and building enough intuition to know where things break.

Final takeaway

If you've been using LLMs for months, or years and still feel like the architecture is a black box, build one yourself.

Not because you'll create the next GPT.

Because once you've watched a sentence move through the encoder, decoder, attention layers, and projection head, the mystery largely disappears.

And that's when the papers, libraries, and production systems start making a lot more sense.

If you're building something interesting this week, hit reply and tell me about it. I'm always looking for the next rabbit hole to explore.

Keep Reading

© 2026 AI Shipped.
beehiivPowered by beehiiv