The complete mental model behind ViT, CLIP, and RT-DETRv2 — with dry runs, real tensor shapes, and the misconceptions that trip almost everyone.
Your model says "car — 95% confident."
Cool. Where is the car?
If answering that question feels like jumping to an entirely separate engineering universe, you are missing the ultimate unifying mental model of computer vision.
Embedding, classification, and detection aren’t three isolated topics. They are a single, continuous pipeline that grows. Each stage is simply the previous stage plus one more architectural "bolt-on."
To ground this, let's track a single image through the entire pipeline: A sunny street scene containing one car and one dog.
By the end of this issue, you’ll be able to draw how a machine processes this scene entirely from memory.
Stage 1 — Embedding: the encoder alone
A Vision Transformer (ViT) starts by treating an image like a sentence:
Raw Image (224 × 224 pixels × 3 RGB Channels)
│
▼
Cut into a grid of 196 patches (Each patch is 16 × 16 pixels)
│
▼
Flatten & project patches → 196 Visual "Tokens" (Shape: 196 × 768 dimensions)
│
▼
Add 1 learnable [CLS] (Classification) token → (Shape: 197 × 768)
│
▼
Pass through [Self-Attention + Feed-Forward Layers] × 12 times
│
▼
Final Output: Extract just the [CLS] Token → (Shape: 1 × 768)Let's Unpack the Math Behind the Tokens
If you look closely at those numbers, it isn't arbitrary magic—it's simple geometry:
The Raw Image (224 * 224 * 3): Represents an image 224 pixels wide, 224 pixels high, across 3 RGB color channels (Red, Green, and Blue).
The 196 Patches: We slice that image into a 14 * 14 grid of patches, where each patch is 16 * 16 pixels (14 * 14 = 196).
The 768 Vector Length: If you unroll one single 16 *16 patch with its 3 RGB color channels into a straight line of numbers, you get exactly 768 values (16 * 16 * 3 = 768). We linearly project these values into our model's hidden embedding dimension (D=768).Two things worth locking in:
The Intuition
Once your input tokens are ready, they enter a stack of 12 identical Transformer Encoder layers. Inside these 12 layers, a mechanism called self-attention allows every single visual token to look at and interact with every other token simultaneously.
A patch containing a rubber tire notices a patch containing a windshield. Because they always appear together, they stop being isolated pixel channels and start "meaning" car together.
Meanwhile, the [CLS] (Classification) token acts like an empty sponge. It rides along through all 12 encoder layers, attending to every single patch in the scene, gradually absorbing a compressed structural summary of the whole image.
The final output is one single vector (1 * 768). No labels. No probabilities. No classes. Just raw, geometric positioning in a semantic space.
What this powers in production:
Store it in a vector database, run a cosine similarity check, and you instantly power:
Visual similarity search ("Find products that look like this car")
Recommendation engines
Image deduplication
Stage 2 — Classification: bolt on one head
Want a definitive label instead of an abstract vector? You don't rewrite the model. You bolt exactly one thing onto the end of your Stage 1 encoder: a classification head.
Think of the classification head as the model's trained dictionary. During training on millions of images, the model learns a fixed "direction" in space for every category. It memorizes a vector for "car-ness," a vector for "dog-ness," and so on.
When our street scene vector emerges from the encoder, we run a quick matrix multiplication to see which learned direction it aligns with best:
[Your Image Vector] · [Learned Class Matrix]
= [Output Logits] (1 × 768) · (768 × 1000) = (1 × 1000)To turn those raw alignment scores (logits) into something readable, we pass them through a softmax function:
Logits [6.8, 3.9, 0.4] → Softmax → Probabilities [Car: 0.95, Dog: 0.04, Tree: 0.01]The Intuition
A classifier doesn’t actually store an image of a car. It learns geometric decision boundaries in embedding space. The dot product simply measures: "How close does the meaning of this new image sit next to our learned memory of what a car is?" The highest alignment wins.
But once you know where the image lives in that semantic geometry, you usually want to give it a name.
🌟 The Zero-Shot Variant: > Instead of using a statically trained class matrix, you can use CLIP’s text encoder to generate vectors for any labels you want (e.g., "electric sedan", "pickup truck"). Swap those text vectors into the head slot, run the exact same equation, and you have custom classification at inference time with zero custom training.
Notice what changed from Stage 1: Absolutely nothing in the encoder. We just added a single mathematical checkpoint at the very end.
Stage 3 — Detection: bolt on a decoder (RT-DETRv2)
A single label like "car: 0.95" fails catastrophically for our street scene. It ignores the dog entirely and gives us no spatial coordinates. To answer "Where?", we need to bolt an entirely different class of machinery onto our foundation: a transformer decoder.
🚨 Architectural Plot Twist: The CNN Hand-Off
Notice a massive shift here. Stage 1 and Stage 2 relied entirely on a pure Vision Transformer (ViT) encoder slicing raw pixels into sequential patches.
Real-time object detection models (like RT-DETRv2) do not use the patch-slicing ViT encoder from the previous stages. Because detection requires incredibly high spatial accuracy to locate small items, slicing an image into stiff 16 ×16 patches loses too much detail. Instead, Stage 3 completely replaces the input processing step with a classical CNN Backbone (like ResNet or HGNet).
Let's look at how the modern real-time detection pipeline steps through this process:
1. The CNN Backbone Feature Extraction
The raw image passes through a convolutional neural network (CNN). The CNN crunches the dense pixels and outputs a vast field of spatial feature tokens—let's say 400 tokens × 256 dimensions.
The Shape: 400 × 256
The Reality: These are not objects yet. They are raw local visual indicators ("shiny metallic texture here", "furry pattern there").
2. Encoder Contextualization
Before the decoder gets involved, these 400 feature tokens pass through a standard Transformer Encoder using self-attention. This layer allows local observations to look at the big picture. After this pass, an isolated "wheel-like" token becomes a "wheel-attached-to-a-CAR" token. The features gain global scene context.
3. Enter the 300 Detectives (Object Queries)
We introduce 300 object queries into the transformer decoder. These queries are learnable, blank vectors. Think of them as 300 trained detectives sitting in a room, each with a specific, highly trained instinct on what type of object to hunt for in a scene.
4. The Decoder Interrogation
Over 6 distinct decoder layers, two critical operations happen simultaneously:
Self-Attention among queries: The 300 detectives talk to each other so they don't accidentally make duplicate claims on the exact same car or dog.
Cross-Attention into the image: Each detective query cross-examines the 400 contextualized image feature tokens, asking: "Is the specific thing I am trained to hunt located anywhere in these pixels?"
After 6 rounds of intense cross-attention negotiation, the decoder outputs 300 distinct object embeddings (Shape: 300 × 256)—three hundred object-specific concepts.
To turn those 300 embeddings into real answers, we pass them through two distinct heads simultaneously:
Head A: The Parallel Classifier
We run Stage 2's classification trick 300 times in parallel. We multiply the decoder output against detection's own specialized class matrix (e.g., 80 COCO classes + 1 extra slot for "no object"):
(300 × 256) · (256 × 81) = 300 × 81 Logits
Softmax runs across the 81 classes, separately for all 300 slots (never across the 300 slots themselves).
Query 17 aligns perfectly with the dog features →
[Dog: 0.89]Query 84 aligns perfectly with the car features →
[Car: 0.96]Query 244 finds absolutely nothing →$
[No Object: 0.99]→ Filtered out
Head B: The Box Regressor
At the exact same time, the 300 embeddings pass through a regression head that maps the features directly to physical spatial coordinates:
300 × 256 → Regression Model → 300 × 4 Coordinates (x, y, width, height)
The output isn't a single guess; it's a structural set of objects: car → box, dog → box.
⚠️ Crucial distinction: There are absolutely no logits or softmax operations here. You cannot run a softmax on a continuous pixel coordinate. It is pure bounding-box numerical regression.
Filtering: Even if there are only 2 objects in our street photo (one car, one dog), the model always makes 300 predictions. The "no object" slots simply get dropped via post-processing thresholds: 300 slots in → 2 active detections out.
The five misconceptions (I hit every one of these)
❌ "In (300, 81), 81 is the number of objects." ✅ 300 = object slots. 81 = class choices per slot. Rows are objects; columns are classes.
❌ "20 objects → the output should be (20, 4)." ✅ Always (300, 4) and (300, 81). Fixed slots in, variable detections out — filtering happens after.
❌ "ViT is encoder-decoder." ✅ Encoder-only. Decoders and object queries are DETR-family machinery.
❌ "The boxes are also logits." ✅ Boxes are regression — continuous coordinates. Logits are class scores only.
❌ "Embedding + vector search = classification." ✅ Retrieval has no labels, no logits, no softmax. Classification needs a head and a fixed label set.
Which one does your product need?
"Find similar things" → embeddings + vector DB (search, recommendations, dedup). Often zero training.
"Label things into my categories" → classification head — or zero-shot if your labels can be written as text.
"Locate or count things" → detection (RT-DETRv2): anything that needs where.
Avoid the classic mismatches: detection is expensive overkill when you only need one label, and embeddings can't give you a definitive label with a confidence score.