Vision Transformer

Transformers for Image Recognition At Scale

PublishedApril 17, 2023

•2 min read•View as Markdown

Vitou Phy

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

A Model that Retrieves Documents When in Doubt

Summary of Active retrieval augmented generation paper

Dec 24, 20234 min read

A Model that Retrieves Documents When in Doubt

Training MNIST with MLX on Apple Silicon

Classifying digits using Apple M1 GPU

Dec 18, 20233 min read

Training MNIST with MLX on Apple Silicon

Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks

Narrowing performance gap with cross-encoder using weakly labeled data

Jul 31, 20233 min read

Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks

LLaMA: Open and Efficient Foundation Language Models

Large Language Model for Commercial Purpose

Jul 25, 20232 min read

LLaMA: Open and Efficient Foundation Language Models

DeiT: Data-Efficient Image Transformer

Training data-efficient image transforms & distillation through attention

Jul 10, 20233 min read

Demystify ML

20 posts

Concept

Transformer Block

We won't go into depth on the main architecture of the Transformer.

Briefly in machine translation, given a sequence of words, we get the vector representation (embedding) for each word and pass them to Transformer.

Additional positional embedding is also added on top of the word embedding so that the model is aware of where that word is.

Transformer takes in a sequence of vectors and computes each timestep (word) independently from the previous timestep, unlike recurrent neural networks (RNN). This allows the model to be trainable in parallel across multiple machines.

To be aware of the surrounding words, Transformer uses a self-attention mechanism. It may associate a word at timestep $t$ with $n$ words before and after it.

Vision Transformer

For computer vision, it's a little different since we are not working with a sequence of words. The only thing we need to change is its input.

Hence, we split an image into $N$ patches of size $m \times m$. Then, we rearrange those $N$ patches into an array (similar to an array of words).

Reshape each patch and apply a linear transformation layer to get its corresponding embedding. Note that this layer will be shared across all timesteps.

Add position embedding and feed them into Transformer.

Vision Transformer Architecture

Strengths and Limitations

Pretrained on a large dataset (at least 100M samples) yields better performance compared to ResNet architecture (non-Transformer SOTA in computer vision).

Experimenting with a dataset of size 10M and 30M shows that the performance is somewhat limited.

Such limitation is not only for vision but also in the NLP domain as well.

Left Figure: Transfer to ImageNet. Right Figure: Few-shots on ImageNet. Both suggest that a big dataset leads to performance improvement on Vision Transformer (ViT)

Vision Transformer

Comments

More from this blog

A Model that Retrieves Documents When in Doubt

Training MNIST with MLX on Apple Silicon

Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks

LLaMA: Open and Efficient Foundation Language Models

DeiT: Data-Efficient Image Transformer

Motivation

Concept

Transformer Block