The Vision Transformer tackled the image resolution problem in using transformers for images by dividing the image into patches. Each patch is mapped to an input embedding via a learned linear transformation, and these representations are fed into the transformer network. Like ImageGPT, standard 1D positional encodings are learned.
ViT is an encoder transformer. A learnable <cls> token is prepended to the sequence of patch embeddings before they are fed into the transformer. During self-attention, the <cls> token attends to all image patches and accumulates information about the entire image. After the final transformer layer, the output embedding corresponding to the <cls> token is mapped via a final linear layer to create activations, which are fed into a softmax function to generate class probabilities.
Unlike BERT, another encoder model, ViT uses supervised pre-training on a large database of 303 million labeled images from 18,000 classes.
After pre-training, the system is applied to the final downstream classification task by replacing this final layer with one that maps to the desired number of classes and is fine-tuned.
For the ImageNet benchmark, the original ViT achieved an 11.45% top-1 error rate. However, it did not outperform the best contemporary convolutional networks without supervised pre-training. The strong inductive bias of convolutional networks can only be superseded by employing extremely large amounts of training data.
dl
ViT
?
- Divide image into patches, which are mapped to input embeddings (with learned 1D positional encodings)
- Learnable
<cls>token is prepended to the sequence of patch embeddings - During self-attention, the
<cls>token attends to all image patches and accumulates information about the entire image - After the final transformer layer, the output embedding for
<cls>passes through an FC layer to create activations, which are then softmaxed to generate class probabilities
+++