Locked-image Tuning: Adding Language Understanding to Image Models

Posted by Andreas Steiner and Basil Mustafa, Research Software Engineers at Google Research, Brain team

The ability to classify images into categories has been transformed by deep learning. It has also been significantly accelerated by transfer learning, whereby models are first pre-trained on large datasets, like ImageNet, to learn visual representations that are then transferred via fine-tuning to a new task with less data (e.g., classifying animals). Previous works such as BiT and ViT employed these methods to …

computer vision google brain image language multimodal learning natural language processing natural-language understanding