CloudCV
Fork me on GitHub

CloudCV: ViLBERT Multi-Task Demo

A single ViLBERT Multi-Task model can perform 8 different vision and language tasks learnt from 12 datasets!

Datasets: VQA v2, GQA, Visual Genome QA, RefCOCO, RefCOCO+, RefCOCOg, Visual7W, GuessWhat, COCO Retrieval, Flickr30k Retrieval, SNLI-VE, NLVR2.

More details about the ViLBERT Multi-Task paper can be found here along with the code for model training & code for model inference + demo interface.

Browsers currently supported by the demo: Google Chrome, Mozilla Firefox.

Try ViLBERT Multi-Task on sample images

Select one or more images to send to our servers. Most tasks require one image. If you're interested in matching a caption to a pair of images (e.g., NLVR2), select 2 images. For image retrieval, select 2 to 4 images. You can also upload your own images in JPG or PNG format.


Note: Nothing is pre-computed for these images. They are treated as a fresh upload with every click.
OR

Upload your own images

Terminal


How it works

  1. You upload an image.
  2. Our servers run the deep-learning based algorithm.
  3. Results and updates are shown in real-time.


Result


Credits

Built by @rishabh jain

Acknowledgements

We thank @jiasen lu for his help.