This is the PyTorch code of the BLIP paper [blog]. The code has been tested on PyTorch 1.10. To install the dependencies, run
pip install -r requirements.txt
Catalog:
Run our interactive demo using Colab notebook (no GPU needed). The demo includes code for:
Try out the Web demo, integrated into Huggingface Spaces 🤗 using Gradio.
Replicate web demo and Docker image is also available at
Num. pre-train images | BLIP w/ ViT-B | BLIP w/ ViT-B and CapFilt-L | BLIP w/ ViT-L |
---|---|---|---|
14M | Download | - | - |
129M | Download | Download | Download |
Task | BLIP w/ ViT-B | BLIP w/ ViT-B and CapFilt-L | BLIP w/ ViT-L |
---|---|---|---|
Image-Text Retrieval (COCO) | Download | - | Download |
Image-Text Retrieval (Flickr30k) | Download | - | Download |
Image Captioning (COCO) | - | Download | Download |
VQA | Download | Download | - |
NLVR2 | Download | - | - |
python -m torch.distributed.run --nproc_per_node=8 train_retrieval.py \ --config ./configs/retrieval_coco.yaml \ --output_dir output/retrieval_coco \ --evaluate
python -m torch.distributed.run --nproc_per_node=8 train_retrieval.py \ --config ./configs/retrieval_coco.yaml \ --output_dir output/retrieval_coco
python -m torch.distributed.run --nproc_per_node=8 train_caption.py --evaluate
python -m torch.distributed.run --nproc_per_node=8 eval_nocaps.py
python -m torch.distributed.run --nproc_per_node=8 train_caption.py
python -m torch.distributed.run --nproc_per_node=8 train_vqa.py --evaluate
python -m torch.distributed.run --nproc_per_node=16 train_vqa.py
python -m torch.distributed.run --nproc_per_node=8 train_nlvr.py --evaluate
python -m torch.distributed.run --nproc_per_node=16 train_nlvr.py
In order to finetune a model with ViT-L, simply change the config file to set 'vit' as large. Batch size and learning rate may also need to be adjusted accordingly (please see the paper's appendix for hyper-parameter details). Gradient checkpoint can also be activated in the config file to reduce GPU memory usage.
python -m torch.distributed.run --nproc_per_node=8 pretrain.py --config ./configs/Pretrain.yaml --output_dir output/Pretrain
pip install decord
python -m torch.distributed.run --nproc_per_node=8 eval_retrieval_video.py
We provide bootstrapped pre-training datasets as json files. Each json file contains a list. Each item in the list is a dictonary with two key-value pairs: {'url': url_of_image, 'caption': text_of_image}.
Image source | Filtered web caption | Filtered synthetic caption by ViT-B | Filtered synthetic caption by ViT-L |
---|---|---|---|
CC3M+CC12M+SBU | Download | Download | Download |
LAION115M | Download | Download | Download |
If you find this code to be useful for your research, please consider citing.
@inproceedings{li2022blip, title={BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation}, author={Junnan Li and Dongxu Li and Caiming Xiong and Steven Hoi}, year={2022}, booktitle={ICML}, }
The implementation of BLIP relies on resources from ALBEF, Huggingface Transformers, and timm. We thank the original authors for their open-sourcing.
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。