Open-Vocabulary Calibration for Fine-tuned CLIP

This is the official implementation for Open-Vocabulary Calibration for Fine-tuned CLIP 🔗 at ICML 2024.

Abstract

Vision-language models (e.g, CLIP) have emerged as formidable tools, showing their strong capability in handling various open-vocabulary tasks in image recognition, text-driven visual content generation, and visual chatbots, to name a few. In recent years, considerable efforts and resources have been devoted to adaptation methods for improving the downstream performance of VLMs, particularly on parameter-efficient fine-tuning methods like prompt learning. However, a crucial aspect that has been largely overlooked is the confidence calibration problem in fine-tuned VLMs, which could greatly reduce reliability when deploying such models in the real world. This paper bridges the gap by systematically investigating the confidence calibration problem in the context of prompt learning and reveals that existing calibration methods are insufficient to address the problem, especially in the open-vocabulary setting. To solve the problem, we present a simple and effective approach called Distance-Aware Calibration (DAC), which is based on scaling the temperature using as guidance the distance between predicted text labels and base classes. The experiments with 7 distinct prompt learning methods applied across 11 diverse downstream datasets demonstrate the effectiveness of DAC, which achieves high efficacy without sacrificing the inference speed.

Setup

1. Installation

For installation and other package requirements, please follow the instructions detailed in INSTALL.md.

2. Data preparation

Please follow the instructions at DATASETS.md to prepare all datasets.

Quick Start

Please refer to ./run for more info about our scripts.

1. Tuning & Evaluation

GPU_ID=1 # replace it with your GPU ID
bash run/classification/zeroshot.sh ${GPU_ID} # zero-shot CLIP
bash run/classification/fewshot.sh ${GPU_ID} # fine-tuned CLIP

2. Calibration (reproduce our results)

GPU_ID=1
bash run/calibration/fewshot_scaling.sh ${GPU_ID} # DAC on fine-tuned CLIP

The results will be logged in output/base2new/logs_base2new.csv.

☑️ Supported Methods

1. Tuning

We mainly use parameter-efficient fine-tuning methods like prompt learning in our experiments.

Method	Paper	Configs	Code
CoOp	IJCV'22	link	link
Co-CoOp	CVPR'22	link	link
ProDA	CVPR' 20'22	link	link
KgCoOp	CVPR 2023	link	link
MaPLe	CVPR'23	link	link
ProGrad	ICCV'23	link	link
PromptSRC	ICCV'23	link	link

---

2. Calibration

type	Method	Paper	Configs	Code
Scaling	Temperature Scaling	ICML'17	link	link
	Density-Ratio Calibration	NeurIPS'24	-	link
Bin	Histogram Binning	ICML'01	-	-
	Isotonic Regression	KDD'02	-	-
	Multi-Isotonic Regression	ICML'20	-	link

---

Citation

If you find this useful in your research, please consider citing:

@inproceedings{wang2024open,
  title={ Open-Vocabulary Calibration for Fine-tuned CLIP},
  author={Wang, Shuoyuan and Wang, Jindong and Wang, Guoqing and Zhang, Bob and Zhou, Kaiyang and Wei, Hongxin},
  booktitle = {International Conference on Machine Learning (ICML)},
  year = {2024}
}

Acknowledgements

Our code is inspired by CoOp and MaPLe. We thank the authors for releasing their code.