VLMEvalKit Versions Save

What's Changed

• [Feature] Support multi_generate by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/1
• [Tool] Minor updates 1205 by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/3
• TranCore-M 20231208 by @PCIResearch in https://github.com/open-compass/VLMEvalKit/pull/8
• [Result] update TransCore Results by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/9
• COREMM Evaluation Benchmark by @youngfly11 in https://github.com/open-compass/VLMEvalKit/pull/6
• Adding MMVet by @llllIlllll in https://github.com/open-compass/VLMEvalKit/pull/7
• [Doc] README Update 12.11 by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/10
• Updating mmvet_eval by @llllIlllll in https://github.com/open-compass/VLMEvalKit/pull/11
• [Feature] Add run.py and simplify the evaluation. by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/12
• [Doc] Refine README by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/13
• [Fix] Fix README by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/14
• add dataset md5 check integrity by @FangXinyu-0913 in https://github.com/open-compass/VLMEvalKit/pull/15
• [Feature] Support two Vision APIs by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/19
• Add COCO dataset by @FangXinyu-0913 in https://github.com/open-compass/VLMEvalKit/pull/16
• [Fix] Fix 1221 by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/21
• [Feature] More Robust API Evaluation by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/22
• [Refactor] Refactor Custom Prompt & Fix mPLUG-Owl2 acc by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/23
• [Dataset] VQA Datasets by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/25
• [Fix] Bug Fix by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/26
• Add MMMU dataset by @llllIlllll in https://github.com/open-compass/VLMEvalKit/pull/18
• Add QwenVLPlus API by @llllIlllll in https://github.com/open-compass/VLMEvalKit/pull/27
• [Result] Update MMMU Acc by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/30
• [Feature] Support LLaVA_XTuner models by @LZHgrla in https://github.com/open-compass/VLMEvalKit/pull/17
• [Result] Update XTuner Performance by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/31
• [Result] Update COCO Caption Results by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/35
• add dataset ChartQA by @FangXinyu-0913 in https://github.com/open-compass/VLMEvalKit/pull/28
• [Feature]: Add ScienceQA by @YuanLiuuuuuu in https://github.com/open-compass/VLMEvalKit/pull/24
• [Dataset] MathVista dataset by @llllIlllll in https://github.com/open-compass/VLMEvalKit/pull/29
• [Dataset] HallusionBench by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/38
• Add sharedcaptioner and cogvlm by @fitzpchao in https://github.com/open-compass/VLMEvalKit/pull/37
• [Fix] Fix GPT error with parallel calling by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/40
• [Eval] multiple_choice.py: E->Z by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/41
• [Eval] Use exact matching for Y/N and multi-choice when OPENAI_API_KEY not set by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/44
• add VLM: monkey by @ShuoZhang2003 in https://github.com/open-compass/VLMEvalKit/pull/45
• [Fix] Fix multiple choice evaluation when OPENAI_API_KEY missing by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/48
• [Improvement] Support non-contiguous choices by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/49
• Add Emu2 and Emu2_chat by @llllIlllll in https://github.com/open-compass/VLMEvalKit/pull/47
• [Dataset] Add DocVQA by @llllIlllll in https://github.com/open-compass/VLMEvalKit/pull/50
• [Benchmark] support AI2D by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/51
• add monkey-chat by @ShuoZhang2003 in https://github.com/open-compass/VLMEvalKit/pull/54
• Add LLaVA-InternLM2 by @LZHgrla in https://github.com/open-compass/VLMEvalKit/pull/53
• [Dataset] Support LLaVABench by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/55
• Support torchrun for emu2&emu2_chat and fix bug by @llllIlllll in https://github.com/open-compass/VLMEvalKit/pull/52
• support sharegpt4v-13b by @xiaoachen98 in https://github.com/open-compass/VLMEvalKit/pull/56
• Fix bug in file.py by @Ezra-Yu in https://github.com/open-compass/VLMEvalKit/pull/58
• [Result] Update Evaluation Results by @kennymckormick in https://github.com/open-compass/VLMEvalKit/pull/60

New Contributors

• @kennymckormick made their first contribution in https://github.com/open-compass/VLMEvalKit/pull/1
• @PCIResearch made their first contribution in https://github.com/open-compass/VLMEvalKit/pull/8
• @youngfly11 made their first contribution in https://github.com/open-compass/VLMEvalKit/pull/6
• @llllIlllll made their first contribution in https://github.com/open-compass/VLMEvalKit/pull/7
• @FangXinyu-0913 made their first contribution in https://github.com/open-compass/VLMEvalKit/pull/15
• @LZHgrla made their first contribution in https://github.com/open-compass/VLMEvalKit/pull/17
• @YuanLiuuuuuu made their first contribution in https://github.com/open-compass/VLMEvalKit/pull/24
• @fitzpchao made their first contribution in https://github.com/open-compass/VLMEvalKit/pull/37
• @ShuoZhang2003 made their first contribution in https://github.com/open-compass/VLMEvalKit/pull/45
• @xiaoachen98 made their first contribution in https://github.com/open-compass/VLMEvalKit/pull/56
• @Ezra-Yu made their first contribution in https://github.com/open-compass/VLMEvalKit/pull/58

Full Changelog: https://github.com/open-compass/VLMEvalKit/commits/v0.1