LMMs-Eval/LLaVA-1.5/1.6

	A	B	C	D	E	F	G	H	I	J	K	L	M	N
1	LMMs-Eval				LLaVA-1.5				LLaVA-1.6				Comments	Update: Mar. 8th, 2024
2	Datasets	Meta Info			1.5-7B (report)	1.5-7B (lmms-eval)	1.5-13B (report)	1.5-13B (lmms-eval)	1.6-7B (lmms-eval)	1.6-7B (lmms-eval)	1.6-13B (lmms-eval)	1.6-34B (lmms-eval)		Env Info:
3	Datasets	Split	Metric	#Num	liuhaotian/llava-v1.5-7b	liuhaotian/llava-v1.5-7b	liuhaotian/llava-v1.5-13b	liuhaotian/llava-v1.5-13b	liuhaotian/llava-v1.6-mistral-7b	liuhaotian/llava-v1.6-vicuna-7b	liuhaotian/llava-v1.6-vicuna-13b	liuhaotian/llava-v1.6-34b
4	AI2D	test	Acc	3,088	-	54.79	-	59.49	60.75	66.58	70.04	74.94		torch 2.2.1 + cuda 12.1
5	ChartQA	test	RelaxedAcc	2,500	-	18.24	-	18.20	38.76	54.84	62.2	68.72		torch 2.2.1 + cuda 12.1
6	CMMMU	val	Acc	900	-	21.80	-	26.30	22.7	24	23,2	39.9		torch 2.2.1 + cuda 12.1
7	COCO-Cap	cococap_val_2014	CIDEr	40,504	-	108.66	-	113.88	107.66	96.98	99.45	103.16		torch 2.2.1 + cuda 12.1
8	COCO-Cap	cococap_val_2017	CIDEr	5,000	-	110.38	-	115.61	109.22	99.93	101.99	105.89		torch 2.2.1 + cuda 12.1
9	DocVQA	val	ANLS	5,349	-	28.08	-	30.29	72.16	74.35	77.45	83.98		torch 2.2.1 + cuda 12.1
10	Flickr	-	CIDEr	31,784	-	74.93	-	79.59	73.14	68.44	66.7	68.48		torch 2.2.1 + cuda 12.1
11	GQA	gqa_eval	Acc	12,578	62.00	61.97	63.30	63.24	54.98	64.23	65.36	67.08		torch 2.2.1 + cuda 12.1
12	Hallusion-Bench	test	All Acc.	951		44.90		42.27	41.74	41.53	44.47			torch 2.2.1 + cuda 12.1
13	InfoVQA	val	ANLS	2,801	-	25.81	-	29.35	43.77	37.09	41.34	51.45		torch 2.2.1 + cuda 12.1
14	LLaVA-W	test	GPT-Eval-Avg	60	63.40	65.3 (0314) 59.6 (0613)	-	72.8 (0314) 66.1 (0613)	71.7 (0613)	72.3 (0613)	72.3 (0613)		LLaVA 1.5 uses GPT4-0314, but it has been deprecated. we use GPT4-0613 and it gives lower score on all model versions	torch 2.2.1 + cuda 12.1
15	MathVista	testmini	Acc	1,000	27.40	26.70	27.60	26.40	37.4	34.4	35.1			torch 2.2.1 + cuda 12.1
16	MMBench	dev	Acc	4377 (dev)\	64.30	64.80	67.70	68.73						torch 2.2.1 + cuda 12.1
17	MMBench-Ch	dev	Acc	4329 (dev)	58.30	57.62	63.60	62.54						torch 2.2.1 + cuda 12.1
18	MME-Cognition	test	total score	2,374	-	348.21	-	295.35	323.92	322.5	316.78	397.14		torch 2.2.1 + cuda 12.1
19	MME-Perception	test	total score	2,374	1510.70	1510.75	-	1522.59	1500.85	1519.29	1575.07	1633.24		torch 2.2.1 + cuda 12.1
20	MMMU	val	Acc	900	-	35.30	36.40	34.80	33.4	35.1	35.9	46.7	Implementation needs to be improved, LLaVA-Next reports results with multiple images while lmms-eval currently only consider single image	torch 2.2.1 + cuda 12.1
21	MMVet	test	GPT-Eval-Avg	218	30.50	30.55	-	35.25	47.75	44.08	49.12			torch 2.2.1 + cuda 12.1
22	MultidocVQA	val	Anls/acc	5,187		16.65/7.21		18.25/8.02	41.4/27.89	44.42/31.32	46.28/32.56	50.16/34.93		torch 2.2.1 + cuda 12.1
23	NoCaps	nocaps_eval	CIDEr	4,500	-	105.54	-	109.28	96.14	88.29	88.27	91.94		torch 2.2.1 + cuda 12.1
24	OKVQA	val	Acc	5,046	-	53.44	-	58.22	54.77	44.25	46.27	46.84		torch 2.2.1 + cuda 12.1
25	POPE	test	F1 Score	9,000	85.90	85.87	-	85.92	86.79	86.4	86.26	87.77		torch 2.2.1 + cuda 12.1
26	ScienceQA	scienceqa-full	Acc.	4,114	-	70.41	-	74.96	0.23	73.21	75.85	85.81		torch 2.2.1 + cuda 12.1
27	ScienceQA	scienceqa-img	Acc	2,017	66.80	70.43	71.60	72.88	0	70.15	73.57			torch 2.2.1 + cuda 12.1
28	SEED-Bench	Seed-1	Image-Acc	17,990	total: 58.6	total: 60.49	image: 66.92	image: 67.06	65.97	64.74	65.64	69.55		torch 2.2.1 + cuda 12.1
29	SEED-Bench-2	Seed-2	Acc	24,371		total : 57.89		total : 59.88	60.83	59.88	60.72	64.98		torch 2.2.1 + cuda 12.1
30	Refcoco	all	CIder			29.76		34.26	9.47	34.2	34.75			torch 2.2.1 + cuda 12.1
31	Refcoco	bbox-test	Cider	5,000		32.45		34.26	9.63	36.17	38.2			torch 2.2.1 + cuda 12.1
32		bbox-testA	Cider	1,975		15.98		16.68	5.9	18.47	18.63			torch 2.2.1 + cuda 12.1
33		bbox-testB		1,810		41.99		45.15	12.5	49.91	51.01			torch 2.2.1 + cuda 12.1
34		bbox-val		8,811		30.35		33.12	9.88	36.28	37.27			torch 2.2.1 + cuda 12.1
35		seg-test		5,000		30.44		32.03	9.42	33.79	33.52			torch 2.2.1 + cuda 12.1
36		seg-testA		1,975		14.44		15.49	5.26	15.43	14.74			torch 2.2.1 + cuda 12.1
37		seg-testB		1,810		40.19		43.47	12.9	47.18	46.97			torch 2.2.1 + cuda 12.1
38		seg-val		8,811		29.12		31.54	9.42	33.1	33.23			torch 2.2.1 + cuda 12.1
39	Refcoco+	all	CIder			28.92		31.01	9.05	31.82	32			torch 2.2.1 + cuda 12.1
40	Refcoco+	bbox-testA	Cider	1,975		20.34		19.78	6.61	22.1	21.62			torch 2.2.1 + cuda 12.1
41		bbox-testB		1,798		39.09		41.61	11.18	43.85	44.93			torch 2.2.1 + cuda 12.1
42		bbox-val		3,805		30.16		33.36	9.56	34.53	35.56			torch 2.2.1 + cuda 12.1
43		seg-testA	Cider	1,975		17.98		18.34	6.05	18.1	17.85			torch 2.2.1 + cuda 12.1
44		seg-testB		1,798		37.46		40.02	11.64	41.15	41.68			torch 2.2.1 + cuda 12.1
45		seg-val		3,805		21.50		31.81	9.12	31.19	30.47			torch 2.2.1 + cuda 12.1
46	Refcocog	all	CIder			57.76		59.23	19.35	52.18	58.02			torch 2.2.1 + cuda 12.1
47	Refcocog	bbox-test	Cider	5,023		58.90		59.86	20.2	53.31	61.83			torch 2.2.1 + cuda 12.1
48		bbox-val		7,573		60.45		61.61	19.77	55	61			torch 2.2.1 + cuda 12.1
49		seg-test	Cider	5,023		55.78		57.34	18.82	49.36	54.28			torch 2.2.1 + cuda 12.1
50		seg-val		7,573		55.63		57.70	18.71	50.19	54.78			torch 2.2.1 + cuda 12.1
51	TextCaps	val	CIDEr	3,166	-	98.15	-	103.92	70.39	71.79	67.39	67.11		torch 2.2.1 + cuda 12.1
52	TextVQA	val	exact_match	5,000	-	46.07	-	48.73	65.76	64.85	66.92	69.31	In the LLaVA paper, the OCR token was utilized as a prompt for the evaluation of TextVQA. You can take this issue as a reference.	torch 2.2.1 + cuda 12.1
53	VizWiz (val)	val	Acc	4,319	-	54.39	-	56.65	63.79	60.64	63.56	66.61		torch 2.2.1 + cuda 12.1
54	VQAv2	val	Acc	214,354	-	76.64	-	78.26	80.32	80.06	80.92	82.07		torch 2.2.1 + cuda 12.1
55	VQAv2	test	Acc	-	78.50		80.00	79.99
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100