VNN-COMP 2024 Benchmark Scoring Voting (Responses)

	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S	T	U	V	W	X	Y	Z	AA	AB	AC
1	Timestamp	Email Address	What tool are you voting on behalf, and please list all team members' names as well ? Please coordinate with your team so as you only vote once on behalf of your tool team	What is the tool website / github link?	Benchmark Voting [https://github.com/AI4OPT/ml4acopf_benchmark]	Benchmark Voting [https://github.com/dynaroars/vnncomp-benchmark-generation]	Benchmark Voting [https://github.com/loonwerks/vnncomp2023]	Benchmark Voting [https://github.com/ANTONIONLP/safeNLP]	Benchmark Voting [https://github.com/Khoury-srg/VNNComp23_NN4Sys and https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/nn4sys]	Benchmark Voting [https://github.com/kollerlukas/cora-vnncomp2024-benchmark]	Benchmark Voting [https://github.com/aliabigdeli/LinearizeNN_benchmark2024]	Benchmark Voting [https://github.com/wu-haoze/dist-shift-vnn-comp and https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/dist_shift]	Benchmark Voting [https://github.com/shizhouxing/LSNC_VNNCOMP2024]	Benchmark Voting [https://github.com/xiangruzh/Yolo-Benchmark and https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/yolo]	Benchmark Voting [https://github.com/huanzhang12/vnncomp2024_cifar100_benchmark]	Benchmark Voting [https://github.com/huanzhang12/vnncomp2024_tinyimagenet_benchmark]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/acasxu]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/cctsdb_yolo]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/cgan]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/collins_rul_cnn]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/collins_yolo_robustness]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/metaroom]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/ml4acopf]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/tllverifybench]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/traffic_signs_recognition]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/vggnet16]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/vit]	What dependencies does your tool have, beyond standard ones like machine learning frameworks? Please ensure you list all important dependencies, particularly those developed by the research community, such as underlying optimization libraries (Gurobi, etc.), satisfiability libraries (Z3, etc.), symbolic data structure libraries (CORA, etc.), etc.	Do you have any other comments?

2	5/28/2024 18:01:39	karthikh@iastate.edu	SafeDNN, Karthik Hanumanthaiah	https://git.las.iastate.edu/karthikh/safeDNN	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Marabou
3	5/29/2024 5:00:51	lukas.koller@tum.de	CORA	https://cora.in.tum.de/	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark, Do you collaborate with the benchmark proposers?	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Matlab, CORA ;), GPU
4	5/30/2024 7:10:17	stefano.demarchi@edu.unige.it	NeVer2	neuralverification.org	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	None
5	5/30/2024 10:24:30	augustin.lemesle@cea.fr	PyRAT	https://pyrat-analyzer.com/	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Cvxpy
6	6/3/2024 12:41:53	haozewu22@gmail.com	Tool: Marabou' Team: Clark Barrett, Guy Katz, Haoze Wu	https://github.com/NeuralNetworkVerification/Marabou	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Gurobi
7	6/3/2024 14:25:27	diego.manzanas.lopez@vanderbilt.edu	NNV	https://github.com/verivital/nnv	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	MATLAB License, Gurobi license (optional), mpt3
8	6/3/2024 22:25:35	aarjomandbig@cs.stonybrook.edu	nnenum; team members: Ali Arjomandbigdeli, Stanley Bak	https://github.com/aliabigdeli/nnenum	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark, Do you collaborate with the benchmark proposers?	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark, Do you collaborate with the benchmark proposers?	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Gurobi
9	6/4/2024 4:23:47	huan@huan-zhang.com	alpha-beta-CROWN	https://abcrown.org	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark, Do you collaborate with the benchmark proposers?	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark, Do you collaborate with the benchmark proposers?	Yes, score this benchmark, Do you collaborate with the benchmark proposers?	Yes, score this benchmark, Do you collaborate with the benchmark proposers?	Yes, score this benchmark, Do you collaborate with the benchmark proposers?	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Yes, score this benchmark	Gurobi
10	6/4/2024 4:26:30	philipp.kern@kit.edu	DPNeurifyFV	https://github.com/phK3/DPNeurifyFV.jl/	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	Yes, score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Yes, score this benchmark	No, do not score this benchmark	Gurobi, LazySets.jl, DNNV
11
12	Votes Yes				1	2	3	8	5	7	8	5	3	3	5	5	9	1	6	6	1	5	1	8	3	4	2
13	Votes No				8	7	6	1	4	2	1	4	6	6	4	4	0	8	3	3	8	4	8	1	6	5	7
14	Number Tools	9			9	9	9	9	9	9	9	9	9	9	9	9	9	9	9	9	9	9	9	9	9	9	9
15	Percent Yes				0.1111111111	0.2222222222	0.3333333333	0.8888888889	0.5555555556	0.7777777778	0.8888888889	0.5555555556	0.3333333333	0.3333333333	0.5555555556	0.5555555556	1	0.1111111111	0.6666666667	0.6666666667	0.1111111111	0.5555555556	0.1111111111	0.8888888889	0.3333333333	0.4444444444	0.2222222222
16	Percent No				0.8888888889	0.7777777778	0.6666666667	0.1111111111	0.4444444444	0.2222222222	0.1111111111	0.4444444444	0.6666666667	0.6666666667	0.4444444444	0.4444444444	0	0.8888888889	0.3333333333	0.3333333333	0.8888888889	0.4444444444	0.8888888889	0.1111111111	0.6666666667	0.5555555556	0.7777777778
17
18	Regular Track				FALSE	FALSE	FALSE	TRUE	TRUE	TRUE	TRUE	TRUE	FALSE	FALSE	TRUE	TRUE	TRUE	FALSE	TRUE	TRUE	FALSE	TRUE	FALSE	TRUE	FALSE	FALSE	FALSE
19	Extended Track				TRUE	TRUE	TRUE	FALSE	FALSE	FALSE	FALSE	FALSE	TRUE	TRUE	FALSE	FALSE	FALSE	TRUE	FALSE	FALSE	TRUE	FALSE	TRUE	FALSE	TRUE	TRUE	TRUE
20
21	Regular Track Benchmarks
22	Benchmark Voting [https://github.com/ANTONIONLP/safeNLP]	Benchmark Voting [https://github.com/Khoury-srg/VNNComp23_NN4Sys and https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/nn4sys]	Benchmark Voting [https://github.com/kollerlukas/cora-vnncomp2024-benchmark]	Benchmark Voting [https://github.com/aliabigdeli/LinearizeNN_benchmark2024]	Benchmark Voting [https://github.com/wu-haoze/dist-shift-vnn-comp and https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/dist_shift]	Benchmark Voting [https://github.com/huanzhang12/vnncomp2024_cifar100_benchmark]	Benchmark Voting [https://github.com/huanzhang12/vnncomp2024_tinyimagenet_benchmark]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/acasxu]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/cgan]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/collins_rul_cnn]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/metaroom]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/tllverifybench]
23	https://github.com/ANTONIONLP/safeNLP	https://github.com/Khoury-srg/VNNComp23_NN4Sys and https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/nn4sys	https://github.com/kollerlukas/cora-vnncomp2024-benchmark	https://github.com/aliabigdeli/LinearizeNN_benchmark2024	https://github.com/wu-haoze/dist-shift-vnn-comp and https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/dist_shift	https://github.com/huanzhang12/vnncomp2024_cifar100_benchmark	https://github.com/huanzhang12/vnncomp2024_tinyimagenet_benchmark	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/acasxu	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/cgan	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/collins_rul_cnn	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/metaroom	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/tllverifybench
24
25	Extended Track Benchmarks
26	Benchmark Voting [https://github.com/AI4OPT/ml4acopf_benchmark]	Benchmark Voting [https://github.com/dynaroars/vnncomp-benchmark-generation]	Benchmark Voting [https://github.com/loonwerks/vnncomp2023]	Benchmark Voting [https://github.com/shizhouxing/LSNC_VNNCOMP2024]	Benchmark Voting [https://github.com/xiangruzh/Yolo-Benchmark and https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/yolo]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/cctsdb_yolo]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/collins_yolo_robustness]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/ml4acopf]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/traffic_signs_recognition]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/vggnet16]	Benchmark Voting [https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/vit]
27	https://github.com/AI4OPT/ml4acopf_benchmark	https://github.com/dynaroars/vnncomp-benchmark-generation	https://github.com/loonwerks/vnncomp2023	https://github.com/shizhouxing/LSNC_VNNCOMP2024	https://github.com/xiangruzh/Yolo-Benchmark and https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/yolo	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/cctsdb_yolo	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/collins_yolo_robustness	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/ml4acopf	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/traffic_signs_recognition	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/vggnet16	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/vit
28
29
30	Regular Track Benchmarks
31	https://github.com/ANTONIONLP/safeNLP
32	https://github.com/Khoury-srg/VNNComp23_NN4Sys and https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/nn4sys
33	https://github.com/kollerlukas/cora-vnncomp2024-benchmark
34	https://github.com/aliabigdeli/LinearizeNN_benchmark2024
35	https://github.com/wu-haoze/dist-shift-vnn-comp and https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/dist_shift
36	https://github.com/huanzhang12/vnncomp2024_cifar100_benchmark
37	https://github.com/huanzhang12/vnncomp2024_tinyimagenet_benchmark
38	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/acasxu
39	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/cgan
40	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/collins_rul_cnn
41	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/metaroom
42	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/tllverifybench
43
44	Extended Track Benchmarks
45	https://github.com/AI4OPT/ml4acopf_benchmark
46	https://github.com/dynaroars/vnncomp-benchmark-generation
47	https://github.com/loonwerks/vnncomp2023
48	https://github.com/shizhouxing/LSNC_VNNCOMP2024
49	https://github.com/xiangruzh/Yolo-Benchmark and https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/yolo
50	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/cctsdb_yolo
51	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/collins_yolo_robustness
52	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/ml4acopf
53	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/traffic_signs_recognition
54	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/vggnet16
55	https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/vit
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100