API testing tool decision matrix

	A	B	C	D	E	F	H	I	J	O	P
1	Criterion/Features	Importance (Corey)	Importance (Daniel)	Importance (all)	Phester (MVP)	Phester (impl cost)	tavern	behat	codeception	SuperTest	People find this important

2	runtime [?!]				PHP	PHP	Python	PHP	PHP	node.js
3	execution model [run from CLI]		-		CLI	CLI	CLI	CLI	CLI	CLI (also Browser?)
4	ease of running locally [Run Tests]	1	5		1	1	2	1	1	1	wd,jr
5	ease of running in CI [CI integration]	1	4		2	2	2.5	2	2	2	jr
6	test language		-		YAML	YAML	YAML	cucumber-esque	php (fluent)	JS (fluent)
7	ease of editing [familiar language]	1	4		2	2	2	2	1	1
8	ease of migration [avoid vendor lock-in]	1	3		2	2	2	2	1	1
9	scope/purpose [functional fit]	1	4		2	2	2	1	0	2
10	IDE integration [IDE integration]	1	3		-2	-2		1	2	2	kh, CC
11	Debugging	1									kh
12
13	documentation [documentation and support]	1	3					2		0
14	license model		-		GPL	GPL	MIT	MIT	MIT	MIT
15	license sympathy [FLOSS]	1	3		2	2	2	2	2	2
16
17	structural body matches [Validate Responses]	1	5		2	2	2	2	2	2
18	regex matches [Validate Responses]	1	4		2	2	2	2	1.5	2	wd
19
20	variables [chaining]	1	5		-	2	1	1	2	2
21	variables in keys [chaining]	1	5		-	1		1	2	2
22	variables from config [configuration]	1	5		-	2	1	1	1.5	2
23	randomized variables [unique fake values]	1	4		-	2		1		2
24	string interpolation or templates [chaining]	1	2		-	1	2	0		2
25
26	cookies [control HTTP requests]	1	5		-	1	1	1		2
27	headers [control HTTP requests]	1	5		2	2	1	1		2
28
29	global fixtures [fixtures]	1	5		-	0	1	1	?	1	wd
30	re-usable resources [resources]	1	4		-	0	0.5	0		2
31	re-usable sessions [Multiple Agents]	1	5		-	1	0	1	1	2
32
33	run all files in a directory [Run all Tests]	1	5		-	2	2	2	?	2
34	filter tests by tag [Run Tests Locally]	1	4		-	2	-1	2	2	0	wd
35
36	built-in parallel execution [Parallel Testing]	1	3		-	0	2	1	0.5	1.5	wd
37	run tests that span multiple sites [Multi-Site Tests]	1	5		-	2				1
38	support data providers [Parametrized Tests]	1	3			1		0		2
39		1
40	CI Integration Effort [CI integration]	-3	-4		-	2		1		0
41	Containerisation Effort [Run Tests Locally]	-3	-4		-	2		1		0
42	Cost of opertational responsibility [?!]	-5	-4		-	2		0		0	wd
43	Cost of code ownership [maintenance burden]	-5	-5		-	2	1	0	1	1	wd, CC
44	Delay to Decoupling Project Start [?!]	-5	-4		-	2		0		0
45	Standard Integraton Testing Feature completeness [?!]	-5	???			2		0
46	Dual-use as monitoring tool [Monitoring]		2			2				1.5
47
48	cost to modify/maintain [maintenance burden]	3	-5		2	2	1	1		1
49	stability/support [documentation and support]	2	3		1	1	0	2	2	2
50	control over development [?!]	1	3		2	2	0	2	0	0
51
52	SCORE				0	0	0	0	0	0
53	SCORE (Daniel)				62	100	99	123	91	167.5
54	SCORE (Corey)				25	-10	25	33	20.5	42.5
55
56	Scoring howto: Features/abilities have a positive weight (i.e. importance), costs/risks/liabilities have a negative weight. The range of weights is 0 to 5. Scores in individuall cells have a range from -2 to +2. +2 indicates "a lot" or "very well", 0 indicates "so/so", and -2 indicates "not at all" or "actively resists". Scores for features should combine out-of-the-box and evetual fit: e.g. if a feature is available out of the box it's a 2, if it takes some effort to build it's a 0, and if it would be painful or impossible it's a -2. Negative scores for costs/risks don't make much sense.
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100