ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
 AI VENDOR ASSESSMENT FRAMEWORK - EVALUATOR'S GUIDE
2
 Practical parameter-by-parameter guide for assessing AI models including black-box models | ISO 42001 | EU AI Act | NIST AI RMF | OWASP LLM
3
 Note: WHY THIS GUIDE EXISTS: When a vendor says 'our AI is accurate' - what does that mean?
This guide translates black-box vendor claims into testable, evidence-based parameters. Use Tab 2 to score, Tab 3 for black-box specific tests, Tab 4 for governance alignment, Tab 5 for vendor comparison.
4
A. THE BLACK-BOX PROBLEM : WHAT YOU CANNOT SEE AND HOW TO EVALUATE IT ANYWAY
5
#What Vendor Hides in a Black BoxWhat You CAN Observe Without Model AccessTest / Evidence to DemandGovernance Reference
6
1Model architecture (layers, weights, training algorithm)Output consistency under identical inputs — does same input always produce same output?Run 50 identical test inputs — flag any non-deterministic outputs; ask for determinism policyISO 42001 A.6.2.3; NIST AI RMF MAP-2.2
7
2Training data — what data was used, from where, with what labelsModel card / data sheet published by vendor; data provenance statementRequest model card. If none exists — that is the finding. Demand: data sources, volume, labelling methodology, known gapsISO 42001 A.7.5; EU AI Act Art 10; NIST AI RMF MAP-2.3
8
3Bias in training data — which demographic groups are under/over-representedDisaggregated performance metrics across demographic groups (gender, age, geography, language)Demand performance report broken down by user subgroup. Run your own test dataset with known demographic diversityISO 42001 A.5.4; EU AI Act Art 10(2); NIST AI RMF MEASURE-2.5
9
4Model failure modes — what inputs cause wrong or dangerous outputsRed team / adversarial test results; documented failure modes in vendor documentationRequest adversarial robustness report. Run 20 adversarial inputs yourself during PoC. Ask: what are the top 3 known failure modes?ISO 42001 A.6.2.4; OWASP LLM01/LLM04; NIST AI RMF MEASURE-2.2
10
5Confidence calibration — is the model's stated confidence score accurateCalibration curve: when model says 90% confident, is it right 90% of the time?Ask for calibration data. Test 100 outputs — compare stated confidence vs actual accuracy. A well-calibrated model is rare; demand evidenceISO 42001 B.6.2.4; NIST AI RMF MEASURE-2.6
11
6Model drift — does performance degrade over time as real-world data shiftsDrift monitoring logs; performance trend reports; alert thresholds for retrainingAsk: how do you detect drift? What triggers retraining? Request last 12 months of performance trend dataISO 42001 A.6.2.6; NIST AI RMF MEASURE-4.1
12
7Retraining pipeline — when and how is the model updatedRetraining cadence documentation; change log; notification process when model version changesRequest retraining SLA. Ask: if the model is retrained and performance changes, how are customers notified?ISO 42001 A.6.2.5; NIST AI RMF MANAGE-4.2
13
8Third-party components — open-source models, libraries, pre-trained foundations embeddedSoftware Bill of Materials (SBOM) for AI components; licence compliance statementRequest AI SBOM. Check for GPL/AGPL licences in commercial use. Ask about foundation model (GPT, Llama, etc.) embeddedISO 42001 A.10.3; EU AI Act Art 25; OWASP LLM05
14
15
B. THE 12 PARAMETER DOMAINS - WHAT EACH MEASURES AND WHY IT MATTERS
16
#DomainWhat It MeasuresWhy It Matters for Black-Box AI SpecificallyWeight in ScorecardISO 42001EU AI ActNIST AI RMF
17
1Model Performance & AccuracyCan the model do the job it claims to do — measured on YOUR data, not vendor benchmarksVendor benchmarks are cherry-picked. A model that is 97% accurate on ImageNet may be 61% accurate on your specific use case. You must test on your own representative dataset.20%A.6.2.4Art 9, Art 13MEASURE-2.1 to 2.6
18
2Explainability & InterpretabilityCan the model explain WHY it produced an output — at a level usable by a business decision-makerFor black-box models, full mechanistic explanation is impossible. What you can evaluate: feature importance scores, counterfactual explanations, SHAP values — and whether these are actually useful to the people making decisions.12%A.8.2Art 13, Art 14MEASURE-2.6
19
3Bias & FairnessDoes the model produce systematically different — and unjustifiable — outcomes for different groupsBlack-box models cannot be inspected for bias internally. You must run controlled experiments with identical inputs that differ only on a protected characteristic and measure output differences statistically.12%A.5.4, A.7.4Art 10(2)MEASURE-2.5
20
4Data Governance & PrivacyWhat data does the model process, where does it go, who has access, and is your data used to train other customers' modelsThe most dangerous black-box risk. The vendor may be training on your confidential data to improve their shared model. This requires explicit contractual prohibition — technical controls alone are insufficient.12%A.7.2 to A.7.6Art 10, Art 28 GDPRMAP-2.2, MAP-2.3
21
5Security & Adversarial RobustnessCan the model be manipulated, poisoned, or evaded by a determined adversaryBlack-box does not mean attack-proof. Prompt injection, model extraction, and adversarial examples all work against models the attacker cannot inspect. Ask for adversarial robustness testing results.10%A.6.2.3, A.6.2.4Art 15MEASURE-2.2
22
6Operational Reliability & PerformanceDoes the model meet SLAs for availability, latency, and throughput under real-world loadAI models often degrade under load — inference latency spikes, batch jobs queue, GPU memory is constrained. Test at 2x your expected peak load before signing.7%A.6.2.6Art 9MEASURE-4.1
23
7Human Oversight & ControlCan a human review, override, or stop the model's outputs and actionsEspecially critical for agentic AI. A black-box model that takes autonomous actions with no override mechanism violates ISO 42001 A.9.2 and EU AI Act Art 14. This is non-negotiable for high-risk AI.7%A.9.2, B.9.3Art 14GV-6.2
24
8Vendor AI Governance MaturityDoes the vendor govern their own AI development with the same rigour they ask you to applyA vendor selling AI governance tools with no internal AI governance programme is a red flag. Ask for their own ISO 42001 posture, responsible AI policy, and internal model review process.5%Cl 5.2, Cl 9.3Art 9, Art 16GV-1.1, GV-1.2
25
9Transparency & DocumentationHas the vendor documented the model's purpose, limitations, and appropriate use cases in a form you can use for your own governanceISO 42001 requires organisations to maintain documentation about AI systems they use — including ones from third parties. If the vendor provides no documentation, your ISO 42001 SoA has a gap.4%A.6.2.7, A.8.2Art 11, Art 13MAP-5.1
26
10Supply Chain & Third-Party RiskWhat AI components, foundation models, and data sources are embedded in the vendor's modelA vendor using GPT-4 or Llama 3 as a backbone inherits those models' risks — jailbreaks, training data privacy concerns, OpenAI's terms of service. You need to know what is inside the stack.3%A.10.2, A.10.3Art 25GV-SC-1
27
11Regulatory Compliance PostureIs the vendor compliant with the regulations that apply to your use of their modelDifferent from their general security compliance. Specifically: is their model classified correctly under the EU AI Act? Do they have a conformity assessment if High-Risk? Can they support your DPDPA audit?3%Cl 9.2Art 9, Art 51GV-3.2
28
12Commercial & Contractual RiskDo the contract terms reflect the AI-specific risks — data training, liability for wrong outputs, exit rights, model version lock-inStandard SaaS contracts were not written for AI. The vendor's liability for a wrong AI output that causes a business loss is typically near-zero under standard terms. This must be negotiated.5%A.10.3, A.10.4Art 25, Art 28 GDPRGV-SC-3
29
30
→ TABS: 1.Evaluation Guide | 2.Scoring Matrix | 3.Black-Box Tests | 4.AI Governance Alignment | 5.Vendor Comparison
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100