ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
Judicial AI Assessment Tool for Courts
2
IV. Ongoing/Continuous Assessment
3
4
This workbook is designed to assist courts in supervising AI systems and keeping them in check to mitigate harms over time
5
As the decision-making process has been completed at this stage, this tool is designed as a simple yes/no checklist meant to assist the court in monitoring the performance of the AI tool and identifying and effectively resolving any lags or shortcomings to be collaboratively addressed with their solution providers
6
7
Rationale for this tool:
8
Courts must be prepared to seek information and/or assistance from technical partners in developing protocols to monitor the performance of tools over time, such as traditional KPIs to track how AI tools improve, adapt, or degrade. This could include evidence of performance under both controlled testing environments and real-world pilots, while acknowledging that actual courtroom conditions can be complex and differ from controlled settings. This tools asks courts to additionally plan for safety stop mechanisms, when to permit models' use of reinforcement learning, recognising and addressing instances of overfitting or underfitting, and approving automatic feedback or retraining loops.
9
10
Instructions:
11
The assessment tools may be used either when a single vendor is under consideration for a particular use case or when the court is choosing among multiple vendors. They are intended to help courts evaluate their institutional capacity, assess potential risks to rights arising from use of the AI tool, and identify mitigation strategies to be put in place by the vendor.

We recommend courts begin with the institutional readiness assessment. Courts that score optimally, reflected in positive response to all essential questions a score of at least 60%, may proceed to the next step. If the score falls below this threshold, the court must remedy limitations before proceeding to the risk assessment. The risk assessment enables courts to classify the intensity of risk associated with the tool. The subsequent action should be calibrated accordingly. For low-risk tools, deployment may proceed with basic safeguards in place. Medium-risk tools should trigger non-negotiable assurances from vendors in the technical assessment. In the case of high-risk tools, we recommend that courts undertake the full technical assessment and proceed only in case of a reasonably high score (>=60%). In case of high risk with prohibition, courts should refrain from use AI, as the risks outweigh potential benefits.

Finally, once AI is deployed and ready for adoption, this monitoring assessment tool provides supervisory questions to keep the AI system in check and mitigate harms over time.
12
Where possible, the tools should be filled up by designated staff with technical background/capabilities who report to High Court IT/Computer Committees
13
1. As a pre-requisite, inform the vendor / service provider that the court will require access to:
Documentation of baseline measurements (current state without AI)
14
Success metrics
15
Datasheets
16
Model cards
17
Performance evaluation benchmarks
18
Performance logs
19
2. Set up a periodic assessment cycle in consultation with technical staff and the vendor / service provider
20
3. The questions are formulated as a checklist. If the calibration indicates 'Proceed', the court is in the right track in monitoring the AI tool and detecting defects and errors in a timely manner.
21
4. If the calibration indicates 'Action Required', the court must seek a suitable response in due time from the vendor. If 'Action required' is from court's side, it is imperative the AI tool be paused and limitations rectified before proceeding.
22
23
To access the full report on the UNDP website - click here or scan the QR code below:
24
#VALUE!
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100