The Role of Justice in Development:

The Data Revolution

​

​

Daniel Chen and Manuel Ramos-Maqueda

Office of the Chief Economist: Middle East and North Africa

The World Bank

March 2022

The Data Revolution

1. The power of data in governments
1. Digital transformations & data availability
2. Opportunities for justice systems in the context of COVID-19
2. How can we leverage the potential of data in justice?
• A data ecosystem for better efficiency, quality, and access to justice
• Examples from DIME’s De Jure Program
3. Why does this matter?
• The role of justice in development: Summary of the evidence
4. Areas for future research & policy
• New avenues for research
• Policy implications

​

​

The power of Data

• The power of data in governments
• Digital transformations & data availability
• Opportunities for justice systems in the context of COVID-19
• How can we leverage the potential of data in justice?
• A data ecosystem for better efficiency, quality, and access to justice
• Examples from DIME’s De Jure Program
• Why does this matter?
• The role of justice in development: Summary of the evidence
• Areas for future research & policy
• New avenues for research
• Policy implications

​

​

Digital technologies in government

• Digital technologies are transforming the work of governments, accelerated by the COVID-19 pandemic
• The World Development Report (2016) notes that “the dramatic decline in the price of digital technologies has led businesses and governments to replace existing factors—labor and non-ICT capital—with ICT capital and to automate some of their activities”
• Digital systems generate a wealth of microdata that is under-utilized
• Tax: electronic filing of taxes; electronic billing
• Justice: eCourts
• Procurement: e-procurement
• These technologies allow vast data generation that creates valuable opportunities to measure and improve performance

Leveraging investments in justice data

At DE JURE, we ask: how can we leverage the potential of rich administrative data to improve justice institutions? And what’s the downstream impact of improving these institutions on poverty reduction and economic growth?

Judiciaries collect vast amounts of data, yet they rarely use it to improve decision-making; they are data-rich but information-poor.

Through the use of cutting-edge research techniques, we leverage data to improve the efficiency, quality and access to justice, and measure impacts on downstream outcomes such as economic growth, conflict and violence, and corruption.

Data

Analysis

Experimentation

• Improving case management systems

• Creating structured data and integrating with administrative data ecosystems

• Developing open-source platforms for data collection

• Identifying core issues in justice system functioning and performance

• Measuring and documenting impacts of justice reforms

• Leveraging machine learning to identify and reduce biases

• Testing new approaches to improve justice systems

• Implementing RCTs to assess interventions

• Creating policy feedback mechanisms

Harnessing the potential of data in justice

A global problem: According to the World Justice Project (2019), 5.1 billion people lack meaningful access to justice, while an estimated 1.5 billion are unable to obtain justice for administrative, criminal, or civil justice needs.

What if we could harness the power of high-frequency data to…?

• Identify precisely what sectors of the population lack access to justice
• Diagnose the exact reasons why cases get backlogged
• Incentivize judges in order to increase their efficiency
• Study the impact of justice on firms’ outcomes
• Target investments in judicial reforms to the most promising regions
• Diagnose and address biases in judicial decisions
• Evaluate the specific impact of any new law on citizens and businesses
• Highlight and share good practices in laws and regulations across countries

The Data Revolution

1. The power of data in governments
1. Digital transformations & Data Availability
2. Opportunities for justice systems in the context of COVID-19
2. How can we leverage the potential of data in justice?
• A data ecosystem for better efficiency, quality, and access to justice
• Examples from DIME’s De Jure program
3. Why does this matter?
• The role of justice in development: Summary of the evidence
4. Areas for future research & policy
• New avenues for research
• Policy implications

Building a data ecosystem

Integrated justice system

• Case data is associated with each milestone in the process, from filing to appeal
• Digitized the text of court rulings and sentences
• Integrate HR data, eg. recruitment, training, transfers
• Automate the creation of indicators associated with the results framework
• Tracking of court-level reforms and innovations & A/B testing

​

Interoperable system with related agencies

• Link case data to firms and users, e.g., through national ID/tax data
• Link case data to other stages of the process, e.g., police, prosecutors, and prisons

​

System that combines the experience of the user, and of the non-user too!

• Combine user surveys with those of judges and court employees
• Link legal needs questionnaires to understand who is not accessing justice and why

How are data-driven interventions improving justice systems?

Judicial Training & Integrity

Case study of Kenya: Taking Science to the Judiciary

OBJECTIVES:

• Specific objectives
• Real-time measurement of performance

INCENTIVES:

• Rewards for courts which perform best
• Performance metrics to evaluate promotion

​

INVESTMENT:

• Data used to justify transfers, hiring & investments
• Request for resources based on quantifiable metrics

Benefits of improved data management:

Kenya: Algorithms identify the greatest sources of court delays

(chemin et.al, forthcoming)

Context:

• Data not being used for decision-making
• Adjournments causing large backlogs
• We leverage data systems to diagnose key metrics and identify top 3 reasons for adjournments in each court

Study design: randomized controlled trial across all 124 court stations in Kenya

• Treatment 1 - Actionable Information: feedback report shared with judges
• Treatment 2 - Information + Accountability: report also shared with Court User Committees

​

​

Kenya: Algorithms identify the greatest sources of court delays

(chemin et.al, forthcoming)

TAKEAWAY 1:

• Providing actionable information (with accountability) improves efficiency
• Reduced adjournments by 20%

Kenya: Algorithms identify the greatest sources of court delays

(chemin et.al, forthcoming)

TAKEAWAY 2:

• More efficient judiciaries improve citizen’s economic outcomes
• Contract-intensive industries saw 23 percent increase in wages

Kenya: smart assignment of cases to mediators

DIGITIZE

MANAGE

TRACK

OPTIMIZE

• ICT platform for mediator and case management​

​

• AI integrated: Smart Assignment of cases to mediators based on experience & skills

​

• Real-time performance management and alerts

​

• Open source no-code tools for data entry and decision-support

Open-source decision support

Kenya: Diagnosing disparities – gender and ethnic biases in decisions

(chen et.al, forthcoming)

• Data: full texts of 130,000 published judgments from 1978 to 2021.
• We leverage data systems to diagnose biases along gender and ethnic lines

​

​

Judges are more favorable to defendants of their own gender and ethnicity

Judges who exhibit stereotypical gender biases in their writings are more likely to rule against female defendants

The Data Revolution

• The power of data in governments
• Digital transformations & data availability
• Opportunities for justice systems in the context of COVID-19
• How can we leverage the potential of data in justice?
• A data ecosystem for better efficiency, quality, and access to justice
• Examples from DIME’s De Jure program
• Why does this matter?
• The role of justice in development: Summary of the evidence
• Areas for future research & policy
• New avenues for research
• Policy implications

​

Key Findings

• Judiciary matters: for its intrinsic value and for development outcomes.
• Macro literature: institutions → key drivers of economic growth
• Rodrik (2000): institutions, rather than geography or openness to trade, are the key drivers of economic development
• Acemoglu, Johnson, and Robinson (2001):
• However, Aboal et.al (2014): systematic review found weak empirical evidence of the impact of contract enforcement on investment
• Is the relationship causal? And if so, why? Empirical literature:
• Better contract enforcement reduces risk in contracting + enhances firm’s productivity + entrepreneurship
• Judicial institutions increase credit availability & development of credit markets, by reducing risk (moral hazard)
• Access to justice increases productivity and protects citizens in cases where informal institutions are biased against them
• Lower corruption (limited evidence)

​

3A. Effective contract enforcement enhances firm productivity and economic growth

Contract enforcement promotes firm growth:

• If weak, affects firms’ incentives to invest and distorts production decisions

Shvets, 2012; Chemin 2009

• Security of land tenure and productive investments

Chari et.al, 2017; Aberra and Chemin, 2018

​

Court quality and efficiency affect access to credit markets:

• ↓ moral hazard + Securing firms against defaulting banks

IMF 2014; Chemin 2009

• Increasing creditors’ willingness to lend

Shvets, 2013

Evaluation of Pakistan's "Access to Justice Programme"

A judicial reform that provided judges with more training in Pakistan led to greater disposition of cases and higher entry rates of new firms. Estimates suggest that this reform increased Pakistan’s GDP by 0.5% [Chemin, 2009]

Efficient judiciaries increase entrepreneurship:

The creation of special tribunals in Brazil, which increased access to justice and its efficiency, led to increased entrepreneurship and new businesses (Lichand and Soares, 2011)

3B. Justice systems may reduce violent resolution of disputes and protect vulnerable populations

Justice systems may reduce violent resolution of disputes:  

• ↑ resolution of land disputes and ↓ violence through ADR

Blattman et.al, 2016

• ↓ domestic violence and ↑ human capital investments

Kavanaugh et.al, 2018; Owen and Portillo, 2003

• Deterring dishonest and criminal acts

Mocan et.al, 2018

​

Justice systems protect vulnerable populations:  

• Protecting vulnerable populations against bias

Sandefur and Siddiqi, 2015

• Stronger property rights reduce HIV by allowing women to negotiate safe sex practices

Anderson, 2018

• Legal aid for women reduce GBV & improves children’s education

Sviatschi and Trako, 2021

​

​

​

​

​

ADR reduces violent resolution of disputes:

Mass education campaigns that promote alternative dispute resolution (ADR) in rural communities of post-conflict Liberia led to higher resolution of land disputes and lower violence in the resolution [Blattman et.al, 2016]

Legal aid for women at risk may reduce GBV:

In Peru and Ecuador, legal aid for vulnerable women reduced gender-based violence, increased children's educational outcomes and reduced child labor [Sviatschi and Trako, 2021]

3C. Efficient and accountable justice systems may reduce corruption and enhance citizens’ trust

Judicial accountability promotes anti-corruption and citizen trust:  

• Reducing rent extraction

Litschig, 2015

• ↑ effectiveness of anti-corruption audits, as it ↑ the non-electoral costs of corruption

Avis et.al, 2017

• Improving citizens’ trust in courts and institutions

Acemoglu et.al, 2018

• Reducing bias in judicial decisions

Mehmood et.al, 2018

• Reducing procedural formalism

Djankov et.al, 2003

20

Efficient judiciaries enhance trust in institutions

Providing information about reduced delays in state courts in rural Panjab (Pakistan) led to higher willingness by citizens to use state courts, willingness to allocate greater funds to the state, and greater trust overall in formal institutions [Acemoglu et.al, 2018]

Judicial presence may reduce corruption

In Brazilian local governments, state judiciary presence reduces rent extraction (measured as waste or corruption) by about 10 percent or 0.3 standard deviations.

[Litschig and Zamboni, 2015]

The Data Revolution

• The power of data in governments
• Digital transformations & data availability
• Opportunities for justice systems in the context of COVID-19
• How can we leverage the potential of data in justice?
• A data ecosystem for better efficiency, quality, and access to justice
• Examples from DIME’s De Jure program
• Why does this matter?
• The role of justice in development: Summary of the evidence
• Areas for future research & policy
• Policy implications
• New avenues for research

​

Data for Better Judiciaries

Key requirements: creating an interoperable data ecosystem & investing in human capital

​

Areas to make better use of data in judicial systems:

1. Access to e-justice for citizens
1. Technologies to match citizens & lawyers or justice systems, (e.g. Whatsapp access to courts)
2. Apps for “missing cases”
2. Improving court performance
• AI as a support tool for justice actors
• Identifying & addressing biases and inconsistencies
3. Enhancing knowledge
• Tools to improve the training of legal actors
• Legal search engine & access to knowledge

1. access to e-justice for citizens: Whatsapp access to courts (Chile)

Enables communication via audio & video, linking cases with documents through digital interoperability

1. BANGLADESH: Increase Access to Helplines/Support Services for Gender-Based Violence Survivors (in-progress)

​

​

​

• Lockdowns and social distancing measures have increased experiences of gender-based violence (GBV) in developing countries, particularly in Bangladesh.

​

• Our team plans to evaluate whether Beiwe, a revolutionary digital phenotyping platform, can identify individuals at-risk for GBV and connect them to helplines and support services.

​

• Phenotyping can provide recognition, cognitive analytics, behavioral anomaly detection, social analytics, and biomarker analytics - potentially maximizing prevention and surveillance of at-risk individuals.

​

​

​

2. Machine learning to predict judicial decisions

25

• In the United States, foreign nationals who fear persecution in their home country can apply for asylum under the Refugee Act of 1980

​

• The project uses a large data set secured under a Freedom of Information Act (FOIA) request detailing United States Asylum court decisions going back multiple decades and machine learning to predict the outcomes of the cases

​

• We are able to predict the final outcome of a case with 80% accuracy at the time the case opens. Using data available on the decision date, our model correctly classifies 82% of all refugee cases by 2019

Using machine learning to predict the outcomes of United States Asylum court decisions

3. Human-centric AI in justice

26

• Reduces pendency of cases and increases the efficiency of courts by facilitating the optimal assignment of cases to judges in courts where there are backlog and delays

​

• The App follows a similar model to Uber, where the “drivers” are judges and the “customers” are case administrators, or those who manage and assign cases

​

• Instead of using “smart” assignments that do not incorporate judges’ own preferences and information about their own skills, human centric AI assumes that every agent/judge has private information and reacts to incentives. Given their information advantage, it is often better to let a judge make their own decisions. Importantly, carefully designed incentives can help shape a judge’s decision to optimize outcomes..

Increasing Access to Justice through AI Integrated Technology

​

27

AI enabled method of personalized case-based teaching using the history of a judge’s past decisions.

• Leverages the history of the judge’s own written decisions to evaluate how such judge would decide on a similar case in comparison to a curricular example or to other similar decisions by peer judges

​

• Brings case-based teaching to the next level, providing users with a personalized training that has the potential to improve the quality and efficiency of judicial training and judicial decisions

​

• Help create culture of precedent

4. Enhancing knowledge: Personalized case-based teaching tools for court actors

5. Enhancing knowledge: creating jobs by helping SMEs understand labor law (Bertrand et.al, 2021)

Perception that it is more difficult to dismiss an employee in South Africa than virtually anywhere else in the world → due to labor law; “Unjustified”

​

Question: does imperfect knowledge of labor regulation hinders job creation in SMEs?

​

RCT across 1824 SMEs to test the impact of improving knowledge of labor laws: a 21-week membership to a labor law club+ newsletters on labor law and HR management + access to a website

Treatment effect: a 11.8 worker increase in employment, or 12-15% increase relative to the control mean

​

Most of this increase for permanent and fixed term workers, rather than casual workers

6. Enhancing knowledge: Information and mediation to improve labor courts (Sadka et.al, 2017)

Context:

• Overburdened labor cours & low-quality legal representation
• Legal fees often > what plaintiffs recover

​

Intervention:

• Information on predicted case outcomes to workers + asking them to meet with court conciliators (mediators)

​

Result:

• Double settlement rates and reduced case duration, but only when the worker is present to receive the information.
• Welfare improvements

New avenues for research

1. Trade-offs between quality, efficiency and access & across actors (policymakers, lawyers and citizens) with different preferences
2. Access to justice in the context of fragility, conflict, and violence
3. Potential of a stronger judiciary to reduce corruption

​

Challenges:

1. Need for greater diversity in study sites
2. Conundrum: countries that stand to gain the most from data-driven technologies lack the resources and data infrastructure to take advantage of them

Appendix

Smart assignment: Can AI-driven recommendations enhance access to mediation and decongest courts in KEnya?

32

Analytical contributions:

• Developing a case management system (“Cadaster”) to measure improve mediation processes
• Leveraging AI-driven recommendations of mediators to cases to improve mediation, increase access to justice and decongest courts

​

Can innovative algorithms to assign cases to the optimal mediator improve mediation outcomes and increase user satisfaction?

​

​

Also in Peru, we have developed an interactive app that leverages the existing, underutilized data systems

ML predicts judicial decisions by legally irrelevant factors (Chen and Eagel, 2017)

Judges deny refugees asylum when the weather is too hot or too cold

The Impact of Effective Contract Enforcement on Firm Growth

1. Weak contract enforcement hinders firm-to firm trade.
1. Legal reform to speed up trials in courts led to fewer breaches of contract in India (Chemin, 2012)
2. Weak enforcement affect firms’ decision to invest and distort production decision (Klein et al., 1978; Chakraborty et al., 2018)
1. An increase in input tariff liberalization lead firms to gain more productivity if located in high judicial efficient regions (Ahsan, 2013)
2. In states with weaker contract enforcement, production and sourcing decisions appear systematically distorted (Boehm and Oberfield, 2018)
3. Slow enforcement increases the opportunistic behavior of borrowers (Chemin, 2012; Jappelli, Pagano, and Bianco, 2005).

Recent innovations have opened up new opportunities for delivery of justice

• Increasingly digitized large-scale datasets

​

• ML applications to produce interpretable data from unstructured text

​

• Predictive models of decision-making to better understand biases and address them with digital interfaces

efficiency & Quality of courts leads to better credit markets, entrepreneurship and healthier business environments

1. There is a positive relationship between better judicial institutions and better credit market.
• Provinces with poor legal enforcement/longer trial durations have less developed credit markets and and credit availability (Christini et al., 2001, Jappeli et al., 2005)
• Firms are more likely to firms to default against banks when legal enforcement is weak (Ponticelli et.al, 2016; Schiantarelli et.al, 2016)
• Slower judiciaries reduce access to credit markets and particularly affect contract-intensive industries (Chemin, 2009)
• Creditors are more willing to lend when their rights are protected by courts with lower appeal rates (Shvets, 2013)
2. Better justice leads to entrepreneurship
• The creation of special tribunals in Brazil increased new business by 10% in 10 years (Lichand and Soares, 2011)
1. An innovative delay reduction program in Pakistan had a positive effect on entrepreneurship in pilot districts (Chemin, 2009)
2. Free legal aid to farmers facing land disputes substantially increased productivity, investment and agricultural production (Aberra and Chemin, 2018)
3. Better legal systems enhance business environment
1. Better quality legal system had positive effects on firms size (Laeven and Woodruff, 2004)
2. A simple procedural reform to reduce the duration of trials lead to positive firm-level effect (Kondylis and Stein, 2018)

Protecting vulnerable populations

The benefits of justice can only be realized if citizens and firms have sufficient access to justice. According to the World Justice Project (2019), 5.1 billion people–approximately ⅔ of the world’s population–lack meaningful access to justice

1. Alternative Dispute Resolution (ADR)
1. The Liberian communities with mass educational campaigns on ADR had higher resolution of land disputes and lower violence (Blattman et al., 2014)
2. Strong rule of law improves vulnerable population’ well-being and health
• Judicial mechanisms ensuring the enforcement of property rights enable women to negotiate safe sex practices with strength and lessen their risk of contracting HIV (Anderson, 2018)
3. Reducing barriers of access to justice
• Economically disadvantaged women accessing legal aid clinics in Ecuador were more likely to receive an award in favor and payment for children (Owen and Portillo, 2003)
4. Reducing procedural formalism and technicality
• In India, lower caste and female entrepreneurs gain more from better judicial enforcement of contract rights because informal dispute resolution mechanisms are dominated by upper-caste men (Chakraborty et al., 2018)

Interventions Involving Disadvantaged Groups

1. Strong rule of law improves vulnerable population’ well-being and health
• Judicial mechanisms ensuring the enforcement of property rights enable women to negotiate safe sex practices with strength and lessen their risk of contracting HIV (Anderson, 2018)
2. Reducing barriers of access to justice
• Economically disadvantaged women accessing legal aid clinics in Ecuador were more likely to receive an award in favor and payment for children (Owen and Portillo, 2003)
• Providing free legal aid to police detainees in Sierra Leone release of detainees without charge increased by 13% and the share of inmates held on remand reduced by 20%
• Women Justice Centers in Peru increased reporting and prosecutions for gender-specific crimes by 40% and reduced gender-based violence by 10%. WJCs also increased human capital investments and academic results of children
3. Protecting disadvantaged groups
• In India, lower caste and female entrepreneurs gain more from better judicial enforcement of contract rights because informal dispute resolution mechanisms are dominated by upper-caste men (Chakraborty et al., 2018)
• In Liberia, litigants in lower social positions formal forums for better protection of their rights
4. Promoting trust in institutions