1 of 35

Systematic Content Analysis of Litigation EventS

S

C

A

L

BLOCKCHAIN

2 of 35

🤷‍♀️ �Insight from millions of court records is functionally inaccessible.

The problem:

$.10/page + nonuniform local practices + no meaningful bulk access

free, but limited in scope or coverage

costly + coverage often unclear + no bulk access

x hundreds of state and local courts

BLOCKCHAIN

3 of 35

SCALES aims to bring transparency

to the systems and processes

of the U.S. courts,

ensuring that they are

fair, efficient, and accurate

BLOCKCHAIN

4 of 35

SCALES team

SOCIAL

SCIENCE

ENGINEERING

JOURNALISM

LAW

BLOCKCHAIN

5 of 35

Interdisciplinary Fruits

BLOCKCHAIN

6 of 35

Mission

Focus on what impacts new research initiatives could have in the justice system
Provide a space for interdisciplinary collaborations to form
Grow data-driven practices of assessing the justice system

BLOCKCHAIN

7 of 35

What SCALES is

�An “open” knowledge network (OKN) would be available to all stakeholders, including the researchers who will help push this technology further. An OKN requires a nonproprietary, public–private development effort that spans the entire data science community and results in an open, shared infrastructure.

BLOCKCHAIN

8 of 35

�

BLOCKCHAIN

9 of 35

What SCALES does

Platform enables systematic analysis of court records

BLOCKCHAIN

10 of 35

What SCALES has

Docket sheets for all criminal and civil cases filed in 2016 and 2017 in all 94 U.S. district courts
Docket sheets for all criminal and civil cases filed in 2002 – March 2021 in the Northern District of Illinois

Downloaded as of Fall 2020

All documents filed in N.D. Ill. in 2016

~750,000+ civil and criminal docket sheets

Court docs from ~12,800+ cases

BLOCKCHAIN

11 of 35

SCALES tools

https://github.com/scales-okn

Software automatically runs PACER in a browser to download cases

Can scrape:

Queries
Dockets
Case summaries
Documents

BLOCKCHAIN

12 of 35

Data extraction and transformation

HTML

JSON

Extracts and keys:

Case header information (nature of suit, judge, filing dates)
Parties (including address) and charges
Lawyers (address, phone, pro hac status)
Docket entries and documents if requested

BLOCKCHAIN

13 of 35

Understand and enrich data

BLOCKCHAIN

14 of 35

Understand and enrich data

Entity Recognition & Disambiguation

Relation to case

Entity Type

BLOCKCHAIN

15 of 35

Judge recognition and disambiguation

IFP decisions: case study

Other settings,

e.g. bond decisions?

BLOCKCHAIN

16 of 35

Named Entity Recognition (NER)

Amy Joan St. Eve

Amy St. Eve

A.J. St. Eve

St. Eve

Honorable

District Judge

Judge

D.J.

159,000 name variants
40 honorific variants
10 million+ string combinations

Rules-Based Pruning

Order signed by District Judge Doe
Preliminary hearing held in chambers of Doe, District Judge. All parties advised...
Plaintiff Doe’s motion for extension of time.

BLOCKCHAIN

We set out to build a massive NER Training set using our corpus of over 300 thousand dockets. Our goal was to tag as many judges as we could within the free text of the dockets.

This task would obviously not be exhaustive; however, we were able to leverage the access to thousands of dockets to search for entities across the corpus.

In that prior example with Judge Jackson and Judge Dolor, we would be able to find other cases that were assigned to Judge Dolor and therefore identify them as a possible judge for extraction in free text.

With our corpus, we generated 159 thousand name variants, including initializations, name reversals, adding possessive clarifiers, adding suffixes, etc. and we coupled this with about 40 honorific variants to create over 10 million string combinations that could constitute reference to a judge.

Regular Expressions could not handle that scale at speed when searching through 300,000 documents. Instead we leveraged the FlashText algorithm, which employs trie-based pattern searches in free-text. We were able to consider 427 million tokens over 10.7 million text entries in just over 40 minutes

We then applied a rules-based pruning to the training data build to ensure false positives were not tagged. For example, if there was a Judge named Robert Doe, we would only extract his surname if it appeared next to honorifics or within his full name.

17 of 35

Custom NER pipeline

Precision compared to spaCy’s out-of-the-box NER model for 2017 dockets

PRESIDE

98.6%

spaCy en_core_web_lg

85.9%

using spaCy’s ‘PERSON’ tag

spaCy v.3 trained from scratch

BLOCKCHAIN

18 of 35

Lawyer and Law Firm Disambiguation

BLOCKCHAIN

19 of 35

Corporate Party Disambiguation

BLOCKCHAIN

20 of 35

Visualizing litigation (civil example*)

Complaint

Defendant’s MTD granted in full

Settlement

Discovery

Defendant’s MSJ

Defendant’s MSJ granted in full

Settlement: Consent Decree

Trial verdict: Plaintiff

Default judgment

Defendant’s MTD denied in full

Settlement

Defendant’s MTD granted in part

Settlement of remaining claims

*criminal in progress.

Defendant’s MTD

BLOCKCHAIN

21 of 35

Litigation event ontology

Goal: Enable users to answer the following types of questions*:

What events happen in litigation?
How do cases and claims conclude?
What is the pathway by which cases travel from beginning to end?

Plus layer in court, judge, party, lawyer, and claim attributes.

*Do better than the Federal Judicial Center’s Integrated Database

BLOCKCHAIN

22 of 35

MVP litigation events

Case beginning

Complaint
Notice of Removal
Writ of Habeas Corpus
Inbound transfer
Other

Answer

Discovery beginning

Scheduling conference?
Initial disclosures?
Work backward from MSJ?

Trial beginning

How to choose which events to include?

Balance comprehensive classification of each docket entry (“boil the ocean”) with MVP achievability.
Beginning 🡪 middle (few events) 🡪 end
Build in user search options as backstop.

BLOCKCHAIN

23 of 35

MVP litigation events (cont.)

Trial

Verdict (jury)
Findings of fact and conclusions of law (bench trial)

Party resolution

Settlement
Rule 68
Voluntary dismissal
Party-provided judgment, e.g. consent decree

Granted/partially granted dispositive motions

Motion to dismiss
Motion for judgment on the pleadings
Motion for summary judgment
Motion for a judgment as a matter of law

Default judgment
Terminating sanctions and sua sponte dismissals
Outbound transfer or remand

Dispositive events (claim, case, or party)

Notice of Appeal

BLOCKCHAIN

24 of 35

Mapping and modeling challenge

Default judgment

Complaint

1 to 1

x 94 U.S. district courts

Many to 1

https://huggingface.co/scales-okn

BLOCKCHAIN

25 of 35

Satyrn

BLOCKCHAIN

26 of 35

GUI

BLOCKCHAIN

27 of 35

Make analysis simple

BLOCKCHAIN

28 of 35

Available on Github

Published and available under GPL licensing

https://github.com/scales-okn

BLOCKCHAIN

29 of 35

Available on Github

Published and available under GPL licensing

https://github.com/scales-okn

General public license giving users 4 freedoms: to run, to study, to share, and to modify the software

BLOCKCHAIN

30 of 35

Documentation site

Available through SCALES main site: https://scales-okn.org/

BLOCKCHAIN

31 of 35

Case Study

How often do judges grant requests for in forma pauperis?

BLOCKCHAIN

Financial limitations prevent full and robust examinations of the records. During Phase I we have been working with the PACER Public User Group and the American Association of Law Librarians to provide our concrete example of what open knowledge in this arena could accomplish to bolster their efforts for structural reform in opening court records to the public. But we are not solely relying on these efforts, we have partnered with the Free Law Project—a non-profit with a browser extension to crowdsource the collection of court records—in order to leverage its partial coverage of the federal court records, thereby avoiding downloading duplicative records and incurring the associated access fees. We are also

We are also focusing on developing specific public-interest efforts that will serve as examples of the possible applications that SCALES OKN will be able to support by examining the courts’ current functions. Our work on analyzing variation in fee waiver granting is one example. We are creating additional examples, such as the variation in the sealing of court records and documents across the country, with our partners in Phase II. With our partner organizations we will present these data products to the district courts as tools to aid their decision making, and simultaneously to ask for fee exemption waivers to download court data to support the application’s continued utility. This is the same model that we have pursued with the Northern District of Illinois as a part of our fee waiver analysis, and that court is currently considering our request for no-cost data access.

32 of 35

Case Study

BLOCKCHAIN

Financial limitations prevent full and robust examinations of the records. During Phase I we have been working with the PACER Public User Group and the American Association of Law Librarians to provide our concrete example of what open knowledge in this arena could accomplish to bolster their efforts for structural reform in opening court records to the public. But we are not solely relying on these efforts, we have partnered with the Free Law Project—a non-profit with a browser extension to crowdsource the collection of court records—in order to leverage its partial coverage of the federal court records, thereby avoiding downloading duplicative records and incurring the associated access fees. We are also

We are also focusing on developing specific public-interest efforts that will serve as examples of the possible applications that SCALES OKN will be able to support by examining the courts’ current functions. Our work on analyzing variation in fee waiver granting is one example. We are creating additional examples, such as the variation in the sealing of court records and documents across the country, with our partners in Phase II. With our partner organizations we will present these data products to the district courts as tools to aid their decision making, and simultaneously to ask for fee exemption waivers to download court data to support the application’s continued utility. This is the same model that we have pursued with the Northern District of Illinois as a part of our fee waiver analysis, and that court is currently considering our request for no-cost data access.

33 of 35

Case Study: Fee Waiver

Nearly 40 percent of judges approve fee waivers at a rate that significantly differs from their peer judges across

all 94 districts

BLOCKCHAIN

Financial limitations prevent full and robust examinations of the records. During Phase I we have been working with the PACER Public User Group and the American Association of Law Librarians to provide our concrete example of what open knowledge in this arena could accomplish to bolster their efforts for structural reform in opening court records to the public. But we are not solely relying on these efforts, we have partnered with the Free Law Project—a non-profit with a browser extension to crowdsource the collection of court records—in order to leverage its partial coverage of the federal court records, thereby avoiding downloading duplicative records and incurring the associated access fees. We are also

We are also focusing on developing specific public-interest efforts that will serve as examples of the possible applications that SCALES OKN will be able to support by examining the courts’ current functions. Our work on analyzing variation in fee waiver granting is one example. We are creating additional examples, such as the variation in the sealing of court records and documents across the country, with our partners in Phase II. With our partner organizations we will present these data products to the district courts as tools to aid their decision making, and simultaneously to ask for fee exemption waivers to download court data to support the application’s continued utility. This is the same model that we have pursued with the Northern District of Illinois as a part of our fee waiver analysis, and that court is currently considering our request for no-cost data access.

34 of 35

Next Steps

BLOCKCHAIN

Financial limitations prevent full and robust examinations of the records. During Phase I we have been working with the PACER Public User Group and the American Association of Law Librarians to provide our concrete example of what open knowledge in this arena could accomplish to bolster their efforts for structural reform in opening court records to the public. But we are not solely relying on these efforts, we have partnered with the Free Law Project—a non-profit with a browser extension to crowdsource the collection of court records—in order to leverage its partial coverage of the federal court records, thereby avoiding downloading duplicative records and incurring the associated access fees. We are also

We are also focusing on developing specific public-interest efforts that will serve as examples of the possible applications that SCALES OKN will be able to support by examining the courts’ current functions. Our work on analyzing variation in fee waiver granting is one example. We are creating additional examples, such as the variation in the sealing of court records and documents across the country, with our partners in Phase II. With our partner organizations we will present these data products to the district courts as tools to aid their decision making, and simultaneously to ask for fee exemption waivers to download court data to support the application’s continued utility. This is the same model that we have pursued with the Northern District of Illinois as a part of our fee waiver analysis, and that court is currently considering our request for no-cost data access.

35 of 35

Next Steps

SCALES

ACCESS TO JUSTICE WORKSHOP

🡨 Ryan

BLOCKCHAIN

Financial limitations prevent full and robust examinations of the records. During Phase I we have been working with the PACER Public User Group and the American Association of Law Librarians to provide our concrete example of what open knowledge in this arena could accomplish to bolster their efforts for structural reform in opening court records to the public. But we are not solely relying on these efforts, we have partnered with the Free Law Project—a non-profit with a browser extension to crowdsource the collection of court records—in order to leverage its partial coverage of the federal court records, thereby avoiding downloading duplicative records and incurring the associated access fees. We are also

We are also focusing on developing specific public-interest efforts that will serve as examples of the possible applications that SCALES OKN will be able to support by examining the courts’ current functions. Our work on analyzing variation in fee waiver granting is one example. We are creating additional examples, such as the variation in the sealing of court records and documents across the country, with our partners in Phase II. With our partner organizations we will present these data products to the district courts as tools to aid their decision making, and simultaneously to ask for fee exemption waivers to download court data to support the application’s continued utility. This is the same model that we have pursued with the Northern District of Illinois as a part of our fee waiver analysis, and that court is currently considering our request for no-cost data access.