Learning Algorithms and Forensic Evidence

Alicia Carriquiry

​

An important report

• Most forensic disciplines
• No scientific foundation
• Rife with subjectivity
• Pattern evidence are especially lacking in scientific validity
• Bite marks
• Hand-writing
• Firearms examination
• Shoe prints

The job of a forensic practitioner

• Is the suspect the source of the evidence found at the crime scene?
• Do two items have a common source?

​

​

Many challenges

• Except for DNA, forensic disciplines rely on subjective assessments.
• Forensic measurements do not lend themselves to the usual statistical modeling approaches.
• Highly multivariate, non-normal variables.
• Images with many thousands of pixels.

​

​

Machine learning is a good alternative

• Ultimate goal is to answer the question of source.
• Two steps:
• Determine whether evidence from the crime scene is “similar enough” to evidence obtained from the suspect.
• If so, does the similarity indicate same source? This is the probative value of the evidence.
• Machine learning may play a role in both steps.

Quantify degree of similarity

• Estimate a score that quantifies the similarity between two items.
• Data needs:
• A large dataset for which we know ``ground truth'' to train the algorithm.
• A dataset to test the algorithm and compute out-of-bag error rates.
• A set of features that can be used to compute the score.
• Selecting good features can be difficult.

​

​

Firearms examination

• Were two bullets fired from the same gun?
• If so, striations on the surface of the bullets should be similar.

​

Part science, part art

• Visual assessment with an optical comparison microscope.
• Categorical decision: match, do not match, inconclusive.
• Decision is subjective.

Striations

A bullet’s signature

Comparing two signatures

Estimating a score to quantify similarity

• We combine features into a single score using a random forest (RF).
• Score between 0 and 1 indicates degree of similarity between two items.

​

​

​

​

• Will RF trained on a bullet set still perform well when applied to a different set of bullets?

​

On a different set of bullets…

Weight of evidence

• What is the significance of a high similarity score?
• Evidence has high probative value, if a high score is rare unless both items have a common origin.
1. Compute score for a large and representative set of same source items and for a large and also representative set of different source items.
2. Compute a score-based likelihood ratio: given a score, the score-based likelihood ratio is the ratio of empirical densities evaluated at the value of the score.

On a different set of bullets…

Drawbacks of ML methods

• Performance of learning algorithms greatly depends on the data used for training.
• How do we define “different source”?
• Guns of different brands?
• Same brand and different model?
• Different brand and model but same caliber?
• Intractability
• Learning algorithms are like black boxes.
• Difficult to understand how the algorithm uses feature values to compute a score.

“I know a match when I see one”

A fairer administration of justice

• By promoting the use of statistics

and data science, we:

• Introduce some objectivity in the evaluation of evidence.
• Shatter the notion that categorical conclusions free of uncertainty are ever possible.
• Highlight the fact that similarity does not always imply a common source.

Our ballistics team

Calling all data scientists!

• Data science has tremendous potential for use in forensic applications.
• Many important open problems:
• Train algorithms to avoid biases and over-fitting.
• Agree on “best” features for each forensic discipline.
• Carry out extensive testing and validation of methods and data.
• Focus on the question of adoption and implementation.

​

We must continue to develop the probabilistic framework for almost all forensic disciplines.

A final thought

​

It is difficult to think of an area in which we can have a more significant, more immediate, social impact.