Using LLMs to Punch Above your Weight!

Cam Feenstra

Hello World

Cam Feenstra

Principal Software Engineer

Anzen

• $23B is spent per year on litigation of employment lawsuits, and rising

​

• We offer insurance to cover businesses when they get sued

​

• We package software with the insurance to help them proactively avoid getting sued

​

• Lean team: < 20 employees

Focus of the talk

• How small businesses can use the power of LLMs to compete with large incumbents, particularly in industries like insurance

​

• I’ll walk you through 2 examples of where we’ve successfully utilized LLMs at Anzen so far, and plans we have for the future

​

• I’ll share some learnings that you might be able to apply to your own work

What’s so hard about competing with large incumbents?

Large incumbents are, well, large�

• More money/resources
• More manpower
• More data
• Network effects

​

But they also have weaknesses�

• Less flexible/agile than smaller companies
• Slower to adopt new technologies

Streamlining Insurance Underwriting - The Problem

• 98% done through retail brokers�
• ~1 week turnaround times�
• Retail brokers are busy!

Streamlining Insurance Underwriting - The Problem

(cont.)

​

​

Streamlining Insurance Underwriting - The Problem

(cont.)

​

​

• Dozens of different�formats

​

​

​

​

Streamlining Insurance Underwriting - The Solution

Streamlining Insurance Underwriting - The Solution

• Classification: fine-tuned a classifier based on Google’s BERT model
• ~300 training examples, manually labelled in an afternoon
• 95% accuracy, ~90% precision, 100% recall
• Tested out 4 models, many training configurations

​

• Extraction: used AWS Textract with question answering

Analyzing Employee Agreements - The Problem

• We wanted to build a feature that allows entrepreneurs to upload agreements and get immediate feedback on potential compliance issues
• Offer letters
• Separation agreements

​

• Accuracy highly important

​

• Gathered domain knowledge from attorneys

Analyzing Employee Agreements

​

The Solution

Analyzing Employee Agreements - The Solution (cont.)

• Used off-the-shelf models to turn domain knowledge into working feature quickly (<1 week)
• Sentence embedding
• Question answering

​

• Only a few dozen examples of each type of document required

Learnings

​

• Deploying LLMs requires strong understanding of engineering fundamentals:
• Infra monitoring/observability
• Load testing �
• Evaluation metrics are still very important ��
• You don’t necessarily need to run GPU instances to serve models in production

​

• APIs - cost of backtesting changes

Future Ideas

​

• Underwriting help w/ ChatGPT/generative AI�
• Generalized compliance help

Conclusion

• What a time to be alive!�
• LLMs can be asset even without generative AI�
• Come work with us!

Thank you / Q&A