1
AI-POWERED
Document Insights
and Data Extraction
Full RAG Studio — amended Pfizer capstone
John Raymond TYABA
u2678314@uel.ac.uk
MAIN FINDING
Reliable document AI needs evidence, not just fluent answers.
Multiple PDFs • scanned pages • MiniLM + FAISS • FLAN-T5 • safe fallbacks • citations • abstention
Updated to support a manual 10-question supplier-pack evaluation and one honest failure.
AI-POWERED DOCUMENT INSIGHTS AND DATA EXTRACTION
2
Problem Statement | Pharmaceutical Document Intelligence
Extracting verifiable answers from mixed supplier packs, scanned pages and unstructured PDFs
CHALLENGE
Supplier packs combine certificates, specifications, declarations and scans. Manual searching is slow, while a fluent answer is unsafe if the user cannot verify the exact file, page and evidence.
SOLUTION
Users upload one or more PDFs, process them once and ask questions manually. The pipeline applies OCR, metadata-aware chunking, semantic retrieval, grounded generation and safe fallbacks.
PAIN POINT
HOW THE PIPELINE SOLVES IT
Low-text scans
Tesseract OCR fallback
Mixed document types
Classification, filtering and auto-routing
Model or index failure
TF-IDF and extractive fallbacks
Unsupported questions
Similarity threshold and abstention
AI-POWERED DOCUMENT INSIGHTS AND DATA EXTRACTION
3
System Architecture | Resilient Open-Source RAG Pipeline
Document upload → OCR → chunking → embeddings → FAISS → retrieval → grounded generation / fallback
INPUT
Multiple PDFs
→
OCR
Tesseract 250 DPI
→
CHUNK
900 / 150 overlap
→
EMBED
MiniLM
→
INDEX
FAISS
→
ANSWER
FLAN-T5 / fallback
Component | Technology | Configuration |
OCR Engine | Tesseract + PDF2Image | 250 DPI; triggered when a page has <40 non-space characters |
Text Chunking | Fixed overlapping windows | 900 characters; 150-character overlap; page metadata |
Embeddings | all-MiniLM-L6-v2 | Normalised dense vectors; semantic preferred path |
Vector Database | FAISS IndexFlatIP | In-memory inner-product similarity |
Retriever | Metadata-filtered dense search | Top-k 1–8; default 4; manual filter + auto-route |
LLM | FLAN-T5-small | Deterministic; max_new_tokens = 180 |
Fallbacks | TF-IDF + extractive answer | Safe retrieval/generation when optional models fail |
Prompt Strategy | Context-only grounded prompt | Abstain below configurable similarity threshold |
AI-POWERED DOCUMENT INSIGHTS AND DATA EXTRACTION
4
Pipeline Performance Metrics | Supplier-Certificate Benchmark
10 labelled questions on the supplied AKTA ready Certificate of Quality; one unsupported question included
50%
Answer accuracy
100%
Citation accuracy
27.5%
Average confidence
0%
Abstention rate
END-TO-END QUALITY
5
correct answers
5
incorrect answers
SCOPE AND FAILURE
10
manual questions in exported JSON
34.4%
average top similarity; no question abstained
Actual exported session: dates, process limits and biosafety passed; product name, lot, article, origin and unsupported storage temperature failed.
AI-POWERED DOCUMENT INSIGHTS AND DATA EXTRACTION
5
Pipeline in Action | Successful Supplier-Certificate Lookup
A grounded numeric answer with source, page, type, score and confidence
QUESTION
“What gamma irradiation range applies to the inlets?”
RETRIEVED EVIDENCE
Certificate of Quality — page 1
“Gamma Irradiation Inlets 25.0–40.0 kGy”
Full RAG Studio
DOCUMENTS
AKTA certificate
Supplier PDF
Scanned page
Process and Index
Questions: 10
What gamma irradiation range applies to the inlets?
25.0–40.0 kGy appears in the returned evidence.
Source: certificate • page 1 • top score 0.551 • confidence 36.8%
PASS
Strength demonstrated: exact process-limit retrieval with visible evidence and citation.
AI-POWERED DOCUMENT INSIGHTS AND DATA EXTRACTION
6
Pipeline in Action | Honest Unsupported Question
The failure case explains why the amended capstone adds an abstention threshold
QUESTION
“What storage temperature is specified?”
BASELINE FAILURE
The certificate contains no storage temperature.
The exported session still returned unrelated nearest context and did not abstain. This is the clearest production-safety gap.
Full RAG Studio
DOCUMENTS
AKTA certificate
Supplier PDF
Scanned page
Process and Index
Questions: 10
What storage temperature is specified?
Returned autoclave, irradiation and biosafety text instead of abstaining.
Top score 0.324 • confidence 22.9% • abstained = false
FAIL
Required fix: calibrate abstention using labelled queries and add field extraction or post-generation validation for exact identifiers.
AI-POWERED DOCUMENT INSIGHTS AND DATA EXTRACTION
7
Design Decision Analysis | Reliable on Limited Hardware
Transparency, graceful degradation and maintainability are prioritised
Decision | Rationale | Trade-off |
MiniLM embeddings | Fast and lightweight semantic retrieval in Colab | Less pharmaceutical-specific than larger encoders |
900 / 150 chunks | Simple, reusable and preserves page evidence | May split tables or clauses across boundaries |
FLAN-T5-small | Open-source, deterministic and CPU-friendly | Lower reasoning depth than a larger LLM |
FAISS IndexFlatIP | Fast in-memory semantic index | Not persistent or distributed |
TF-IDF fallback | Keeps retrieval available if semantic stack fails | Weaker semantic matching for paraphrased questions |
Extractive fallback | Keeps answers tied directly to source text | Less fluent than generated answers |
Abstention threshold | Prevents unsupported confident answers | Must be calibrated on representative supplier files |
KEY TRADE-OFFS
Speed vs. accuracy
Smaller models and in-memory indexing improve demo reliability but reduce domain depth and scale.
Complexity vs. maintainability
Fallbacks and abstention add code paths, but prevent the entire workflow from failing or guessing.
AI-POWERED DOCUMENT INSIGHTS AND DATA EXTRACTION
8
Current Limitations & Next Steps | Supplier-Pack Roadmap
INPUT QUALITY
Low-DPI, skewed or mixed-layout scans can weaken OCR.
RETRIEVAL
Broad or cross-document questions can spread evidence across chunks.
SCALABILITY
In-memory indexing and synchronous ingestion remain prototype constraints.
PROPOSED ENHANCEMENTS
SHORT TERM
Run ≥10 labelled questions on the real multi-document supplier pack; calibrate abstention and OCR preprocessing.
MEDIUM TERM
Add hybrid BM25 + MiniLM retrieval, cross-encoder reranking and table-aware parsing.
LONG TERM
Persist indexes; add asynchronous ingestion, access control, monitoring and audit logs.
AI-POWERED DOCUMENT INSIGHTS AND DATA EXTRACTION
9
Project Impact & Learning Outcomes
A working, transparent pipeline designed for technical and non-technical review
TECHNICAL LEARNING
Multiple-file ingestion
OCR and PDF extraction
Dense and sparse retrieval
Grounded prompting
Gradio state and export
POTENTIAL IMPACT
Faster supplier-document lookup
Traceable evidence for review
Safer unsupported-question handling
Reusable architecture across document types
Exportable 10-question sessions
PROFESSIONAL SKILLS
Reliability engineering
Honest metric reporting
Debugging and resilience
Technical documentation
Stakeholder communication
Question for Pfizer: Would a smaller model with grounded extractive fallback be acceptable, or would the team always prefer a larger LLM?
AI-POWERED DOCUMENT INSIGHTS AND DATA EXTRACTION