BrainBench Milestone 3
Orion Powers and Daniella Seum
Milestone 3 Progress Matrix
Task | Daniella | Orion | Completion | To-do |
Compile math problem dataset for advanced Probability and Statistics. | 50% | 50% | 100% | N/A |
Finalize free or local LLMs we are using for initial testing. | 50% | 50% | 100% | N/A |
Finish developing testing pipeline, complete with robust output parsing to verify correctness of responses. Implement, test, demo. | 50% | 50% | 100% | N/A |
Test LLMs on problem dataset. | 50% | 50% | 25% | Finish running tests, just need time |
Stats Problems
01
Problem Count
Completed a dataset of 1,000 probability and statistics problems.
02
Topics Covered
03
XML Export
Local LLMs
Model Selection
Local GPU Feasibility
Models are open-license and optimized for local GPU inference without cloud costs through Ollama.
Prompt Forces Keyword
01
Keyword-Based Extraction
The system prompt includes 'Print FINAL_ANSWER: <value> on its own line.' This keyword is captured using regex, ensuring high extraction success rates with a confidence score of 1.0.
02
Fallback Mechanism
If the keyword is missing, the system falls back to other extraction methods, maintaining an overall extraction success rate of over 99% across all models.
Fallback Chain
LaTeX Boxed Notation
If the keyword is not found, the system scans for LaTeX boxed notation (\boxed{...}) with a confidence score of 0.8, capturing answers in mathematical format.
Natural Language Patterns
If LaTeX notation is absent, the system searches for natural language patterns like 'Answer is' with a confidence score of 0.6, handling descriptive responses.
Last Numeric Token
As a last resort, the system extracts the last numeric token with a confidence score of 0.4, ensuring answers are captured even in less structured responses.
Auto-Type Detection
Regex-Based Type Detection
The system uses regex patterns to detect answer types: fractions, decimals, integers, coordinates, ranges, expressions, and text. Each type is routed to a specific normalization function.
NormalizedAnswer Object
Detected types are stored in a NormalizedAnswer object, ensuring structured and consistent data for accurate comparison and verification.
Per-Type Cleanup
01
Fraction Normalization
Fractions are kept in their raw tuple form (numerator, denominator) without reduction, preserving the exact values provided by the LLM.
02
Decimal Normalization
Decimals are converted to float values, with precision bits tracked to ensure accurate tolerance calculations during comparison.
03
Expression Normalization
Expressions are stripped of spaces, sorted by exponent in descending order, and simplified (e.g., x^1 becomes x) to ensure consistent string comparison.
04
Text Normalization
Text answers are converted to lowercase to ensure case-insensitive comparison, handling qualitative responses like 'rational' or 'irrational'.
Per-Type Rules
01
Fraction Comparison
Fractions are reduced to their lowest terms using the greatest common divisor (GCD) and compared as tuples to ensure mathematical equivalence.
02
Decimal Comparison
Decimals are compared with a tolerance based on their precision, ensuring accurate matching within acceptable error margins.
03
Integer Comparison
Integers are compared exactly, requiring an exact match without any tolerance, ensuring precise verification.
04
Expression Comparison
Expressions are compared as whitespace-free strings, ensuring exact symbolic equivalence without evaluating the expressions.
Milestone 4 Task Matrix
Task | Daniella | Orion |
Finish running local LLMs on problem datasets. | 33% | 66% |
Begin development on the website and add data from local LLMs. | 66% | 33% |
Explore possibilities of expanding to larger LLMs and their APIs, or expanding on problem types. | 50% | 50% |
Finish Runs
01
Batch Completion
Allow ongoing batch runs to complete without interruption.
02
Data Harvesting
Harvest accuracy and latency data into CSV format for analysis.
03
Log Backup
Backup logs and checksums to ensure data integrity.
04
Report Generation
Generate detailed per-model reports summarizing results.
Build Dashboard
01
Angular Skeleton
Develop the Angular skeleton for the evaluation dashboard.
02
Chart.js Integration
Integrate Chart.js for dynamic charts and tables.
03
Responsive Design
Ensure the dashboard is mobile-responsive for accessibility.
Expand
01
API Quotes
Research and quote costs for GPT-4/5, Claude, Gemini, and DeepSeek-R1.
02
Other Problem Types
Coding, logic, image generation, etc.
Thank you.