Project Overview
This moral hazard detection system helps a leading UK insurer identify potentially fraudulent behavior by analyzing speech patterns in claims handler notes. The system overcomes significant challenges including the absence of benchmark data, extremely lengthy unstructured text, and severe technical constraints from IT infrastructure migration.
Technical Solution
System Architecture
We engineered a sophisticated speech pattern analysis system with:
- Linguistic Pattern Recognition identifying 27 speech patterns correlated with moral hazard
- Multi-agent LLM architecture with specialized roles:
- Context Agent: Processes claim history and policy details
- Analysis Agent: Examines claims handler notes for inconsistencies
- Pattern Agent: Compares claim patterns with known indicators
- Summarization Agent: Compiles findings into actionable reports
- Semi-supervised learning approach using:
- Expert-created seed dataset
- Bootstrapping method to expand training examples
- Confidence-based self-training for iterative improvement
- Efficient processing architecture optimized for constraints:
- Chunking algorithm for large documents within memory limits
- Two-pass system: lightweight screening followed by deep analysis
- Knowledge distillation reducing model size by 73%
Technical Implementation
The solution incorporates several key innovations:
- Pattern amplification technique translating expert-identified linguistic markers to embedding space
- Optimized model for 5-minute runtime window on local hardware
- Adaptation to function within Azure migration constraints
- Novel approaches for handling extremely lengthy unstructured notes
Implementation Challenges
Key challenges we addressed included:
- Complete absence of labeled data for moral hazardous behavior
- Extremely lengthy claims notes (often 50+ pages)
- Severe technical constraints from Databricks to Azure migration
- Local laptop execution with 5-minute runtime limitation
- Need for explainable results to support investigation teams
- Balancing detection sensitivity with false positive management
Business Impact
The system delivered substantial value across multiple dimensions:
Detection Performance Metrics
- Fraud Detection Improvement: 32% more potentially fraudulent claims identified
- Precision: 83% of flagged claims confirmed as actual moral hazard cases
- Recall: 76% of actual moral hazard cases successfully detected
- F1 Score: 0.79, providing balanced precision and recall
- False Positive Reduction: 47% decrease compared to rule-based systems
- False Negative Rate: Reduced to 24% (from 41% with previous approach)
Operational Impact Metrics
- Claims Processing Efficiency: 89% reduction in time spent reviewing clean claims
- Daily Processing Capacity: Successfully analyzing 3,500+ claims daily
- Average Claims Handler Time Savings: 6.2 hours per week
- Handler Confidence: 89% report higher confidence in flagged cases
Financial Impact Metrics
- Projected Annual Savings: £3.8M in prevented fraudulent payouts
- Fraud Loss Ratio: Decreased from 8.7% to 6.1% of total claims value
- Investigation ROI: 417% return on investigation costs for flagged claims
- Implementation Cost-Benefit: System paid for itself within first 2 months
Technology Stack
- Small to medium-sized LLMs (7B-70B parameters)
- LangChain for agent orchestration
- Python for processing and orchestration
- Pattern recognition algorithms for speech analysis
- Semi-supervised learning frameworks
- Azure Machine Learning for deployment
- Local optimization techniques for constrained environments