Research evidence
This page reports results from the dissertation, including models evaluated across four datasets and the validation statistics for the Explanation Stability Index (ESI). The interactive demo is a separate, smaller example: it runs a hand-authored logistic model on synthetic data (see Method). Its results should not be treated as dissertation findings.
BAF cross-variant model comparison
Logistic Regression, LightGBM, and later XGBoost, trained on the Bank Account Fraud (BAF) benchmark across six variants, each with different injected bias and noise.
| Dataset | Model | PR-AUC | ROC-AUC | ECE | ESI (geo) |
|---|---|---|---|---|---|
| BAF (base) | Logistic Regression | 0.1639 | 0.8790 | 0.2506 | 0.8258 |
| BAF (base) | LightGBM | 0.1965 | 0.8937 | 0.2459 | 0.8430 |
| BAF (variant III, hardest) | Logistic Regression | 0.2805 | 0.9182 | 0.1982 | 0.8258 |
| BAF (variant III, hardest) | LightGBM | 0.5335 | 0.9540 | 0.1503 | 0.8430 |
| BAF three-model extension esi_geo mean across 12 comparisons; only 1/12 below the empirical reference threshold | XGBoost | 0.2152 | 0.8786 | 0.2051 | 0.9669 |
| BAF three-model extension esi_geo mean across 12 comparisons; 10/12 below the empirical reference threshold | LightGBM | 0.2199 | 0.8813 | 0.2187 | 0.8648 |
Cross-dataset generalisation
The pipeline was also applied to three datasets outside BAF to test how far the findings transfer beyond one benchmark.
| Dataset | Model | PR-AUC | ROC-AUC | ECE | ESI (geo) |
|---|---|---|---|---|---|
| IEEE-CIS (public benchmark) Representative bin3→bin4 evaluation transition; closer-to-real-world labelled-training benchmark, not a fraud replication | LightGBM | 0.4601 | 0.8697 | 0.1768 | 0.9730 |
| ULB credit-card fraud (real-data boundary) 24h→48h holdout, real card-present/not-present transactions; reported with an explicit low fraud-count warning | Logistic Regression | 0.6382 | 0.9178 | 0.0363 | 0.8613 |
| Freddie Mac (multi-year credit risk) 2007→2008 historical-stress transition; real mortgage credit-risk data, not a fraud replication | Logistic Regression | 0.1566 | 0.7974 | 0.1926 | 0.9675 |
ESI validation
- Reference threshold: 0.9206, the 5th percentile (p05) of a 9,600-estimate null distribution.
- Convergent validity vs. direct SHAP-profile change: ρ = -0.7174, p = 0.0001.
- Discriminant validity vs. Population Stability Index (PSI): ρ = 0.2784, p = 0.1877, not significant, which is the point: ESI isn't just measuring plain population shift.
- 1,152 synthetic drift experiments across 4 mechanisms. 4/4 monotonicity checks passed.
Where these numbers come from
Copied by hand from frozen, SHA-256-hashed research output, see reproducibility_evidence/RESULT_TRACEABILITY_MATRIX.csv. This app didn't compute any of it.