DOCUMENTATION · RESEARCH EVIDENCE

Research evidence

This page reports results from the dissertation, including models evaluated across four datasets and the validation statistics for the Explanation Stability Index (ESI). The interactive demo is a separate, smaller example: it runs a hand-authored logistic model on synthetic data (see Method). Its results should not be treated as dissertation findings.

TABLE 01

BAF cross-variant model comparison

Logistic Regression, LightGBM, and later XGBoost, trained on the Bank Account Fraud (BAF) benchmark across six variants, each with different injected bias and noise.

DatasetModelPR-AUCROC-AUCECEESI (geo)
BAF (base)Logistic Regression0.16390.87900.25060.8258
BAF (base)LightGBM0.19650.89370.24590.8430
BAF (variant III, hardest)Logistic Regression0.28050.91820.19820.8258
BAF (variant III, hardest)LightGBM0.53350.95400.15030.8430
BAF three-model extension
esi_geo mean across 12 comparisons; only 1/12 below the empirical reference threshold
XGBoost0.21520.87860.20510.9669
BAF three-model extension
esi_geo mean across 12 comparisons; 10/12 below the empirical reference threshold
LightGBM0.21990.88130.21870.8648
TABLE 02

Cross-dataset generalisation

The pipeline was also applied to three datasets outside BAF to test how far the findings transfer beyond one benchmark.

DatasetModelPR-AUCROC-AUCECEESI (geo)
IEEE-CIS (public benchmark)
Representative bin3→bin4 evaluation transition; closer-to-real-world labelled-training benchmark, not a fraud replication
LightGBM0.46010.86970.17680.9730
ULB credit-card fraud (real-data boundary)
24h→48h holdout, real card-present/not-present transactions; reported with an explicit low fraud-count warning
Logistic Regression0.63820.91780.03630.8613
Freddie Mac (multi-year credit risk)
2007→2008 historical-stress transition; real mortgage credit-risk data, not a fraud replication
Logistic Regression0.15660.79740.19260.9675
STATISTICS

ESI validation

  • Reference threshold: 0.9206, the 5th percentile (p05) of a 9,600-estimate null distribution.
  • Convergent validity vs. direct SHAP-profile change: ρ = -0.7174, p = 0.0001.
  • Discriminant validity vs. Population Stability Index (PSI): ρ = 0.2784, p = 0.1877, not significant, which is the point: ESI isn't just measuring plain population shift.
  • 1,152 synthetic drift experiments across 4 mechanisms. 4/4 monotonicity checks passed.
PROVENANCE

Where these numbers come from

Copied by hand from frozen, SHA-256-hashed research output, see reproducibility_evidence/RESULT_TRACEABILITY_MATRIX.csv. This app didn't compute any of it.