A credit risk model's approval rate shifted 8% in a month. How do you investigate and what controls should exist?

Short answer

Check data pipeline issues first (missing or changed features), then input drift versus the training distribution, population changes (new channel, marketing campaign), and model or threshold changes. Compare performance on matured labels. Controls: feature and prediction drift monitoring, data validation, versioned models and thresholds, alerting, independent validation and documented revalidation triggers.

Detailed answer

  1. Data integrity: null rates, default values, schema changes, upstream system changes (a bureau field changed format).
  2. Drift: PSI/KS per feature vs training baseline; segment analysis (channel, region).
  3. Population shift: business changes (new partner sending different applicants).
  4. Model/threshold changes: deployment logs, configuration history.
  5. Performance: once labels mature, compare default rates against expectations.
  6. Action: fix data issue, adjust thresholds with risk approval, or retrain/revalidate.

Controls align with model risk management practices: inventory, validation, monitoring, change control, documentation and explainability for adverse action notices.

What the interviewer expects

Data-first debugging, statistical monitoring and governance awareness in regulated contexts.

Common mistakes to avoid

Retraining immediately without understanding the cause.