Why Attribution Stability Matters More Than Attribution Accuracy
Enterprise teams treat SHAP and LIME explanations as ground truth. They aren't — they're samples from a distribution that varies with background data, sampling strategy, and runtime conditions. For regulated AI, the property that actually matters under audit isn't whether the explanation is right. It's whether it's stable enough to defend.
A credit risk team at a UK retail bank deploys an imbalance-aware fraud detection model. The model uses TreeSHAP to generate feature attributions for every flagged transaction, attached to the case file for compliance review. Six months later, a regulator challenges a specific decision. The compliance team pulls the original case file, presents the attribution: "the decision was driven by transaction velocity, geographic anomaly, and merchant category code."The regulator asks the team to rerun the attribution. They do, using the same model, same input, same TreeSHAP implementation. The ranked feature importances come back in a slightly different order....
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
