Data Debt in Production ML Pipelines: Detection and Remediation at Scale

Every machine learning team hits the same obstacle. A model performs well during training and staging, but after a few weeks of production, it begins to make nonsensical decisions. No one has altered any code. What has actually changed is the data.

Data debt refers to this state, a gap dating the model’s training data and the production data the model is encountering. Like technical debt, it creates no immediate failures. It results in a slow, tedious decline of the model’s decisions. By the time the degradation is noticed, the cause is impossible to determine. Upstream systems have changed multiple times and no one has documented the changes.

In this piece, I present methods for addressing data debt before it becomes costly. The methods I present are data debt prevention measures and not abstract ideas of data debt containment.

Where Data Debt Comes From

Data debt accumulates from a small,...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE