Trusting in artificial intelligence systems is not easy. Given the variety of edge cases on which machine learning models may fail, as well as the lack of visibility into the processes underlying their predictions and the difficulty of correlating their outputs to downstream business results, it’s no wonder that business leaders often look upon AI with some skepticism.
Data and concept drift are frequently mentioned in the context of machine learning model monitoring, but what exactly are they and how are they detected? Furthermore, given the common misconceptions surrounding them, are data and concept drift things to be avoided at all costs or natural and acceptable consequences of training models in production? Read on to find out. In this article we will provide a granular breakdown of model drift, along with methods for detecting them and best practices for dealing with them when you do.
Monitoring is critical to the success of machine learning models deployed in production systems. Because ML models are not static pieces of code but, rather, dynamic predictors which depend on data, hyperparameters, evaluation metrics, and many other variables, it is vital to have insight into the training, validation, deployment, and inference processes in order to prevent model drift and predictive stasis, and a host of additional issues. However, not all monitoring solutions are created equal. In this post, I highlight three must-haves for machine learning monitoring, which hopefully serve you well whether you are deciding to build or buy a solution.
Before you launch a project to build an artificial intelligence monitoring system from scratch, consider whether or not this would be a good use of your resources. When does it make sense to buy instead? Let’s discuss. This post explores the advantages and disadvantages of both alternatives so that you can make an informed decision about what’s best for your organization.