Automating Data Monitoring: Detecting Structural Breaks in Time Series Data Using Bayesian Minimum Description Length
share
In modern business modeling and analytics, data monitoring plays a critical role. Nowadays, sophisticated models often rely on hundreds or even thousands of input variables. Over time, structural changes such as abrupt level shifts or trend slope changes may occur among some of these variables, likely due to changes in economy or government policies. As a part of data monitoring, it is important to identify these changepoints, in terms of which variables exhibit such changes, and what time locations do the changepoints occur. Being alerted about the changepoints can help modelers decide if models need modification or rebuilds, while ignoring them may increase risks of model degrading. Simple process control rules often flag too many false alarms because regular seasonal fluctuations or steady upward or downward trends usually trigger alerts. To reduce potential false alarms, we create a novel statistical method based on the Bayesian Minimum Description Length (BMDL) framework to perform multiple change-point detection. Our method is capable of detecting all structural breaks occurred in the past, and automatically handling data with or without seasonality and/or autocorrelation. It is implemented with computation algorithms such as Markov chain Monte Carlo (MCMC), and can be applied to all variables in parallel. As an explainable anomaly detection tool, our changepoint detection method not only triggers alerts, but provides useful information about the structural breaks, such as the times of changepoints, and estimation of mean levels and linear slopes before and after the changepoints. This makes future business analysis and evaluation on the structural breaks easier.
READ FULL TEXT