A Diagnostic Procedure For High-Dimensional Data Streams Via Missed Discovery Rate Control

Abstract

Monitoring complex systems involving high-dimensional data streams provides quick real-time detection of abnormal changes of system performance but accurate and efficient diagnosis of the streams responsible has also become increasingly important in many data-rich statistical process control (SPC) applications. Existing diagnostic procedures, designed for low/moderate dimensional multivariate process, may miss too much important information in the out-of-control (OC) streams with a high signal-to-noise ratio (SNR) or waste too many resources finding useless in-control (IC) streams with a low SNR. In addition, these procedures do not differentiate between streams according to their severity. In this article, we formulate the diagnosis problem of high-dimensional data streams as a multiple testing problem and provide a computationally fast diagnostic procedure to control the weighted missed discovery rate (wMDR) at some satisfactory level. The proposed procedure overcomes the limitations of conventional diagnostic procedures by controlling the wMDR and minimizing the expected number of false positives (EFP) as well. We show theoretically that the proposed procedure is asymptotically valid and optimal in a certain sense. Simulation studies and a real data analysis from a semi-conductor manufacturing process show that the proposed procedure works very well in practice.