| |
Click here for full text:
Capturing, Indexing, Clustering, and Retrieving System History
Cohen, Ira; Zhang, Steve; Goldszmidt, Moises; Symons, Julie; Kelly, Terence; Fox, Armando
HPL-2005-158
Keyword(s): system performance; Bayesian networks; information retrieval; problem signatures
Abstract: We present a method for automatically extracting from a running system an indexable signature that distills the essential characteristic from a system state and that can be subjected to automated clustering and similarity-based retrieval to identify when an observed system state is similar to a previously- observed state. This allows operators to identify and quantify the frequency of recurrent problems, to leverage previous diagnostic efforts, and to establish whether problems seen at different installations of the same site are similar or distinct. We show that the naive approach to constructing these signatures based on simply recording the actual "raw" values of collected measurements is ineffective, leading us to a more sophisticated approach based on statistical modeling and inference. Our method requires only that the system's metric of merit (such as average transaction response time) as well as a collection of lower-level operational metrics be collected, as is done by existing commercial monitoring tools. Even if the traces have no annotations of prior diagnoses of observed incidents (as is typical), our technique successfully clusters system states corresponding to similar problems, allowing diagnosticians to identify recurring problems and to characterize the "syndrome" of a group of problems. We validate our approach on both synthetic traces and several weeks of production traces from a customer-facing geoplexed 24 X 7 system; in the latter case, our approach identified a recurring problem that had required extensive manual diagnosis, and also aided the operators in correcting a previous misdiagnosis of a different problem. Notes: Copyright 2005 ACM. Published in and presented at Symposium on Operating Systems Principles 2005 (SOSP 2005), 23-26 October 2005, Brighton, UK
14 Pages
Back to Index
|
Printable version
