Monitoring and Testing of Distributed and Component-Based Systems

During the development and evolution of such a large-scale component-based distributed application, common questions are:

• What caused the system to crash? 

• Where was time spent? 

• How to package different components onto processors? Does the second processor really provide significant performance enhancement? 

• Have all the specified component interactions and paths been exercised in the integration testing? 

•  If a component is changed, what are the units and subsystems that are required to perform regression testing? 

Our research effort here at the Imaging Systems Lab is to develop new techniques and tools to help answer the above questions, with the focus on the applications that are running in embedded devices, and have complex interactions with enterprise services. 

Monitoring of Distributed and Component-Based Systems

We have developed a monitoring toolset called coSpy for distributed and component-based systems. So far, the toolset has been applied to the applications that are built upon the ORBlite (CORBA-based runtime developed by HPL), or a particular COM-like infrastructure (developed by HP). The uniqueness of our approach is to facilitate system-wide causality propagation in the monitored application, with no modification to application components. One of the most important causality information is the caller/callee relationship due to component-level method (call) invocations. These coarse-grained invocations are the ones that have their user-defined IDL interface definitions, and potentially happen across different threads, processes, and even processors.

The coSpy toolset consists of three parts: the augmented IDL compiler to automatically deploy instrumentation probes onto the stubs and skeletons of application components, the monitoring-related library to collect per-thread runtime execution information across the entire distributed application, and the postmortem analysis framework to analyze the collected monitoring data, determine inter-component interactions, and annotate timing latency and CPU consumption information onto each component-level method invocation. In particular, we are able to construct a system-wide dynamic call graph that structures all component-level method invocations ever happened in the monitored application run, following their caller/callee relationships. 

An example of our dynamic system call graph from a large-scale application, configured with 4 processes and 32 threads, is shown below. Over 100,000 component-level method invocations are contained in this particular graph. Each invocation is represented by a graph node. The hyperbolic tree SDK from Inxight is employed to visualize the call graph.

Testing of Distributed and Component-Based Systems

The cost and complexity of developing and maintaining test suites to achieve defect-free applications often exceeds that of developing and maintaining these applications. It becomes much more significant for large-scale distributed applications that keep evolving within their life span. Our research effort is to build a test automation infrastructure and its tool support to reduce the cost and complexity of developing and maintaining such complex test suites by automating: test cases generation, test effectiveness evaluation, regression testing, and impact change determination. 

In particular to distributed and component-based applications, we adopt the two-folded approach: first, to exploit the distributed systems monitoring techniques that we have developed to perform test execution runtime monitoring, and to collect and store test execution information; and secondly, to augment the IDL for test design specifications and automate the generation of test cases and the test harness. Overall, our approach is to explore the linkages among three parties altogether: the application under test, its behavior model, and the underlying middleware runtime infrastructure. 

Publications:

• Jun Li, “Monitoring and Characterization of Component-Based Systems with Global Causality Capture,” Proceedings of the 23rd International Conference on Distributed Computing Systems, pp.422-31, May 2003. Also published as HPL Technical Report HPL-2003-54. (file in pdf) 
• Jun Li, “Monitoring of Component-Based Systems,” HP Labs Technical Report HPL-2002-25 (R.1). (file in pdf)
• Jun Li, “Characterization and Impact Estimation of CPU Consumption in Multi-Threaded Distributed Applications,” HP Labs Technical Report HPL-2002-50 (R.1). (file in pdf) 
• Jun Li, Keith Moore, “Exploiting Global Causality in Testing of Distributed and Component-Based Applications,” will appear as a short paper at the Proceedings of the 20th International Conference on Software Maintenance, Chicago Illinois, September 2004.

For more information about the technology, please contact Jun Li (jun.li@hp.com) at Imaging Systems Lab, HP Labs.