Technical Reports
HPL-2009-161
Click here for full text:
Semantic Automation from Screen Capture
Barkol, Omer; Bergman, Ruth; Pnueli, Ayelet; Schein, Sagi
HP Laboratories
HPL-2009-161
Keyword(s): Automation technology, Image analysis, Visual parsing
Abstract: IT organizations want efficiency and are constantly attempting to reduce or eliminate manual work. Automation technology facilitates the transition from manual to automatic. However, until now, automation technology has been applied in limited settings in IT organizations. For example, automated software testing uses automation technology, but only about 5-10% of software testing is automatic. One of the reasons automation technology is not more widely adopted, is that there is a very high barrier to using it. The objects and constructs used for today's automation originate from the underlying application or computing environment, so that using automation technology is as complex as programming. We propose a new automation technology for applications with Graphical User Interfaces (GUI). Unlike existing automation technology, which uses the objects defined by the technology that was used to build the GUI, the technology we propose uses screen capture. The system described in this paper infers GUI semantics from screen images. The system is modelled after the notion of compilers for programming languages. It has a structural analysis module, which corresponds to the lexical analyzer (Lex), and a semantic analysis module, which corresponds to the parser (Yacc). Structural analysis uses image analysis techniques including segmentation and object recognition, and semantic analysis uses graph grammars and visual parsing.
10 Pages
External Posting Date: July 21, 2009 [Fulltext]. Approved for External Publication
Internal Posting Date: July 21, 2009 [Fulltext]