SmartFrog

Q: What does the name “SmartFrog” stand for?

A: SmartFrog stands for “Smart Framework for Object Groups”, which captures the notion that this is a framework rather than a complete product, and that its purpose is to intelligently orchestrate the activities of groups of software objects or components.

Q: What is SmartFrog?

A: SmartFrog is a software framework for helping to build distributed, component-based software systems in a way that makes them easy to configure, automatically install and start, and automatically shut down. Such systems typically have multiple software components running across a network of computing resources, where the components must work together to deliver the functionality of the system as a whole. It’s critical that the right components are running on the correct computers, that the components are correctly configured, and that they are correctly combined together into the complete system. This profile fits many of the services and applications that run on today’s computing infrastructure.

Concrete examples are three-tier web applications, which often consist of a database server, application logic middleware, web server software, firewalls and load-balancers. All of these can be thought of as components that need to work together to deliver the complete web-service. Each component must be installed on an appropriate resource and correctly configured. Components must be started in a certain sequence, and linked together into the complete system.

With SmartFrog, system configuration details such as: which components make up the system, on which computers those components should run, how the individual components are configured, in what order they are started, and how they are combined into a complete system, are captured in “system descriptions”. These descriptions, written in the SmartFrog description language, are interpreted by the SmartFrog distributed runtime engine in order to install the required software components, to configure them, and to start the complete software system according to the details of the description.

Q: What advantages does SmartFrog offer?

A: First, SmartFrog allows you to take all the details of a software system that can be considered configuration-related and record them in one place as part of a SmartFrog system description. This means it is very easy to make changes to the configuration of the system and hence change the operation of the system when it is installed and started. The description method also allows systems to be built from pre-defined templates, and allows systems to be constructed from sub-systems. We call such systems “configuration-driven”.

Secondly, given the SmartFrog descriptions, systems can be installed, started, stopped and uninstalled completely automatically. This means that a given system description can be installed automatically, with multiple instances, and that this can be done repeatably and correctly.

This functionality makes SmartFrog ideal for utility computing applications – which need to be automatically installed and started on utility computing resources, repeatedly, and removed cleanly when they are no longer needed.

Q: What technologies does SmartFrog consist of?

A: There are three main parts to the SmartFrog framework. The first is a configuration description language to define the configuration data appropriate to the SmartFrog components. The language has many convenient features for describing configuration data, and provides the notions of abstraction and template creation which are necessary to build complex configuration definitions.

The second part is a “component model” -- a specification of how SmartFrog components are written and integrated into the SmartFrog framework. This includes aspects such as the lifecycle of a SmartFrog component and mechanisms for checking what state the component is in (such as whether it is still “alive”). The components are responsible for interpreting the configuration details given in the SmartFrog language and then carrying out the necessary configuration tasks base on these details. SmartFrog includes some standard components – offering features such as component sequencing, distributed workflows, failure detection, script execution, installation of software, dynamic discovery and naming -- and new ones can easily be added for new applications.

The third part is the runtime system, made up of SmartFrog “daemons” that run on every element in the network. The runtime system interprets SmartFrog system descriptions and uses them to control the creation of components in the right location, with the correct configuration parameters. It also provides facilities for tools to locate and interact with components, for monitoring their lifecycles, and for carrying out other runtime tasks. It is fully distributed, with no required central point of control and therefore scales very well. Additionally, because of the fully-distributed nature of the runtime system, any component can use it directly to create and manage its own sub-systems as part of a dynamic and adaptive system configuration infrastructure.

In addition to the specific parts of the framework, SmartFrog has a methodology -- a way of thinking about the problem -- for extending the set of key elements in a consistent style to cover areas not yet handled by the framework.

Q. You say that SmartFrog is a “framework”; what do you mean by this?

A. This is intended to indicate two things: first that SmartFrog is not a packaged product or solution; rather it is a building block to help build configurable software systems. So, while it has wide applicability, new SmartFrog components will usually need to be developed for each new application. Secondly, that SmartFrog can be extended by “plugging in” new components and system descriptions into the framework. (In fact, the system itself is built out of components that satisfy its own component model, so that many system behaviours can be changed by replacing or modifying the existing components.)

Q: Where was SmartFrog developed?

A: SmartFrog was created in HP Laboratories as part of a research project working on configuration management for domains ranging from distributed measurement and management systems, through to dynamic configuration of adaptive services on utility computing infrastructures such as those offered by the Grid.

Q: How secure is SmartFrog?

A: Without security, SmartFrog would be a near-perfect virus transmission engine! Fortunately, we have taken security seriously, and the system protects itself from malicious use using a public key infrastructure (PKI) system. Each node that participates in a SmartFrog runtime system is supplied with a certificate. Furthermore, all software components and system descriptions are “signed” with a certificate. The certificates are used to permit only validated nodes to participate in the runtime system, and those nodes will only manipulate components and descriptions that have been appropriately signed. Additionally, all network communication takes place using an encrypted transport. Currently there is a single-level security model, where nodes and components are either fully trusted or not trusted at all. Finer grained models of security are currently being researched.

Q: How is SmartFrog implemented, and what platforms does it run on?

A: The core SmartFrog system is implemented completely in Java, and therefore runs on any platform that supports a full Java environment. We have used SmartFrog on Microsoft Windows, Linux and HP-UX. Some SmartFrog components are designed to work on specific operating systems, or to interface with specific technologies that have platform-specific requirements, so these are more limited in their portability.

Q: Do I have to completely rewrite my applications as SmartFrog components in order to use SmartFrog?

A: No. In fact it’s generally fairly easy to write simple SmartFrog components that “wrap” or encapsulate non-SmartFrog things. For example, we’ve done this with the Apache web server where we created simple SmartFrog components that are capable of installing, configuring, starting, stopping and uninstalling the Apache system. This allows us to completely manage the Apache system from SmartFrog – to install it on a specific system, to configure it correctly, to start and stop the system and to connect it in to other system components, all of which can be driven from a system description in the SmartFrog language.

One caveat about this approach is that it does rely on the encapsulated system being cooperative – in the sense that controlling it must be possible via an API or command line scripts. However, if everything to be done through a graphical user interface (GUI), this can cause problems for the types of automation that are required.

Q: What is the status of SmartFrog?

A: SmartFrog has been developed over a number of years, and has been used as an embedded component in a product context. It is in active use as a framework for experimentation. Our recent work has focused on making the system general-purpose and on extending our experience of using it to configure and deploy a range of services and applications. We are now at the stage where want to work with external partners to co-develop the technology and the approach, so we have released the system under an open source licence.

Q: So it’s open source; what’s the licence?

A: We are releasing the system under the LGPL open source licence. This means that third parties can use the SmartFrog framework freely for commercial or non-commercial purposes, and can build proprietary, private components and templates that use the framework. However, they must acknowledge the use of the system, must provide access to any core SmartFrog source code, and must submit any changes to the SmartFrog core system itself back to open source under the same LGPL licence.

Q: Where and when can I get the system?

A: The system is available right now from SourceForge: http://sourceforge.net/projects/smartfrog

Q: What are the costs and benefits of using SmartFrog?

A: There is a cost to using SmartFrog. The largest element of cost is that of initially learning the intricacies of the framework and its methodology since the model of configuration is probably unfamiliar. Once familiarised, though, the cost of creating new SmartFrog components is in direct proportion to the complexity of the task with little or no overhead imposed by the use of the framework. Indeed, the task is often actually simplified, as one of the key aspects of the component model is that the programmer’s interface to configuration data is consistent regardless of how the data is to be obtained - from a static text file through to being dynamically discovered at runtime. The SmartFrog framework hides the precise mechanism being used, and allows new mechanisms to be easily incorporated.

The largest benefit comes once components have been designed and implemented. The flexibility and ease of using SmartFrog components is greatly enhanced over the more traditional model of using configuration scripts and installers, or various ad-hoc models of lifecycle management that might be developed for a specific purpose. These benefits come from the strong separation between configuration data and the functional parts of the components, plus the provision of a rich variety of ways of collating the configuration data.

Consequently, many different configuration models may be implemented, from static data defined in configuration descriptions through to dynamic models using naming or discovery services. These may all be implemented without the core configuration components having any knowledge of how the data is defined embedded in the component code. In practice, this has been shown to greatly improve component reuse and allows component developers to greatly simplify their tasks.

Q: Name the three most important benefits of using SmartFrog

A: The three primary benefits are increased flexibility; increased reliability through removing from the developers the need to consider many aspects that are handled by the system; and improved development time, particularly when the complexity of the overall system starts to grow.

Q: Is there anything else like SmartFrog available today?

A: There are many software configuration systems, but none that take exactly the same approach. Most products fall into one of two classes of system: image installers and software distributors. The image installers use boot-time protocols to install an operating system image. The software distributors, are mostly targeted at distributing software and OS patches to running computers according to some overall policy defined in some database. Neither of these two approaches handle the runtime aspects of lifecycle management, those relating to dynamic binding, rapid reconfiguration to cope with loading or failure, and so on.

Q: How does a user describe the service they want SmartFrog to deploy? How long does it take? Is there a graphical interface?

A: The user must use the SmartFrog notation to define a number of aspects of the configuration - in particular which SmartFrog components to use, their configuration data (or how to obtain it), the relationships between the components (such as start-up order, etc.), where they should be installed and started, and so on. Specific descriptions, parameterised by a number of configuration attributes, or patterns for such descriptions, may be captured in templates that may be used and extended for more specific purposes.

There are currently no GUI tools supporting the development of templates or final descriptions. Such tools are possible and this is an area of active research. However the complexity of large systems is such that GUI-based tools are unlikely to scale well - they work best in limited domains and so supporting GUI tools are likely to be frameworks to allow users to easily develop a specialised tool to go with specific templates. This makes it easier to release templates to end-users who can see high-level abstracted models rather than have to work with the details. A general purpose GUI editor is probably of little use.

Q: How robust is SmartFrog?

A: The core framework has been developed over several years, and has been used for developing many services. Consequently, it is considered to be fairly robust by regular users.

Q: Where can I get more information?

A: Start at www.smartfrog.org, and use the contact methods listed there if you need to get in touch.

Updated 24 June 2007