SmartFrog Reference Manual

Introduction

This manual is aimed at those wanting to use and understand the workings of SmartFrog. It is not a basic tutorial, though hopefully it is not too obscure, either. The notation is described fully, as is the component model. The framework, however, is only outlined. For a detailed reference description of the framework APIs, users should refer to the accompanying Javadoc files.

The manual is divided into several sections:

1. The aims of the SmartFrog system: defining the basic goals of the system, thus ensuring that there is an awareness of these aims to aid in understanding the technical details.

2. The SmartFrog notation, describing the details and semantics of the first configuration description notation to be supported by the SmartFrog framework; other notations are in preparation but are not included in this manual.

3. The SmartFrog component model and framework, defining how to write components and run them within the SmartFrog system.

4. The SmartFrog security infrastructure, describing how SmartFrog ensures that systems are appropriately protected.

A separate document covers the details of installing and running the SmartFrog system. A number of examples are also provided and documented as part of the framework.

This document contains sections that assume differing levels of knowledge and familiarity with the SmartFrog system. It is suggested that a first-time user read only those parts that are essential before experimenting, then progressing to more advanced topics as familiarity develops. To aid in this, sections or sub-sections are tagged with one of the following labels: basic, advanced and expert indicating progressively more advanced topics. If a section is tagged as a particular level of complexity, and a sub-section is considered to be of higher level, the sub-section will be tagged with this higher level.

Aims Of The SmartFrog Framework - Basic

Configuration

For many years HP Labs has been involved in the development of large-scale distributed systems, and in particular management and measurement systems. From this experience, it became clear that configuration is often the major hurdle in the development, adoption and use of such large systems. This experience is supported by evidence from other domains, such as telecom service platforms, large scale e-service hosting environments, and so on. The weight of evidence clearly indicates that many of the problematic aspects of developing, delivering and maintaining such systems are resolved by the introduction of a well-designed, intuitive configuration system. These observations led to the development of the SmartFrog configuration framework described in this manual.

There are several significant reasons for investing in a powerful and flexible configuration environment, which in combination illustrate why this area is in many cases essential for the success of a large system. These are discussed below as a clear understanding of these reasons help in determining the requirements for a supporting environment.

Increased operational reliability

Configuration errors are the major cause of system failure. It is no coincidence that at least one system development inside of HP has termed the development of a tailored configuration system as its ‘high-availability programme’. It is pointless spending money on expensive replicated databases and computation if they contain wrong data, or are carrying out the wrong calculations. From hard experience, they know that the human element is by far the weakest point in any system of even moderate complexity.

Many systems are required to be resilient to a (small) number of failures, providing support for dynamic system reconfiguration in the case of such failures. This should be provided via failure detection mechanisms triggering re-configuration actions within the system components themselves (such as instigating fail-over) and through the configuration system to ensure a consistent view of the current configuration and to provide appropriate re-configuration policy (for example, where to create the replacement components in the case of a processor failure).

Improved quality

After examining the architecture and design of several large-scale systems it became clear that the developers of the various component sub-systems had each created their own configuration infrastructure, often not realizing that this area is of great importance to the overall system. Each makes separate decisions as to format of the data, how it is stored, and so on. In addition, since some aspects of configuration such as configuration description or failure detection and recovery can be extremely complex, the separate development groups frequently do not utilize best practice.

Reduced cost

Costs can arise for several reasons and in several areas such as development, installation and maintenance. For each of these, providing well-defined best-practice procedures and well-implemented support environments for configuration can save significant time and hence money. From experience with several systems, the majority of support calls for these systems (and hence source of recurring cost to the platform provider) come from configuration issues.

Assured correctness and consistency

Validation rules need to be provided to ensure that a configuration is correct before it is deployed into the running system. These rules should include dependencies between various system components (e.g. version dependencies) as well as rules governing repetition (e.g. each web server should run the xxx process and …), replication (e.g. two cooperating instances of this component should exist for reliability...), location (e.g. this component should be close to the database...), and so on. Tools for modelling and reasoning about the configurations are required.

Given a configuration that has been defined and validated, the configuration must then be correctly and verifiably instantiated, preferably automatically, with appropriate error handling in the case of failure. Discovery services must be present to enable binding of services to each other as defined in the configuration, and status monitoring capabilities are required to provide management tools with the ability to monitor the overall state of the system and to ensure it is correct with respect to the desired configuration.

Complex systems may in fact be impossible to configure manually if the requirements change faster than individuals ability to track these changes and carry out the complex reconfiguration tasks. In these cases, automated, adaptive configuration, driven from general rules and auto discovery, is the only solution.

Increased security

System configurations are vital to the integrity of the system. Consequently, in many environments where physical and network isolation cannot be guaranteed, a high level of basic system security must be provided. This involves not only protecting the configuration data itself from unauthorized access, but also the run-time environment must be secure. This includes discovery protocols, component instantiation services, management services and so on. It is typically hard to provide a secure environment when many independent and diverse techniques are used to provide the configuration, so again a single solution implementing best practice is an essential step to ensure system integrity.

Improved Customer Experience

A major issue to be considered in designing systems is that different classes of user have different requirements. All too frequently, the configuration information is designed for the convenience of the system developer not the system operator. Data is required in a form that often does not reflect the skills of the administrator, or maybe is replicated in several files, or distributed over many processors, each of which can lead to a slow and error-prone configuration process. Configuration should be done in ways useful to the operator and adapted to the system and not by expecting the operator to adapt. This can be expensive and hard to implement unless there is extensive support for the systems developers.

The SmartFrog Framework

SmartFrog is a framework for the development of configuration-driven systems. It was originally designed as a framework for building and managing large monitoring systems where flexible configurations are essential. SmartFrog is currently in use within several products, though it is not a product in its own right.

The name reflects its basic design concept – the Smart Framework for Object Groups. It defines systems and sub-systems as collections of software components with certain properties. The framework provides mechanisms for describing these component collections, deploying and instantiating them and then managing them during their entire lifecycle.

The framework consists of three major aspects:

1. The SmartFrog configuration description environment, consisting of a description notation and tools to enable the storage, validation and manipulation of these descriptions.

2. The SmartFrog component model, defining the interfaces that a software component (or a management adapter for a component) should implement. These interfaces are to support the various lifecycle operations such as creation, versioning and termination, as well as management actions such as accessing status information.

3. The SmartFrog configuration management system, which uses these descriptions and management adapters to instantiate the software components and to monitor them throughout their lifecycle in a secure way, including an integrated run-time environment providing capabilities such as discovery and naming.

Notation

The SmartFrog ‘notation’ is in fact defined as a set of open data structures. In principle, this definition can support a number of parsers that provide different textual versions of the notation (for example using XML as a surface syntax). Additionally, it’s possible to develop GUI tools that allow the users to “drag-and-drop” their configurations using the data structures as the common form. At this stage, no generic GUI tools are available for SmartFrog, though experimental versions have been built; usually such tools are normally best tailored to a specific class of system.

The notation is object-oriented, supporting inheritance and extension of configuration descriptions. These descriptions consist of component definitions, associations and relationships between the components, and workflows associated with the lifecycle of the components and the system as a whole. The descriptions may be parameterized enabling multiple instantiations with different configuration data, and validations may be provided which verify that these instances are correct before an attempt is made to deploy the configuration.

The current version of SmartFrog, though in principle able to support multiple textual languages, just provides its own specialized notation “out-of-the-box”. Others are in preparation for future releases.

The notation is not used to define behaviour, merely the structure of collections of components and their relationships with other collections. It is not a programming language. The behavioural part of a component is assumed to be defined in an existing programming language (such as C or Java) and the component will be started as needed by the SmartFrog configuration management system. Currently only Java is tightly integrated. Java adaptors must be used to wrap code written in other languages, and these are relatively simple to implement.

Components

The component model supported by SmartFrog is a simple, extensible set of interfaces providing access to key management actions – such as instance creation, configuration, termination, and so on. A component may be fully integrated (i.e. it may implement the defined management interfaces directly, and hence be written in Java) or it may be independent, in which case a management adapter must be provided. Several standard management adapters or base integrated components have been written to provide common behaviours and these may be extended or modified as appropriate.

Each component (or adapter) must implement a standard lifecycle, implemented as a set of action routines that the environment invokes in the appropriate order and at the right time to carry out the configuration or other management task required. The lifecycle process is governed and controlled by the definition of workflows within the SmartFrog system to provide a very flexible and adaptable environment for carrying out the various configuration tasks.

A complete set of APIs is available to the components that allow them to access the configuration information, locate other components as defined in the configuration and to alter the running configuration if so desired.

Environment

The SmartFrog configuration and management infrastructure is supported by a collection of services, such as:

• deployment – the distribution of code, configuration data and the instantiation of components in the right place with certain ‘transactional’ guarantees

• discovery and naming – providing a number of binding services to allow components to locate each other and communicate

• management – every component is manageable via tools provided with the framework, via the web, or other consoles (if so configured) with no developer effort.

These services are incorporated so as to provide a seamless and coherent programming and configuration model. The benefits of this approach are in providing configuration abstractions to component developers that allow multiple configurations of different scale to be produced without altering the components in any way. The environment is broken into several well-defined functional units, each of which has some specific role to play. Furthermore each of these operates through well-defined and open interfaces, so it is easy to replace the existing functional units, or even to make the selection of which functional unit to use part of the configuration description.

For example, suppose a component, say an SS7 stack, requires the use of another, a real-time database for storing connection information to help the recovery process in the case of system failure. This may be done in many ways. For example, the database could name itself under some well-known name in some well-known naming service, and the stack could find it there. Alternatively, the system may use SLP discovery to locate the database, or perhaps look in a file for this location information. Each approach has advantages in different system contexts, but the programmer typically has to decide up front which to support.

Not so with the SmartFrog integrated environment. The SmartFrog system supports the notion of a binding and provides multiple ways – determined by the environment and driven by the configuration descriptions – for these bindings to be resolved. This includes all the above approaches and others may be added as required. So a programmer need only obtain its binding from the environment and the precise mechanism is handled by the SmartFrog environment as defined by the configuration.

SmartFrog is a framework, and is designed to make it easy to provide additional binding mechanisms as they are required – for example changing the naming service or adding a specialized binding service which uses some other technologies such as databases or directories.

This is equally true of the other services. Consider deployment; it is possible to provide different mechanisms for ensuring that a component is created in the right place. For instance, it might be by hostname, or perhaps by some computer’s role within the system, or perhaps it needs to be close to another existing component. Each of these location mechanisms may be integrated into the run-time environment and then referenced freely within the configuration descriptions.

Final Comments

The design goals for SmartFrog were to produce a very lightweight and flexible configuration and management infrastructure capable of scaling from small systems to very large. This has been achieved through the use of the framework concept and providing users with the ability to alter the low-level semantics by replacing functional units, yet providing standard capabilities by offering default implementations of these units. The system also provides a flexible configuration description notation, with potential for multiple textual or GUI syntaxes to be used targeted at specific system architectures.

Applications of SmartFrog have clearly demonstrated that systems are more quickly implemented using the technology, and that the structure imposed upon the implementations by the use of SmartFrog is beneficial to long-term reliability, usability and manageability.

The Anatomy of SmartFrog

This section attempts to lay out the main aspects of the SmartFrog service deployment framework, describe their relationships, and map them into the structure of the reference document.

As described in the introduction, it consists of three main aspects:

1. The SmartFrog notation, a language in which to describe the configurations, also known as service descriptions.

2. The SmartFrog component model, the way in which programmers create components that are created and managed by SmartFrog as part of a service and which can interact with the system. These are deployed according to the service description.

3. The SmartFrog runtime, the collection of services that exist as part of the SmartFrog system. This is also know as the deployment engine, but is strictly a misnomer since it is in reality a collection of predefined components.

These various components can be seen from the following outline diagram of a SmartFrog system.

These three aspects will now be examined in a little greater detail.

Building Systems with SmartFrog

Introduction

The statement that there is a SmartFrog notation is a simplification of reality. SmartFrog may support many notations, though it provides a 'standard' primary notation out of the box. To enable this, SmartFrog provides a well-defined interface between the language processing parts of SmartFrog and the run-time as a well-defined data model: the set of Java classes that must be used by the language processing to represent the data delivered to the runtime system.

Roughly speaking, the model of SmartFrog language handling is shown in the following diagram:

As is illustrated in this diagram, there may be many notations, each with their own language processing, which at the back-end of that processing produces an instance of the data model that can be understood by the remainder of the SmartFrog system. Alternatively, programmes such as a drag-and-drop gui can produce the data in the correct form directly.

To support the development and use of additional languages, the SmartFrog framework provides a rudimentary structure for integrating language processors. A language processor is assumed to consist of three major steps: parsing, executing some processing phases, and then conversion to the standard data format. The set of processing phases are assumed to be language specific, including having the empty set of phases.

This is illustrated in the diagram above, also showing the associated Java calls used within the framework. These are not important at this stage and are explained in detail later in the reference manual.

Note that the core data mode and the primary notation are closely coupled. This means that in effect the core model can in some ways be seen as a true subset of the primary notation – it could be unparsed into the primary notation and parsed back directly into the core form without requiring any language processing.

Indeed, the two are sufficiently close that the Java classes that are used to directly represent parse-trees of the notation are derived from those of the core model, and much of the same terminology is used in both. So for example, an attribute-set in both is called a Component Description, the only difference being that in the primary notation this may have a super-type from which it inherits, whereas in the core model it may not.

Each notation is assumed to have an associated name, and this name is used in the construction of a parser (selected via a standard language-name to parser-classname mapping). Furthermore, if text files or URLs are handled by the SmartFrog system, the extension associated with that file is assumed to indicate the name of the notation in use. Thus for the primary notation, files should end with “.sf”.

Once converted to the core format, the data represented may be used in several ways:

1. It can be data that is passed to components in the same way as any other data. Indeed many of the components provided as part of the Smart frog distribution exchange such data through their APIs.

2. It can represent the set of components that should be deployed by the SmartFrog run-time.

Now the second case is in fact just a special case of the first, where the data is passed to one of the standard SmartFrog 'Compound' components, such as the ProcessCompound, that understands how to interpret these descriptions as that of a distributed set of components. This duality is described in the following diagram:

The reference manual the primary notation and the core data model.

1. The primary notation is covered in this part of the reference manual. This is the only notation covered in the reference manual.

2. The programming model for interacting with the language framework is given in section REF. This provides details of how to invoke a parser for a specific notation, how to drive the phase-resolution steps of the language, and finally how to covert to the standardized form for handling within the rest of the SmartFrog framework.

3. The core data model is described in section REF. This is only a partial description and the primary source of this information should be the Javadoc for the classes involved.

The Primary SmartFrog Notation

Background

The primary SmartFrog notation has been designed to provide users of the SmartFrog framework a simple, yet powerful, attribute description language. As such, the language has similar aims to that of XML - though it predates XML by a couple of years. There are a number of significant differences between XML and the SmartFrog notation that are worth explaining and this is done in a section 16.

The primary notation is designed to be very close to the common core data model, but provides a number of additional important useability features, including inheritance, linking, attribute placement, functions and various types of well-formedness predicate. Indeed all of the classes used by the parser to represent the abstract syntax tree are derived from those of the core data model.

Attributes

A SmartFrog description consists of an ordered collection of attributes. The attributes are ordered because several of the operations in the SmartFrog framework require an order, for example the order in which the configuration should be instantiated.

Each attribute has a name and a value, this value being either a simple value (integer, string, etc.), or an ordered collection of attributes known as a component description. This recursion provides a tree of attributes, the leaves of which are the basic values. A value may also be provided by reference to another attribute. This is described by the following BNF, where Stream indicates the entry point to the SmartFrog parser.

Stream ::= AttributeList

AttributeList::= (Attribute

| #include String

| ; // allow arbitrary extra ";"

)*

Attribute::= Name Value

Name::= -- | (WORD [ : Name ])

Value::= Component

| SimpleValue ;

| ; // instance of SFNull

From this it is clear that the input to the parser is a collection of attributes, each named and having an optional value. If the value is not present, the value is defined to be an instance of the class SFNull (note that the other way of defining a value of class SFNull is to use the basic value NULL). The reason for providing this feature is to enable the use of attributes where the presence of the attribute is what is important, not its value.

The syntax for a name will be covered later, but for now it can be considered to be either a simple sequence of letters and digits, starting with a letter, or the double-hyphen “--“). The double hyphen is for use at times when the attribute name is not important and so a new unique name is generated and used. This is particularly useful with the function syntax described in Section 5, and most specifically the nary operators.

Include files are covered in more detail in section 3.1, but in general they consist of parseable SmartFrog text which are parsed as attribute lists and unpacked into place within the container attribute list.

Values can be divided up into two main categories: nested attribute sets (components) and the rest (simple values) which include numbers, strings, vectors of these, and so on. In addition it is possible not to provide a value for the attribute, or more precisely to give a null value to it (an instance of the SFNull class). This is captured by the third clause of the BNF for values above.

Simple Values

Values are expressible in several syntactic forms.

SimpleValue::= Basic

| Reference

| Operator

| IfThenElse

| Vector

Basic Values

The primary way is to provide a basic value, a literal syntactic form for the basic core values in the SmartFrog language. The syntax for the basic values is best given by example.

Integer: 345

Long: 65325L or 65325l

Float: 34.76F or 34.76f or 34.76E-10F or 34.76e+10f or 34.76E10f

Double: 1534.45 or 1534.45D or 1534.45d or 1534.45E10 or 1534.45E-10D

String: "this is a string"

Multi-line String: ## This is a string

Over many lines #

Boolean: true

SFNull: NULL // alternatively, leave the value empty

Byte Array: #HEX#AB348eAb#

Consequently, an example of a piece of SmartFrog text is as follows

portNum 4074;

hostname "ahost.smartfrog.org";

isHighPriority false

defining three attributes with the appropriate values.

In addition to these basic values, it is also possible to give vectors of basic values (as opposed to the more extensive vector syntax given below). These vectors are limited to containing basic values, and other vectors of basic values.

userList [| "fred", "harry" |];

empty [| |];

listOfLists [| [| 1,2,3 |], [| 4,5,6 |] |];

The full syntax for the basic values is

Basic::= String

| Number

| Boolean

| ByteArray

| [| [Basic (, Basic)*] |]

| NULL

Number::= DOUBLE

| FLOAT

| INTEGER

| LONG

String::= STRING // "....."

| MULTILINESTRING // ##....#

Boolean::= true | false

ByteArray::= #HEX#....#

| #DEC#....#

| #OCT#....#

| #BIN#....#

| #B64#....#

Note that byte arrays will be definable as hexadecimal (HEX), decimal (DEC), octal (OCT), binary (BIN) and base64 (B64), however only hexadecimal is currently implemented. Depending on the definitional form, the characters that may be used and the number that must be present are different. White space characters are ignored so that neat tabbed layouts may be used. They are treated in the syntax as single tokens.

References

The second form of simple value provided by the language is the reference. A reference is a link between the value of one attribute and that of another. This allows for the definition of data in one place and reused in many, easing maintenance issues for descriptions. References are dealt with in more detail in section 2.2.4, but as a first example the use of a name in the value position provides the link to the attribute of the same name. So for example:

x 42;

y x;

defines y to be the same value as x, namely 42;

Operators

The remaining three forms of value definition are syntactic sugar for the use of functions. The semantics of functions are outlined in section 5 and described in detail in section . However, their syntax is

Operator::=

(

(UnaryOp SimpleValue )

| (SimpleValue [ BinaryOp SimpleValue ])

| (SimpleValue [ (NaryOp SimpleValue)* ])

)

UnaryOp::= !

BinaryOp::= - | / | == | != | >= | > | <= | <

NaryOp::= + | * | ++ | <> | && | ||

This states that the use of an operator is always defined within brackets (...) and that there are three types of operator: unary, binary and nary. Although with the nary operators, more than one instance of the operator symbol is present, it must always be the same operator; they cannot be mixed. However, other operators may be nested within another set of ( ). The following examples may help to make the syntax clear:

aTruthValue true;

anotherValue (! aTruthValue ) // the only unary operator: not

aNumber 45;

aMinus (100 – aNumber) // a binary operator

aSum (aNumber + aMinus + 100) // an nary operator

These operators are all converted at time of parsing into the template representation of a function, and hence at no time willl operators appear in an description generated from the parsed form.

Note that attribute names can contain rather a large number of special symbols, such as “+” and “-”. This means that there is a danger that an operator may lexically stick to a name if not separated from it by white space. As a consequence, it is good practise to always use white space around operator symbols.

If-Then-Else

Similarly to operators, if-then-else expressions are shorthand for the template form. This is described in detail in section . The syntax for this expression form is

IfThenElse::= IF SimpleValue

THEN SimpleValue

ELSE SimpleValue

FI

The line breaks being, of course, optional. The “if” value is a boolean and depending on the result the expression takes the value of the “then” or “else” values. The FI is merely a closing keyword. An example of its use is:-

val1 42;

val2 43;

diff IF (val1 > val2) THEN (val1 – val2) ELSE (val2 – val1) FI;

Vectors

The final form of simple value is the vector. Vectors are lists of values and are constructed using the vector function described in section . However, to simplify its use, the following syntactic form has been provided.

Vector::= [ [SimpleValue ( , SimpleValue) *] ]

Thus a vector is a sequence of values separated by “,” and delimited by “[ ]”. If no value is provided within the vector, an empty vector is returned. Vectors may be nested to produce vectors of vectors. Example uses of vectors are:

v1 [1,2,3];

v2 [9,8,7];

v3 [v1, v2]; // same as [[1,2,3],[9,8,7]]

Note that there are two syntaxes for vectors – the one given here which provides the ability to embed references and which therefore requires a degree of processing (known as resolution). It is parsed into the use of the vector function rather than directly into a vector. The other form, using the “[| |]” delimiters, parses directly into a vector and hence may not have references within the definition. The reason for having simpler form in addition to the more general form is is that there are times when the fully processed (resolved) data structures need to be unparsed, and then re-parsed at some future time without further transformation. A example of this is during the signing of a description for security purposes.

Values of Other Classes

The set of values that can be described by the use of the language is limited to a few basic classes and collections of these. It would be useful to be able to include values from other classes in Java. These in principle can be generated in functions, or some user-defined phase, and added to the attribute sets. However, there are problems with this for SmartFrog, and in particular with some aspects of the security where descriptions transformed to core form need to be signed and this is restricted to the known classes.

Consequently the conversion to the core form ensures that the values represented in the attribute sets, the component descriptions, are limited to these core classes. If other values need to be held within the tree, it is recommended that they are held in serialized form within a ByteArray value. This will need to be deserialized by the component at the time of deployment.

Component Descriptions

Attributes may have values that are collections of other attributes, known as component descriptions. They obtain their name from the fact that they may be interpreted by the framework as the description of a component, though they may equally be used to describe structured data.

A component description consists of two parts, a reference to another component description to act as a source of attributes (the type), and a collection of attributes that are then added to, or override, the attributes of the referenced collection (the body). The syntax is:

Component::= extends [LAZY] Type Body

Type::= [ NULL | BaseReference ]

Body ::= ( ; | { AttributeList } )

The LAZY keyword may be largely ignored; it is merely a tag and only has a semantic effect during the deployment of a SmartFrog application. Extension of an existing LAZY component description does not inherit the tag.

Both the reference and the attribute list are effectively optional. If neither is present, the resultant attribute list is defined to be empty. The syntax is most easily explained through an example:

SFService extends { // an implicit extension of NULL

portNum 4047;

hostname "ahost.smartfrog.org";

administrators ["patrick"];

}

UseableService extends SFService {

// an extension of the previous compnent

portNum 4048; // override the definition of portNum

users ["fred", "harry"]; // add a new attribute

}

The text consists of two attributes, both of which have values that are collections of attributes. The second of these, UseableService, is defined as an extension of the first SFService, with two attributes added to or overwriting those inherited. The text is semantically identical to the following:

SFService extends {

portNum 4047;

hostname "ahost.smartfrog.org";

administrators ["patrick"];

}

UseableService extends {

portNum 4048;

hostname "ahost.smartfrog.org";

administrators ["patrick"];

users ["fred", "harry"];

}

Note that the attributes in a component description are ordered and that when an attribute is overwritten it maintains its position, but when it is a new attribute it is added to the end. The process of expansion of the inheritance in this way is known as Type Resolution and is explained further below.

Note also that the parsed stream is considered to be in an implicit, anonymous (i.e. not named in an outer component description), component description known as ROOT.

The example is also shown in the diagram. It clearly shows that there are two kinds of relationship between component descriptions. One is the containment relationship, where a component description contains an attribute that is itself a component description. The second is the inheritance or extension relationship. This second class of relationship is one that can be transformed, by type resolution, to an equivalent one containing no extension (also indicated by the NULL extension).

Whilst the extension relationship is merely a convenient way of defining attributes, the containment hierarchy is a more fundamental construct. It should be noticed that that containment hierarchy effectively provides a naming scheme by which attributes may be referenced. In this it is similar to other such named hierarchies, such as directory hierarchies common in files systems.

Types vs. Prototypes

SmartFrog does not define types for attributes and components. Rather it defines the notion of a prototype (c.f. the programming language Self). Each attribute whose value is a component description may be considered as a prototype for another: it may be taken and modified as appropriate to provide the value for the new attribute. The mechanism for this is the extends construct.

Any attribute whose value is a component description may be, at a later juncture, selected and modified to provide a new component description to be bound to a name. This new attribute may be further modified by subsequent attributes. In this way, it is possible to provide partial definitions, with default values for attributes, to be completed or specialized when used. This provides a simple template mechanism for components.

Consequently, there are no separate spaces of types and instances; every component is logically an instance, but may also be a prototype for another. However, it is clear that in providing descriptions, some components will be defined with the intention that they be used as prototypes for other components, whilst others will be defined without that expectation. Whilst this may appear strange in the first instance, it turns out to be one of the main strengths of the SmartFrog notation.

References

References may occur in three places in the syntax: as the name of an attribute – known as a placement, as a reference to the extended component (the prototype) of a component description, and as an attribute value referring to another attribute whose value is to be copied – known as a link.

The primary purpose of a reference is to indicate a path through the containment hierarchy defined by the components. In this, it is similar to the notion of path common in file systems in operating systems such Linux. A path defines a traversal of the directory hierarchy, a structure similar to the component hierarchy.

The syntax for references is as follows:

Reference::= [LAZY] BaseReference

BaseReference::= ReferencePart ( : ReferencePart)*

ReferencePart::= ROOT

| PARENT

| (ATTRIB WORD)

| (HERE WORD)

| THIS

| WORD

| (PROPERTY WORD)

| (IPROPERTY WORD)

| (HOST (WORD | STRING))

| PROCESS

Thus, a reference is a colon-separated list of parts each of which indicates a step in the path through the containment tree. Examples of references are:

PARENT:PARENT:foo:bar

ATTRIB a:b

ROOT

x

HOST 15.144.56.65:foo:bar

Normally a reference indicates a path through the containment tree to an attribute whose value should be copied, or a component description in which an attribute should be placed. These references are “resolved” during the language processing to eliminate them and to carry out the appropriate copying or placement.

However, occasionally the reference itself is the desired value, or the reference cannot be resolved during language processing as the data referenced is not available until a later stage. Under these circumstances, the keyword “LAZY” is prefixed to the reference to indicate that the reference resolution should be delayed.

The general rule for the interpretation of a reference is that the reference is evaluated in a context (a component description somewhere in the description containment tree), and that each step moves the context to a possibly different component for the remainder of the reference to be evaluated. This is equivalent to path evaluation in a Linux file system, the path is evaluated in a current directory, and each part of the path moves the context to another directory.

The semantics of each of the reference parts is as follows: starting at component in which the reference is defined…

• PARENT - move context to the parent (container) component if it exists, fail otherwise (c.f. Linux “..”)

• HERE WORD - look for the attribute named “word” in the current context, fail otherwise

• ATTRIB WORD - look for the attribute named “word” in the current context or anywhere in the containment hierarchy (the closest is chosen), move to the context defined by this attribute, fail if no attribute is found in the containment hierarchy

• ROOT - switch context to the outer-most component (normally the implicit root component (c.f. Linux “/ “)

• THIS – keep the context the same, don't switch (c.f. Linux “.”)

• WORD – the interpretation of the WORD depends on the location. If it is the only part in the reference, or the first part, it is interpreted as ATTRIB. If it is the second or later part of a reference it is interpreted as HERE.

Some examples of references (in this case link references) are as follows:

The arrows in the left-hand text show the path followed as the references are resolved to obtain the referenced attribute values, noting that the resolution of ref3 will follow the resolution of ref2. The contexts traversed as the resolutions progress are shown boxed and the right-hand text shows the result of resolving the three links.

In addition to these four structural reference parts, there are four others that are not appropriate for all circumstances and are not related to the containment hierarchy. These are

• PROPERTY WORD – return the value that is the Java system property named WORD. It may only occur at the end of a reference, and only in a link. Syntactically it may occur anywhere, however the remainder of the link is ignored. It is usually used in conjunction with LAZY. Without LAZY, the value of the property at the time of parsing will be used; with LAZY the application run-time value of the property will be used when the link is resolved – see section 3. The property is always a string.

• IPROPERTY WORD – as for PROPERTY, but the property is interpreted as indicating an integer which is parsed and returned as such.

• HOST (WORD | STRING) – switch to the context of the process compound on the host name WORD (or STRING – which must be used if supplying an IP address, but may also be used with a host name). This reference part is also used in conjunction with the LAZY keyword and only in links. It is used to provide a naming service for applications within the SmartFrog system. Again, without LAZY the parser will look-up the value in the remote process compound, with LAZY this will be done at run-time when the link is resolved – see section 3.

• PROCESS – switch to the context of the process compound of the current process. This is also used in conjunction with the LAZY keyword and only in links. It is used to provide a naming service for applications within the SmartFrog system. Again, without LAZY the parser will look-up the value in the remote process compound, with LAZY this will be done at run-time when the link is resolved – see section 3.

The above rules determine the general interpretation of references. However, each of the syntactic contexts has its own slight semantic variation; these variations appear in the detailed definition of the semantics for references.

Reference Elimination – Resolution

The key to the semantics of the SmartFrog notation is the process by which references are eliminated. This is necessary for each of the three syntactic locations where references may occur – prototype references, placement references and link references. The process by which references are eliminated is known as reference resolution. However, each type of reference has a different notion of resolution and so each has a specific resolution action – known respectively as type resolution, placement resolution and link resolution. This last name is historically also known as deployment resolution; this old name appears in parts of the API and is kept for backward compatibility. The resolution steps are described in more detail in the next few sub-sections, and then revisited as a whole to examine their interaction with each other.

Prototype References

References to prototypes, as defined in the following syntactic context,

Component ::= extends [LAZY] BaseComponent

BaseComponent ::= [Reference] ( ; | { AttributeList } )

are resolved as described above except in one respect: if the reference to the prototype consists of a single WORD part, it is interpreted as ATTRIB WORD in the usual way.

Thus, the following are equivalent

Foo extends Bar { …}

Foo extends ATTRIB Bar {…}

This is to provide a greater degree of convenience when referring to a prototype as these are most often defined in the outermost implicit root context, and frequently defined in an included file. Using this re-interpretation using ATTRIB, rather than adding an implicit ROOT reference part to the front, ensures that global definitions of prototypes at the top level may be locally overridden if required.

The following example demonstrates most of the situations:

Foo extends { a 1; }

Bar extends {

foo extends Foo;

}

Baz extends {

foo extends {

b 2;

}

foo1 extends Foo; // recall - this is equivalent to ATTRIB Foo

foo2 extends ROOT:Foo;

foo3 extends PARENT:Foo;

foo4 extends PARENT:PARENT:Foo;

}

After type resolution, which includes the merging and overwrite of attributes as described in section 2.2.2, the example is equivalent to:

Foo extends { a 1; }

Bar extends {

foo extends { a 1; } // ATTRIB Foo finds the outermost

}

Baz extends {

foo extends { b 2; }

foo1 extends { b 2; } // ATTRIB Foo finds the closest enclosing

foo2 extends { a 1; } // ROOT:Foo finds the one in the root

foo3 extends { b 2; } // PARENT:Foo finds that in the parent

foo4 extends { a 1; } // PARENT:PARENT:Foo finds that in

// the root (in this case)

}

Placement References

An attribute’s name may be a reference, as described in the syntactic clauses

Attribute ::= Name Value

Name ::= BaseReference

This is not completely accurate, as the syntax in fact limits references to being a reference containing WORD values, the other reference parts are considered erroneous.

The resolution of the reference is again largely as described above, with the following modification.

The last reference part of the reference must be a WORD and is treated differently. This word part is not strictly part of the reference, but is used to identify the name of an attribute that is to be created (as opposed to referenced) in the context of the prefix part of the name reference. Thus in the attribute definition

foo:baz:bar 42;

the foo:baz is a reference to a location, bar is the name of the attribute to be created in that context.

In most cases, the name consists only of that final WORD leaving the prefix reference empty, indicating the current context. Thus, the attribute is defined in that current context. Where a non-empty reference prefixes the final word, the reference is used to determine the appropriate context and the attribute with the given name is placed into that context.

Consider the example

Service extends {

portNum 4089;

}

Service:portNum 4074;

Service:hostname "ahost.smartfrog.org";

The prefix reference Service: is de-referenced to indicate the Service attribute. The two prefixed attributes are therefore placed within that reference context, overriding or placed at the end of the context as appropriate. Thus, the example is roughly equivalent to the following (there are some differences in their behaviour as prototypes):

Service extends {

portNum 4074;

hostname "ahost.smartfrog.org";

}

The act of placing the attributes into a location is known as placement resolution, and it occurs simultaneously with the removal of the reference-prefixed attribute from its defining context.

Placement of attributes can lead to a great deal of confusion if not used properly. It reacts in interesting ways with type resolution; this interaction explained in the section on resolution.

Link And LAZY Link References

Frequently, attributes need to take on the same values as other attributes. This can be for many reasons:

• to avoid repetition of values at many points in a description making it easier to maintain that description

• to hide the structure of the description to a program; explained further in section 3.

• to provide a means of simple parameterization; explained further in the section .

This association between the value of one attribute and that of another is defined by providing a reference in the place of a value of the attribute. This reference is resolved relative to the context at the point of definition.

Consider the following example, in which a server and a client both need to know the TCP/IP port on which the server will listen.

System extends {

server extends {

portNum 4089;

}

client extends {

portNum ATTRIB server:portNum;

}

}

The system contains a server and a client. The server and client both have an attribute portNum, with that of the client being defined as a link to that of the server.

There is a resolution step, known as link resolution (and occasionally deployment resolution), which replaces references by the values that they reference. During the resolution phase, chains of links are resolved appropriately.

In the above example, the definition of System is equivalent to the following:

System extends {

server extends {

portNum 4089;

}

client extends {

portNum 4089;

}

}

Consequently, both the server and client share the same value and maintenance is eased in that should the port number need be changed, this need happen in only one place in the description.

It is frequently the case that the link itself is required as a value; i.e. the link should not be resolved to the value that it might refer to within the description. This reference may then be used within a SmartFrog application after deployment, for resolution at run-time rather than at the time of parsing the description. The primary use for this is described in section 3.

In order to provide a reference value, rather than have it resolved to the value of another attribute during link resolution, the keyword LAZY may be prefixed to the link to indicate that the link resolution should not resolve the link. An example of this is:

System extends {

server extends {

foo 42;

}

client extends {

myServer LAZY ATTRIB server;

}

}

In this case, the client’s attribute myServer is a reference to the server, not a copy of the server component. As is, resolution will have no effect, as the link will be left to be the attribute value. If the keyword LAZY had not been present, the following would have been the result of resolution:

System extends {

server extends {

foo 42;

}

client extends {

myServer extends {

foo 42;

};

}

}

The word LAZY is an indication that it will be resolved at run-time – so far as the notation is concerned, this means that the link is the value.

Comments

The SmartFrog notation follows most modern languages in providing both end-of-line comments and multi-line bounded comments. The syntax for these is identical to that of Java, namely

• // this is a comment to the end of the line

• /* this is a comment which is terminated by */

Include Files

A stream of text may reference include files at certain points in that text. Unlike a C include file, though, the include file is not merely textually embedded into the original stream. Rather the include file is itself parsed (and must be syntactically correct) as a stream in its own right. Every stream must parse as a collection of attribute definitions, and this is equally true of the include files.

Include files may only be used within attribute lists (i.e. at the top level or within a component definition). The collection of attributes from the include file are simply added to the attribute list being parsed in the container stream.

Consider the following example:

• file foo.sf contains:

foo extends {

a 42;

}

• the primary stream is:

#include "foo.sf"

system extends {

myFoo extends foo;

#include "foo.sf"

}

After the parsing is complete (but before type resolution), the following is obtained:

foo extends {

a 42;

}

system extends {

myFoo extends foo;

foo extends {

a 42;

}

}

It should be noted that because includes may occur within other component descriptions, this may be used as a naming mechanism to prevent clashes of attribute name within multiple include files. Consider

• file foo1.sf contains

foo extends { a 42; }

• file foo2.sf contains

foo extends { b 42; }

• the primary stream contains

foo1 extends { #include "foo1.sf" }

foo2 extends { #include "foo2.sf" }

sfConfig extends {

bar extends ATTRIB foo1:foo;

baz extends ATTRIB foo2:foo;

}

If the includes had not been buried within separately named components, but both had been included into the top level, only the second of the two mentioned foo attributes would have been available for extension. The second would override the first.

sfConfig

A stream contains a whole collection of attributes at the top level. Most are merely there to act as building blocks – prototypes for building others. Typically, there is only a single attribute that is the essence of the description – that which describes the desired configuration and is not merely a building block on the way. By convention in SmartFrog, the reserved attribute name sfConfig defines this special attribute and all the tools provided respect this convention.

Thus, when a stream is parsed to an attribute set, the top-level attribute sfConfig defines the system; the rest are ignored, apart from providing definitions for extensions and other resolutions. This is equivalent to the Java language use of the “special” method main(…) to indicate the entry point to a program. The entry point to a configuration description is sfConfig.

Thus in the following example, the attributes def1, def2 and def3 are only present for the purposes of defining sfConfig, and it is only this last attribute that represents the actual configuration description.

def1 extends Prim {…}

def2 extends Compound {

foo extends Prim {…}

bar extends Prim {…}

}

def3 extends Prim {…}

sfConfig extends Compound {

d1 extends def1;

d2 extends def2;

d3 extends def3;

}

Resolution – Semantics For The SmartFrog Notation

Resolution is the process by which the raw SmartFrog definitions, with their extensions, placements and links, are turned into the set of attributes that they semantically represent.

In addition to these three steps, there are other steps (phases) in the complete semantic manipulation of the SmartFrog notation, such as function resolution, predicate checking and any user-defined phases. These are described in separate sections as they are somehow less core to understanding the language.

There are two ways of representing the semantics, both roughly equivalent.

1. By defining how the value of an attribute identified by a reference is obtained from a description; defining the semantics by providing a function from reference to value for all possible references.

1. By defining a set of transformation rules that eliminate the complexity of the typing (by expansion), placement (by relocation) and linking (by value copy), resulting in a normalized form of a description containing merely a hierarchical set of attribute lists.

Either of these two forms of semantic definition would do, however the definition of the semantics through transformation has a distinct advantage: these transformations are required in practice and hence are implemented within the SmartFrog system. Thus, an understanding of these transformations is essential to the use of SmartFrog.

The three transformation steps are known in SmartFrog as resolution steps. These are respectively type resolution, placement resolution and link resolution. They are carried out in that order: first the types are expanded, then attributes placed into the correct context from the context in which they were defined, and finally links are resolved.

It should be noted that the entire description is type and place resolved, but only the top-level sfConfig attribute is normally link resolved. In general if the other top-level attributes are link resolved, errors will occur; they are only present to be available as prototypes. Further, unnecessary work will have been done.

The algorithms defined here for the transformations are the result of much empirical experimentation – other transformation algorithms produce more regular semantics, others are more efficient. However, those presented here are a balance between performance and semantic simplicity. They provide a great deal of control over the semantics of the resolution process.

Type Resolution

Type resolution is the expansion of the prototype reference optionally provided in the extends part of a component description. The syntactic form for a component description is roughly

name extends Reference { AttributeList }

The reference refers to a prototype that is to be extended by the attributes in the provided attribute list. This process of type resolution is a depth-first pass over the root component description, in the order of definition of the attributes.

• Copying the prototype indicated by the reference, creating a new component description

• Replacing the attribute values of the new component description also mentioned in the attribute list (i.e. the value, but not the order, changes)

• Adding the remaining attributes at the end of the new prototype.

• Type-resolving each of the component description’s attributes if they are component descriptions.

If the prototype reference indicates a component description that is not yet resolved, it resolves it first before copying: i.e. each type resolution is carried out with respect to the location where the prototype is defined. The other point to note is that if the reference is only a word, it is interpreted as ATTRIB word for the purposes of locating the prototype for the component description.

If, at the end of the process, one or more component descriptions have failed to resolve, in that their prototypes cannot be found, the whole resolution process ceases and an exception is thrown indicating the missing prototypes and the locations at which they are referenced.

Note that any references that may be copied as part of the extension process are not modified. Hence, copied placements are now relative to the new location and copied links similarly. Prototype references are never copied since a prototype is always resolved before copy.

Placement Resolution

Placement resolution is the process by which the attributes are placed into the correct location. Attributes are named, and this name may contain a reference to a component description as well as the name by which it is to be known in that component description. If the reference is not present, the attribute is assumed to be in the correct component description as defined.

Thus in the example attribute declaration:

foo:bar:baz 42;

The foo:bar: defines the target component description, and baz defines the name for the attribute in that component description.

Placement resolution is the transformation process that results in the attribute definitions being removed from their point of definition and placed in the target component descriptions. The process is a multi-pass process, for each pass:

• traverse the component description hierarchy

  • depth first

  • visiting the attributes in the order of definition (as determined by type resolution)

• each attribute visited is examined, if it should be placed elsewhere – try to do so, if it fails – leave as is.

The pass is repeated until one of the following occurs:

• there are no placements left to transform

• no placements have been successfully carried out, and at least one placement has failed

In the first instance, the placement resolution has successfully completed, the second it has not and an error is generated.

To see why multiple passes are necessary, consider the following:

foo extends {

a 21;

}

foo:bar:a 42;

foo:bar extends { b 34; }

In the first pass, the attribute foo:bar:a is first to be placed, but it fails since foo does not yet contain foo:bar as a component description. Also in the first pass, but later since it is defined later, foo:bar is placed, giving

foo extends {

a 21;

bar extends { b 34; }

}

foo:bar:a 42;

This leaves a placement incomplete so a second pass is required. This time it succeeds, resulting in

foo extends {

a 21;

bar extends {

b 34;

a 42;

}

}

This order dependency does not have much of an effect, except for when two identically named attributes are placed into the same component description. At this point understanding the order of resolution becomes important.

Since placement resolution is carried out after type resolution, the following consequences should be noted:

• As type resolution is carried out before placement, attributes placed into a prototype will not be inherited by those extending the prototype.

• Again, as type resolution is carried out before placement, do not place an attribute that is to be used as a super-type; it will not be found.

• Wherever possible, placement should be restricted to referencing downwards into a structure from the point of attribute definition. Descriptions can be very hard to understand if PARENT, ROOT or ATTRIB are used in a placement reference; this particularly so within a component description to be used as a type. As a consequence, this release of SmartFrog does not permit these reference parts to be used in a placement.

The reason why type resolution is done before placement resolution is that the normal use for placement is to “fill-in” empty “attribute slots” in a prototype. As each instance of the prototype will in general need differently filled slots, placement must be done after the type has been resolved for each instance.

Note that placement of attributes whose values are links do not modify the links to correct for the new location. Thus, links are resolved with respect to where they are placed, not where they are defined.

Link Resolution

Link resolution is the most straightforward of the three forms of resolution; all links are resolved in their location after type and place resolution, and the referenced value replaces the link as the value of the attribute. There are a number of points to note:

• Only links that are not LAZY are resolved; those that are LAZY are left unresolved with the link itself being the value.

• If the value of the attribute is a link, this is first resolved and the result of that resolution is used.

• Links are always resolved in the contexts in which they are located after the type and placement resolution phases are over, not necessarily those in which they were defined.

• Links referring to an attribute whose value is a LAZY link will leave the LAZY link unchanged, this being the attribute’s value.

• In resolving a link, the value of the attribute referenced is not copied, but shared, at the original point of definition if this is relevant (e.g. For component descriptions and their parent). Thus any operation that affects the value of this data has an impact on all parts of the tree that share this data. The only operations that affect attribute values in this way are functions (or possibly a user phase).

  Sharing has almost no effect on the language semantics unless the data shared is a component description. In this case the parent of the data remains that of the location of definition. This has an impact on how links within that component description are resolved, using the original parent, and not relative to the context in which the link was defined.

  An explanation of the consequences of sharing is given in section REF.

The Difference Between Types and Links

On the surface, there are many similarities between the definitions of x and y in:

Foo extends {

a 10;

}

x extends Foo;

y Foo;

They both appear to end up by having the definition of a component description containing a.

One obvious difference is that since they occur each side of place resolution, a placement into Foo will affect y but not x. However there are more subtle differences to do with the sharing of data with links, rather than the copying of data with extends. Consider the following example:

data 1;

Foo extends {

a data;

}

example extends {

data 100;

x extends Foo;

y Foo;

}

In this definition, example:x:a has the value 100, whereas example:y:a has the value 1. The reason for this discrepancy is that the extends copies the definition of Foo and the following link resolution for data is done relative to the copy's location. The link, on the other hand, simply links to the definition of Foo in its existing position, and there the value of data on resolution is 1.

The difference can also be highlighted using one of the functions, such as next that return a different value at each use. Consider the following description:

#include "org/smartfrog/functions.sf"

example extends {

x extends next;

y extends x;

z x;

}

Assuming that this is the first use of next, example:x will have the value 1, example:y will have the value 2, but example:z will have the value 1. This is because it shares the result of the function bound to example:x.

Note that at the very end of the language processing as part of the conversion to the core data model, the sharing is eliminated and each attribute will have its own copy of the value. This is explained in detail in section REF.

Template Parameterization Pattern

When extending a prototype, it is normal to override the values of certain attributes to customize the prototype to its actual use. The simplest way is to extend with the replacement attribute – however this only works for a top-level attribute. Modification of attributes deep in the structure requires the placement of the overriding attribute into the correct context, as in the example:

Service extends {

hostname "localhost";

portNum 4567;

}

ServicePair extends {

service1 extends Service ;

service2 extends Service ;

}

sfConfig extends ServicePair {

// user needs to know structure of ServicePair

service1:hostname "riker.smartfrog.org";

service2:hostname "ackbar.smartfrog.org";

}

This works adequately, but it has the disadvantage that the use of the ServicePair prototype requires knowledge of its structure, though it does have the advantage that any attribute in the structure may be changed if necessary. However, under normal circumstances, there are attributes whose values are expected to change, and others that are not. Under these circumstances, it would be good if the description could be parameterized on these attributes. However, the normal form of parameterization as provided in programming language functions is not a good fit to the SmartFrog notation semantics – so the language provides a way of finding a way of hiding the structure of a description and making it easier to override “deep” attributes.

This technique, more of a pattern for the use of links, is shown in the following example:

Service extends {

hostname "localhost"; // default value

portNum 4567;

}

ServicePair extends {

s1Host "localhost"; // provide default value

s2Host "localhost";

service1 extends Service { hostname s1host; } // lift attribute

service2 extends Service { hostname s2host; } // ditto

}

sfConfig extends ServicePair {

// user needn’t know structure of ServicePair

s1host "riker.smartfrog.org";

s2host "ackbar.smartfrog.org";

}

It is clear that the use of ServicePair requires only the extension with top-level attributes to set the attributes deeply defined in the Service prototype. This pattern, of the use of links lifting an attribute value to one provided in the outermost context, is called the parameterization pattern and is very frequently used.

Note that if a default value for a lifted attribute is not given within the description (in this case ServicePair provides defaults for both the lifted attributes s1Host and s2Host), a deploy resolution error will occur if the parameter is not provided at time of use, since the value to resolve the link will not be found.

Functions and operators

SmartFrog provides users with a small number of predefined functions to improve the expressiveness of the descriptions. In addition, it provides mechanisms by which users may add their own functions, effectively providing an escape mechanism into Java. These functions, whilst not part of the SmartFrog language, are provided for convenience. The mechanism, a special case of a more general phase mechanism, is described in detail in section .

Functions appear, to the language, as predefined component descriptions that may be extended; the parameters are given as named attributes within the body of that description. For example, a use of the string concatenate function is

#include "/org/smartfrog/functions.sf" // the standard functions

val 42;

myString extends concat {

-- "the meaning of life is ";

-- val

}

that results in the value of the myString attribute being "the meaning of life is 42". The names of the attributes have no effect in this case, the strings being concatenated in the order of definition, but may be important for some other functions.

Functions are evaluated inner-first, providing for the nesting of function application, and are evaluated after all the other resolutions steps have be completed. The definitions are themselves affected by these resolutions. Thus a function may be extended with the resultant extension also be a function. The current set of predefined functions is given in section 14.

In order to make the use of functions more natural, some syntactic forms are provided that appear to be infix or prefix operators. However, these are simply translated into the relevant template form during parsing. A more compete description of this process is given in section 14.

Predicates, Assertions and Schemas

It is frequently useful to be able to define a set of well-formedness conditions on the use of a template in order to guarantee that its use is correct. However, this should be done in a way in which all the benefits of template extension are not lost. To this end, an additional phase, similar to that defined for functions, is included which will check predicates defined and attached to a template.

There are three predicate types provided as part of the SmartFrog framework. These are the assertion predicates, schema predicates and the TBD (to be defined) predicates.

The most flexible predefined predicate supplied by the SmartFrog framework is the schema, a description that describes the set of attributes a template should contain. Users may add their own predicate types through a similar escape mechanism to Java provided for functions. Schemas are described in detail in section 15.

Schemas are best described through the use of an example, in this case of a template for a web server component. The example defines a schema for a web server template, and defines the template linked to the schema.

// the definition of schemas

#include "/org/smartfrog/predicates.sf"

WebServerSchema extends Schema {

port extends Integer;

directory extends OptionalString;

}

WebServerTemplate extends Prim {

schema extends WebServerSchema;

port 80; // default value

}

Note that the name for the attribute linking the template to its schema need not be, as in this case, schema. Indeed, a template may have more than one schema attached as attributes, in which case the uses of the templates are checked against all schemas attached. Schemas must extend the base schema template Schema.

Schemas may be extended in the same way as other templates, and their uses may easily be extended through placement as illustrated in the following examples.

// the definition of schemas

#include "/org/smartfrog/predicates.sf"

ThreadedWebServerSchema extends WebServerSchema {

minimumThreads extends Integer;

}

ThreadedWebServerTemplate extends WebServerTemplate {

// overwrite existing schema with extended schema

Schema extends ThreadedWebServerSchema;

minimumThreads 7;

}

AlternativeThreadedWebServerTemplate extends WebServerTemplate {

// add to existing schema

schema:minimumThreads extends Integer;

minimumThreads 7;

}

Note that schemas are entirely optional and need be used only if desired. The value of a schema is that it provides a strict definition and the potentially type of the attributes, both required and optional, of a component. This should make it easier to work with, and so benefit users of the component.

Similarly to schemas, assertions are descriptions that are interpreted as a predicate. An assertion consists of a description that contains attributes that should all evaluate to true - any attribute that evaluates to false, or indeed any other value, is considered to be an assertion failure. The names of these boolean attributes are not significant other than as documentation. There is an implicit conjunction (and) between the various assertion attributes given.

An assertion description must extend Assertion, and must be included in the description to which it applies in the same way as a schema must.

An example of an assertion is

// the definition of assertion

#include "/org/smartfrog/predicates.sf"

WebServerAssertion extends Assertions {

portValid ((port == 80) || (port == 8080) || (port == 8088));

}

WebServerTemplate extends Prim {

schema extends WebServerSchema;

assert extends WebServerAssertion;

port 80; // default value

}

In the same way that attributes may be added to an existing schema, attributes may also be placed into an “Assertions” description, or more than one “Assertions” may be provided.

The TBD predicate is used to indicate that a specific attribute still requires to be assigned a value. If it has not been assigned, and an attempt is made to use it, an appropriate error message is given.

An example of the predicate is as follows:

#include "/org/smartfrog/predicates.sf"

aTemplate extends Prim {

sfClass “org.smartfrog....”;

anAttribute TBD;

}

sfConfig extends Compound {

anInstance extends aTemplate;

anotherInstance extends aTemplate {

anAttribute 45;

}

}

Here, the attribute anAttribute of aTemplate is defined as TBD, so any use of the template that does not set this value will generate an error. In the definition of sfConfig, the first use, to define anInstance, is erroneous whereas the second to define anotherInstance is valid.

The TBD attribute ("To be determined") is a simple substitute for the more rigorous schema declaration. Note that the type of the attribute is not defined, which can be a useful feature.

Mapping to the Core Data Model

The attribute sets produced by the above phases are now simple enough to be mapped into the core data structures supported by the SmartFrog runtime. These data structures do not support extension, placements, functions or predicates – so all these have to be resolved away. Links are supported, but they are considered as values and have no further special meaning – they are all assumed to be LAZY links.

The translation into these core data structures is therefore straight-forward apart from one additional point: the structures produced by the phases can share data, but this is eliminated by copying. If this copying involves Component Descriptions, these are also parented into the part of the tree into which they are being copied.

The reason for this sharing elimination is to do with the semantics of the distributed system. Whilst all the data is local it could make sense to share data as it is more efficient, although care has to be taken when data is changed behind the scenes with side-effects on other parts of the tree. However, when parts of the tree get mapped to different processes during deployment, the data has to be copied and the sharing broken in any case. To ensure a common semantics between local and remote deployments, separate copies are taken at all times.

This sharing elimination is illustrated by the following diagram. Note that the parent link from back from the foo attribute's data only exists if the attribute is itself an attribute set (a component description).

Primary Language Processing

Phases are a way transforming the SmartFrog parse tree into the final form ready for deployment (or other purpose). Each phase is a pass over the component description hierarchy carrying out an action controlled, in the case of user-defined phases, by attributes defined within the descriptions.

Under normal circumstances users will not need to know about phases or how to modify on adapt them, the default collection of phases is already correct for most purposes.

The predefined phases for the default language are as follows:

• type – carry out type resolution on the component description hierarchy; this is predefined and does not rely on attributes in the tree to trigger it.

• place – carry out place resolution on the component description hierarchy; this is predefined and does not rely on attributes in the tree to trigger it.

• link – carry out link resolution on the component description hierarchy; this is predefined and does not rely on attributes in the tree to trigger it.

• sfConfig – not really a phase, rather it controls where the phases are applied. Its effect is that for the remaining phases in the current phase list, they are only applied to the sfConfig attribute.

• print – again, not really a phase, but it triggers the printing of the tree to the standard output. This provides a debugging mechanism as it can be placed between any other phases to view the intermediate state of the tree.

• function – in reality a user-defined phase, but one which is provided by default. It causes all the functions to be evaluated. It is triggered in the same way as the other user-defined phases, by the occurrence of attributes with the name phase.function.

• predicate – also a user-defined phase which is provided by default. It causes all predicates to be checked and errors reported. The schema mechanism is an instance of the use of the predicate phase, though others may be added by users. The phase is triggered in the same way as other user-defined phases, by the occurrence of the attributes with the name phase.predicate.

Phases are triggered in a specific order, as determined by the top-level attribute phaseList. If the attribute is not present, it is as though the attribute were defined as follows:

phaseList [“type”, “place”, “sfConfig”, “link”, “function”, “predicate”];

This default definition provides the semantics described in the section 1.

In addition to the pre-defined phases, a user may introduce their own. User phases are defined as follows:

• A class must be created which implements the interface PhaseAction in package org.smartfrog.sfcore.parser. The interface is fully defined in the Javadoc, but in summary, it provides two methods:

• forComponent – which initializes the instance of the action with the component description on which it is to operate

• doit – which triggers the action of the phase,

• In whichever component description the action must take place, an attribute whose name starts with the string phase.nnn must be provided, set to the string containing the class name, where nnn is the desired name of the phase.

• The phaseList attribute must be set at the top level of the description, containing the phase name nnn at the appropriate point relative to the other phases. It is recommended that this is placed after all the standard resolution phases, though occasionally it may be necessary to place the phase earlier.

There are a few points to notice. Firstly, the descriptions are traversed depth-first so the inner descriptions are visited before the outer. This makes sense for functions, for example, that are evaluated from the inside. The second point is that the action is independent of the phase, in that the attribute name determines the phase; the action is determined by the attribute value. Thus, it is possible for the same action to be used in two different phases, and for different actions to be invoked in the same phase – as is the case with all functions. It is also possible to have more than one action for each phase in a component description since the attribute name merely needs to start with the phase.nnn string so several may be provided.

Note that both the phaseList attribute and the phase.nnn attributes are removed from the description after the action is invoked.

Consider the following example. A class is provided that adds the sfProcessHost attribute (used to determine on which host a component should be deployed) to a component description, based on the value of an attribute sfLogicalHost. It maps the logical host to the physical host in some way not defined here – say by using the method mapHost.

The class might be defined as follows:

package org.smartfrog,example;

class MapHost implements PhaseAction {

ComponentDescription cmp = null;

public void forComponent (ComponentDescription c) {

cmp = c;

}

public void doit() {

String logicalHost = c.sfResolve(

Reference.fromString("sfLogicalHost"));

c.addAttribute("sfProcessHost", mapHost(logicalHost));

}

private String mapHost(String logical) { … }

}

This class may then be used in a description, to be acted on in the phase mapHosts, as follows

phaseList ["type", "place", "sfConfig", "link",

"function", "predicate", "mapHosts"];

MappedCompound extends Compound {

phase.mapHosts "org.smartfrog.example.MapHost";

}

sfConfig extends MappedCompound {

sfLogicalHost "databaseHost";

component1 extends Prim { …}

component2 extends Prim { … }

}

The phase list adds the mapping phase to the end, providing for the host mapping. The MappedCompound, when used, carries its phase attribute with it. Consequently, it is now contained within sfConfig. Thus during that last phase, sfConfig will be mapped to the correct physical host.

Functions

Functions are evaluated during a predefined phase, named function, with the effect that an attribute obtains the value of the evaluated function. To make functions easier to write, a predefined abstract PhaseAction, called BaseFunction from package

org.smartfrog.sfcore.languages.sf.functions

is provided that makes writing new functions easier.

New functions should extend the class BaseFunction and provide the method doFunction(), returning the result of the function as an Object. Any attribute may be accessed during the evaluation process.

BaseFunction is documented in the Javadoc and predefined functions are documented in section 14.

Predicates

Predicates are evaluated during a predefined phase, named predicate, with the effect that the associated predicate class is evaluated and any errors notified to the user by generating an appropriate exception. Most predicates will be instances of Schema, however users may define their own. To make user-defined predicates easier to define, a class BasePredicate from package

org.smartfrog.sfcore.languages.sf.predicates

is provided that makes writing new predicates easier.

New predicates should extend the class BasePredicate and provide the method doPredicate(), throwing the exception

SmartFrogCompileResolutionException

if there is an error. Any attribute may be accessed during the predicate evaluation.

BasePredicate is documented in the Javadoc and the predefined predicate Schema is documented in section 15.

Programming with the Parser

Background

The SmartFrog framework is designed to support a range of possible languages to define configurations for the deployment engine to instantiate. The languages are all required to follow a common model for their processing, and to eventually produce data structures that are suitable for the deployment system. The default language is the base SmartFrog language defined above, and which uses the file extension “.sf”.

The first stage of language processing is the parser – a tool for turning text into data structures for further processing. The parser interface allows programmers to select the parser based either on the language type of the file (as defined by file extension), by direct selection, or simply using the default (sf) parser.

After parsing, the data structures produced must implement an interface for driving the remaining resolution phases. This interface is

org.smartfrog.parser.Phases

Following the invocation of the various phases, the data is converted into a hierarchy of data supporting the ComponentDescription interface, which may then be passed to the deployment system.

Using this model, it is reasonably easy to define a new language and integrate it into the system. The default SF language is the first such, but others such as XML based languages, or the more advanced SF2 language currently under development are also possible.

The remainder of this section describes how to invoke the parser, how to step the language data structures through the various processing phases, and finally the nature of the resultant ComponentDescription data structures.

Summary of Language Processing

All of the tools provided with the SmartFrog system handle a SmartFrog text in an identical way to produce a fully resolved deployable description. The process is basically:

• parse the text stream to produce hierarchical data structures

• carry out all the phases, which for the default primary language are

  • type resolve the root

  • place resolve the root

  • extract attribute sfConfig from the root

  • link resolve sfConfig

  • evaluate any functions in sfConfig

  • check predicates and schemas in sfConfig

• convert to standard data model, creating simple normalised attribute tree

The Parser

The SmartFrog parser is implemented as a Java class with a method to parse an InputStream producing an instance of the class ComponentDescription, the Java class representing the parsed text allowing programmatic manipulation of the information. Any InputStream may be used, thus the parser may be invoked on a String, a File, a URL, or any indeed any object that provides a stream model.

During parsing, a number of include files or URLs may be specified indicating text that should be included into the current parse. It should be noted that unlike C, the text is not merely embedded into the source text, rather the files are parsed independently by the parser and the consequent data embedded into the resultant ComponentDescription data structure produced by the initial stream. Note that in principle, the parsers of include files may be different from the parser for the main stream, thus providing a means for including files in different notations. However, the mechanisms for doing so are not covered in this manual.

The Parser API

Under normal circumstances, users of SmartFrog will not be expected to use the parser directly. Rather the parser will be invoked on the users behalf by the tools and scripts provided to start and run the SmartFrog framework. However, just in case the need arises to invoke the parser within user code the parser API is now described.

Two aspects must be considered:

1. Ensuring that security properties are maintained: if security is required, the appropriate actions should be taken to ensure that only streams from signed and trusted sources are used.

2. Invoking the parser itself on the stream.

The security model is covered in section 8, and this should be read in detail before implementing any secure code, however enough of the security API is defined here for completeness.

Ensuring Security

Two important steps must be carried out to ensure that the security of the SmartFrog framework is not compromised. The first is to initialize the SmartFrog security infrastructure, if this is not already done, and the second is to ensure that every resource (test file, URL, etc) is loaded through the secure mechanisms provided.

Under normal circumstances, users will be using the parser from within the SmartFrog system itself; writing components that use the parser. However, just in case this is not so and security is still required, initializing the security mechanisms is carried out by invoking the initSecurity() static method on the SFSecurity class from package org.smartfrog.sfcore.security, as follows:

import org.smartfrog.sfcore.security.SFSecurity;

…

SFSecurity.initSecurity();

Once the security has been initialized, streams may be created on strings or files as required. However, to ensure that security is maintained, it is important that the correct class loaders are used for accessing any external resources. This is achieved by using the following invocation to create a stream from the resource:

import org.smartfrog.sfcore.security.SFClassLoader;

…

InputStream stream = SFClassLoader.getResourceAsStream(url);

Invoking The Parser

Once a stream is created, a parser instance may be created and the input stream parsed to generate the data model. This is done through the following code

import org.smartfrog.sfcore.parser.SFParser;

import org.smartfrog.sfcore.parser.SFPhases;

…

Phases component = new SFParser().sfParse(stream);

The getParser() method returns a parser for the currently selected language (currently only one is available) and this parser supports the sfParse(InputStream s) method to parse the input stream.

If the parser for a different language is required, say for the sf2 language, the following code is required

import org.smartfrog.sfcore.parser.SFParser;

import org.smartfrog.sfcore.parser.Phases;

…

Phases component = new SFParser("sf2").sfParse(stream);

The parser is built for the correct language, then asked to parse a stream.

The SFSystem command-line parameters that represent URLs of descriptions to load are examined to determine from the extension which parser should be used.

Evaluating The Phases

Once the parser has completed, the resultant data structures must implement the Phases interface. Through the use of this interface the various phases of the language processing are carried out – either as a single step or by carrying them out one at a time. After each phase, date structures that implement the Phases interface must be returned.

The complete description of the API is given in the Javadoc, but the following examples are probably sufficient to illustrate the process.

To evaluate all phases in one go:

Phases phases = new SFParser().sfParse(stream);

phases = phases.sfResolvePhases();

To extract the phases, then apply them one at a time:

Phases phases = new SFParser().sfParse(is);

Vector thePhases = phases.sfGetPhases();

for (Enumeration e = thePhases.elements(); e.hasMoreElements();) {

phases = phases.sfResolvePhase((String) e.nextElement());

}

Converting to ComponentDescription

ComponentDescription

Before handing the data to the deployment system, the languages own data structures must be converted to those expected by the deployment system – namely the standard data model implementing the ComponentDescription interface (normally, but not necessarily) an extension of ComponentDescriptionImpl.

This is done using the sfAsComponentDescription method defined in Phases. The full code for parsing and processing a stream in the default language is

Phases component = new SFParser().sfParse(stream);

phases = phases.sfResolvePhases();

ComponentDescription component.sfAsComponentDescription();

The Common Data Model

This section describes the data structures produced after the complete cycle of language processing (i.e. phase resolution and conversion to the core data model using sfAsComponentDescription). These are the structures that are understood and accepted by the SmartFrog run-time system.

The primary data structures that are generated as the output of this process implement the interface ComponentDescription, and it is this interface that users must understand to be able to create interesting tools or components. These data structures define the concept of an ordered attribute set.

In addition, the classes that are used to represent the various attrubute values need to be considered: both the basic values such as Integers and Booleans, and the references (all LAZY by this time).

In all cases, these interfaces and classes are fully defined in the accompanying Javadoc. The description provided here is only partial and is to give an overall feeling for the overall structure of the Java representation. The details of exceptions should also be obtained from the Javadoc.

Basic Values

Each of the basic values that have a syntax in the SmartFrog notation are mapped to different classes in Java. Wherever possible, they are mapped directly to the most obvious class in Java.

• Numbers are mapped to the equivalent Java subclass on java.lang.Number.

• Booleans are mapped to the class java.lang.Boolean.

• Strings are mapped to the class java.lang.String.

• NULL is mapped to the new SmartFrog class org.smartfrog.sfcore.common.SFNull. This is defined so that there is exactly one value of SFNull.

• Vectors are mapped to java.util.Vector.

• Byte arrays are mapped to the new SmartFrog class org.smartfrog.sfcore.common.SFByteArray. From instances of this class, the byte array (byte[]) can be obtained. Instances are immutable.

Values of other types can be contained within Component Descriptions, but these may not be properly handled by some of the operations in SmartFrog. In particular, although SmartFrog functions can in principle return values of any class, and these will be patched into the attribute tree, when the conversion to the final component description form occurs with the sfAsComponentDescription method, these will be rejected. (This is because the soon-to-be-introduced description signing will not know how to handle these arbitrary values properly.) Values of arbitrary types can be serialized into byte arrays, and then extracted and deserialized at the appropriate time.

Reference

Reference is the Java representation of the references that may be used in the three areas: references to super-types, placement references and as links. References are lists of ReferencePart, each indicating a single step in the resolution that must occur.

The only interesting methods are those for constructing references, namely the constructors and the method fromString. The other methods are typical of those required for manipulating lists, such as adding and removing parts and enumerating over the elements.

There are two constructors – one for an empty reference (one with no parts) and one for constructing a reference with a single part. Other parts must either be added to these basic references, or the reference may be created by parsing a string.

static Reference fromString (String refString)

Utility method to create a reference from a string. The method is extremely expensive as it creates an instance of a parser and should therefore not be used too freely.

ReferencePart is the parent class of all reference parts, there is one per syntactic reference part (ROOT, ATTRIB, etc.). Again, the main interest is in the constructors for these. There are a couple of static helper methods for their construction defined in ReferencePart.

ComponentDescription

A ComponentDescription is an interface, with default implementation ComponentDescriptionImpl, which represents the concept of an attribute set in the syntax. Consequently, it has a number of methods that enable the creation and traversal of the containment and extension hierarchies.

ComponentDescription, in addition to defining its own methods, extends three further interfaces, two of which needs further description: ReferenceResolver and Copying. The third, ComponentDeployer, defines methods that are used internally by the SmartFrog framework.

A base implementation of the interface ComponentDescription is the class ComponentDescriptionImpl is provided by the framework. This class may be generated directly by the language processor, or users may produce a class which extends it in some way.

The interface can be considered in three parts:

1. the core interface for construction and traversal.

2. a copying interface which provides a deep copy operator essential when handling descriptions.

3. a reference resolution interface, defining methods to look up attribute values given references that describe paths through a description hierarchy.

Core

TO BE DONE

cover add/remove/replace attribute, iterate over attributes, get the parent.

Copying

The interface defines two methods of note – a deep copy operator that returns an equivalent structure of data and a clone method that returns a shallow copy. The copy method is recursive, in that it clones the top level component description, then embeds within it all the data contained in the copied description - invoking the copy method first if this data implements the Copying interface.

public Object copy();

Produce a deep copy of the component description

public Object clone();

Produce a shallow copy of the component description

ReferenceResolution

The reference resolution interface contains a number of methods to locate attributes within the hierarchy of component descriptions. The main method provided is the following:

Object sfResolve(Reference r)

Resolve a given reference in the ComponentDescription hierarchy starting from this component.

In addition to this method, there is a whole family of variants, such as methods which take strings rather then references, or define the specific return type so that users can avoid the class-caste, and so on. These are fully documented in the Javadoc.

The SmartFrog Grammar Rules

SmartFrog defines the default language's grammar using the Java Compiler Compiler system from Sun. This is a tool known as JavaCC. The SmartFrog grammar rules described here are part of the JavaCC input, the file DefaultParser.jj, which is available in the source distribution. The listing is derived from this file.

/*

* Main entry point to the grammar

*/

AttributesEntry ::= AttributeList EOF

/*

* Entry point for Reference parser (as used in sfResolveFromParser)

*/

ReferenceEntry ::= Reference EOF

/*

* Entry point for any value parser (used in tools such as sfGui)

*/

AnyValueEntry ::= Value EOF

/*

* Entry point for primitive value parser (used in tools such as sfGui)

*/

PrimitivevalueEntry ::= Basic EOF

/*

* Main body of grammar

*/

AttributeList::= ( Attribute

| #include String

| ;

)*

Attribute::= Name Value

Name::= -- | (WORD [ : Name ])

Value::= Component

| SimpleValue ;

| ;

SimpleValue::= Reference

| Basic

| Operator

| IfThenElse

| Vector

Operator::=

(

(UnaryOp SimpleValue )

| (SimpleValue [ BinaryOp SimpleValue ])

| (SimpleValue [ (NaryOp SimpleValue)* ])

)

IfThenElse::= IF SimpleValue THEN SimpleValue ELSE SimpleValue FI

Vector::= [ [SimpleValue ( , SimpleValue) *] ]

Component::= extends [LAZY] Type Body

Type::= [ NULL | BaseReference ]

Body ::= ( ; | { AttributeList } )

Reference::= [ LAZY] BaseReference

BaseReference::= ReferencePart ( : ReferencePart)*

ReferencePart::= ROOT

| PARENT

| (ATTRIB WORD)

| (HERE WORD)

| THIS

| WORD

| (PROPERTY WORD)

| (IPROPERTY WORD)

| (HOST (WORD | STRING))

| PROCESS

| WORD

Basic::= String

| Number

| Boolean

| ByteArray

| [| Basic (, Basic)* |]

| NULL

Number::= DOUBLE

| FLOAT

| INTEGER

| LONG

String::= STRING

| MULTILINESTRING

Boolean::= true | false

ByteArray::= Hex|Dec|Oct|Bin|B64

// only Hex currently implemented

UnaryOp::= !

BinaryOp::= - | / | == | != | >= | > | <= | <

NaryOp::= + | * | ++ | <> | && | ||

The SmartFrog Lexical Rules

The SmartFrog lexical rules described here are part of the JavaCC input, the file DefaultParser.jj, which is available in the source distribution. The listing here is a slight simplification of this file.

/* White Space */

SKIP : " "| "\t"| "\n"| "\r"| "\f"

/* Comments */

SINGLELINECOMMENT: "//"(~["\n”,”\r"])*

FORMALCOMMENT: "/**" ... "*/ "

MULTILINECOMMENT: “/*" ... ”*/"

/* Reserved Tokens */

RESERVED: ";" | "," | "{" | "}" | "[" | "]" | ":" |

"true" | "false" | "NULL" | "--" |

"extends" | "LAZY" | "ROOT" | "ATTRIB" | "HERE" |

"THIS” | "PROPERTY" | "IPROPERTY" | "PARENT" |

"HOST" | "PROCESS" | "#include" | "(" | ")" |

"==" | "!=" | ">=" | ">" | "<=" | "<" | "<>" | "!" |

"+" | "-" | "*" | "/" | "++" | "&&" | "||" |

"IF" | "THEN" | "ELSE" | "FI"

/* Tokens – using Unicode */

WORD: LETTER (LETTER|DIGIT|SPECIAL)*

SPECIAL: [".", "_", "-", "+", "@", "#", "~", "$", "%", "^", "&"]

LETTER:

[

"\u0024",

"\u0041"-"\u005a",

"\u005f",

"\u0061"-"\u007a",

"\u00c0"-"\u00d6",

"\u00d8"-"\u00f6",

"\u00f8"-"\u00ff",

"\u0100"-"\u1fff",

"\u3040"-"\u318f",

"\u3300"-"\u337f",

"\u3400"-"\u3d2d",

"\u4e00"-"\u9fff",

"\uf900"-"\ufaff"

]

DIGIT:

[

"\u0030"-"\u0039",

"\u0660"-"\u0669",

"\u06f0"-"\u06f9",

"\u0966"-"\u096f",

"\u09e6"-"\u09ef",

"\u0a66"-"\u0a6f",

"\u0ae6"-"\u0aef",

"\u0b66"-"\u0b6f",

"\u0be7"-"\u0bef",

"\u0c66"-"\u0c6f",

"\u0ce6"-"\u0cef",

"\u0d66"-"\u0d6f",

"\u0e50"-"\u0e59",

"\u0ed0"-"\u0ed9",

"\u1040"-"\u1049"

]

/* Literals */

STRING: ("\"" (

(~["\"","\\","\n","\r"])

| ("\\"

( ["n","t","b","r","f","\\","'","\""]

| ["0"-"3"] ["0"-"7"] ["0"-"7"]

)

)

)* "\"")

MULTILINESTRING: ("##" (

(~["#","\\"])

| ("\\"

( ["n","t","b","r","f","\\","'","#"]

| ["0"-"3"] ["0"-"7"] ["0"-"7"]

)

)

)* "#")

NUMBER: <INTEGER> | <FLOAT> | <LONG> | <DOUBLE>

INTEGER: (("-")? ["1"-"9"] (["0"-"9"])*) | "0"

FLOAT_BASE: ("-")?

(

(["0"-"9"])+ "." (["0"-"9"])* (<EXPONENT>)?

| "." (["0"-"9"])+ (<EXPONENT>)?

| (["0"-"9"])+ <EXPONENT>

| (["0"-"9"])+ (<EXPONENT>)?

)

EXPONENT: ["e","E"] (["+","-"])? (["0"-"9"])+ >

DOUBLE: <FLOAT_BASE> (["d", "D"])?

LONG: <INTEGER> (["l", "L"])?

FLOAT: <FLOAT_BASE> ["f", "F"]

Hex: "#HEX#" (["A"-"F", "a-f", "0"-"9", "\n", "\t", " "])* "#"

Oct: "#OCT#" (["0"-"7", "\n", "\t", " "])* "#"

Bin: "#BIN#" (["0"-"1", "\n", "\t", " "])* "#"

Dec: "#DEC#" (["0"-"9", "\n", "\t", " "])* "#"

B64: "#B64#" (["A"-"Z", "a-z","0"-"9", "/", "+", "\n", "\t", " "])* "#"

Predefined SmartFrog Functions

SmartFrog provides a number of functions. These functions are all available as templates that are defined in a file which must be included if they are to be used. However, some are also available as operators, using the SmartFrog operator syntax, and in this case the include file is not required.

The operators are all converted into an instance of the expanded template at time of parsing, so may in every respect be treated in the same way as a use of the template itself. Furthermore, it should be noted that any references that are used within an expression containing operators, these references will be resolved in the context of the templates – this means that use of reference parts such as PARENT are hard to use. ATTRIB reference parts are useable in the normal way.

Note also that since attribute names may contain many of the operator symbols, it is best to always surround the operators with space characters to ensure that they do not accidentally “stick” to the names.

The functions are defined by including the functions.sf file as follows:

#include "org/smartfrog/functions.sf"

The functions defined as operators may be grouped into three main categories: unary, binary and nary.

Unary Operators

There is currently only one unary operator, the Boolean negation operator. The syntax for unary operators is

( opsymbol value )

The surrounding ( ) symbols must be present. All templates for unary operators have as their parameter the attribute “data”. Other attributes are allowed, but are ignored for the purpose of evaluating the function. They may, of course, be used for the definition of the data attribute during earlier phases.

not

Operator symbol: !

The function not is defined as the negation of the boolean attribute “data”. If the attribute is not present or of the wrong type, an exception is reported.

x true;

foo (! x);

bar extends not {

data x;

}

Binary Operators

There are a number of binary operators covering primarily the arithmetic, comparison and logical operators. The syntax for binary operators is

( value opsymbol value )

The surrounding ( ) symbols must be present. All templates for binary operators have as their parameter attributes the names “left” and “right”, to indicate which value it is. Other attributes may be present and are ignored whilst evaluating the operator.

Minus

Operator symbol: -

The minus operator subtracts the right attribute from the left, resulting in a number which satisfies the Java rules for numbers. If either of the two attributes are not numbers, an exception is thrown. Other attributes that may be defined in the template are ignored.

minus10 extends minus {

value;

left value;

right 10;

}

foo extends minus10 {

value 34;

}

aFoo ( 34 – 10 );

divide

Operator symbol: /

The divide operator divides the left attribute by the right, resulting in a number which satisfies the Java rules for numbers. If either of the two attributes are not numbers, an exception is thrown. Other attributes that may be defined in the template are ignored.

percent extends product {

fraction extends divide {

enum; denom;

left enum;

right denom;

}

-- 100;

}

foo extends percent {

fraction:enum 34;

fraction:denom 56;

}

EQ, NE

Operator symbols: ==, !=

These operators are the comparator operators, equals and not equals respectively. The two attributes, left and right, are compared using the Java equals method (left.equals(right)). The result of the function is the boolean value that is returned by that test.

GE, GT, LE, LT

Operator symbols: >=, >, <=, <

These operators are the numeric value comparators, testing to see if the left attribute value is greater than or equal to (or whatever operator is used) the right attribute. The Java rules for numeric comparison are used.

Nary Operators

Nary operators are operators that are may have a arbitrary number of attribute parameters. All the attributes provided within the template are assumed to be part of the function, and the names used to provide these attributes are ignored. Thus the “unique” name – is normally used for these operators.

The syntax for an nary operator is as follows:

( value opsymbol value opsymbol value opsymbol ... )

Each of the operator symbols must be identical, though other may be used by nesting the use of operators wherever a value is expected. The above form is converted to the expanded template form during parsing, so any references that are used when a value is expected is resolved relative to the template and not the operator expression.

concat

Operator symbol: ++

The concatenate function takes each of its attribute parameters and concatenates them in the order of definition. These attributes are converted to strings using the toString() Java method. An example of the use of the concatenate function is:

myString extends concat {

a "the meaning of life is ";

b 42;

c extends concat {

a " by ";

b "Douglas Adams";

}

}

which results in the string “the meaning of life is 42 by Douglas Adams”.

append

Operator symbol: <>

The append function is similar to the vector function, except that all parameters must be vectors and these are expanded in-line. The difference can be seen by considering the same example

myVector extends vector {

-- ["the meaning of life is "];

-- [42];

-- extends vector {

-- " by ";

-- "Douglas Adams";

}

}

which results in the vector

[ "the meaning of life is", 42, "by", "Douglas Adams"]

The operator form can be used for the same purpose. The following definition is equivalent to the definition of myVector above.

MyVector ( ["the meaning of life is"]

<> [42]

<> ["by", "Douglas Adams"] );

sum

Operator symbol: +

The sum function sums each of its attributes which must be numbers, failure will result in an exception. The names of the attributes are, of course, irrelevant. An example of the use of the sum function is:

val1 34;

val2 45;

num extends sum {

-- val1;

-- 345;

-- val2

}

This will result in num being set to 424. An equivalent expression is

num (val1 + 345 + val2);

product

Operator symbol: *

The product function multiplies each of its attributes type-cast to integers, failure will result in an exception. The order is irrelevant. An example of the use of the product function is:

times10 extends product {

ten 10;

}

myNum extends times10 {

val 34;

}

This will result in myNum being set to 340.

and

Operator symbol: &&

The conjunction operator takes the logical “and” of all its attribute parameters. Each of the parameters must be a boolean and if they are not, an exception is thrown. As with all nary operators the names of the attributes are irrelevant.

or

Operator symbol: ||

The disjunction operator takes the logical “or” of all its attribute parameters. Each of the parameters must be a boolean and if they are not, an exception is thrown. As with all nary operators the names of the attributes are irrelevant.

Other Functions

In addition to the operators, there are a number of other functions provided. Some of these also have specialised syntactic forms – most notably the ifThenElse and the vector function. These syntactic forms are converted to the template form at time of parsing, and do not require the use of the include file “functions.sf”. If the templates are used directly, then this file must be included.

IfThenElse

The ifThenElse function is provided to conditionally provide a value for an attribute. The template uses three well-known attributes: if, then and else. Other attributes may be present but are ignored. If one of the attributes is not provided, or is the if attribute is not a boolean value, an exception will be thrown. Since the ifThenElse template is defined through attributes, these may be changed by extension or placement.

aStepFunction extends ifThenElse {

boundary 10;

value;

if (value > boundary);

then 20;

else 30;

}

This may now be used as follows:

aValue extends aStepFunciton {

value 5;

}

anotherValue extends aStepFunction {

value 15;

}

Which would give values for these two attributes as 20 and 30 respectively.

The special syntactic form for ifThenElse is as follows:

foo IF (10 > 20) THEN 5 ELSE 15 FI;

defining foo to be 15. Note that as with all special forms, any references used within it are evaluated relative to the transformed syntax, and not that given. The definition for foo above is equivalent to:

foo extends ifThenElse {

if (10>20);

then 5;

else 15;

}

vector

The vector function takes each of its attribute parameters in the order provided and creates a vector whose elements are the values of the attributes (names are ignored, hence unique naming is useful). An example is

myData extends vector {

-- "the meaning of life is ";

-- 42;

-- extends vector {

-- " by ";

-- "Douglas Adams";

}

}

which results in the vector with three elements, the third of which is itself a vector.

There is a special syntactic form for vector which is as follows:

myData ["the meaning of life is", "42", ["by", "Douglas Adams"]];

This is the equivalent of the earlier definition using the template. Again, any reference provided within the vector syntax is resolved relative to the transformation into the template form.

formatString

FormatString is a function that takes a format string and a set of parameters and creates a resultant string which has the values of the parameters embedded. The format string attribute itself should be named format and the various parameter strings should be named sx where x is a single digit. The format string should identify the places where the various parameter strings should be embedded using the characters “$x” for a single digit x. An example is

myString extends formatString {

format "the meaning of $2 is $1";

s1 42;

s2 "life";

}

The attributes may of course be links to other values, but not LAZY links as these are not resolved in time for the function phase.

random

The random function, which in truth is not really a function since it returns a different value for each invocation, returns a random number as follows:

• if the attribute integer is set to true, an integer between attributes min and max is returned, otherwise a floating point value between 0 and 1. The default values for min and max are 0 and 10 respectively.

• if the attribute seed is provided, and the random number generator has not yet been initialized, that seed is used.

Examples of the use of the random are:

dice extends random {

integer true;

min 1;

max 6;

}

myConfig extends Compound {

throw1 extends dice;

throw2 extends dice;

}

Each of throw1 and throw2 will be some random integer between 1 and 6. Note that each invocation in myConfig is independent. Each JVM contains a single random number generator for use during function resolution.

next

The next function is one that returns a monotonically increasing value, guaranteed never to return the same number twice within a single description. Again, it is not strictly a function since it never returns the same value for the same parameters. The only parameter attribute is the base attribute, setting a minimum value for the values. If the base is below the next value, it is ignored. If it is above, the next value will be the base. The default base is 0.

An example of the use of next is

unique extends concat {

prefix "xxyyyqqq";

postfix extends next;

}

myConfig extends Compound {

name extends unique;

otherAttr 42;

}

ref

The ref function converts a string to a reference, and then optionally resolves the reference in place or leaves it as a LAZY link. This allows links to be created by using functions over strings, such as concatenation, to generate a reference. The attributes for the ref function are reference which is the string to convert, and the optional attribute lazy which defaults to false and controls whether to leave the reference as a lazy link, or to resolve to replace it with the value obtained.

An example of the use of ref is:

x "ROOT";

y "a";

a 24;

myConfig extends ref {

reference (x ++ y);

lazy false; // default

}

This results in myConfig being set to 24.

date

The date function returns a string representation of the current date. There are no formatting parameters. Again, this is not strictly a function.

userinput

The userinput function asks the user for an input on the command line. It returns the value entered. The prompting message may be specified in the prompt attribute.

anything extends userinput {

prompt "Enter any value";

}

This will result in anything being set to whatever has been entered on the command line. A default value may also be set using the attribute “default”. This function is not really meant for serious use, but more for experimentation and testing.

Schemas

Schemas are descriptions that may be attached to other descriptions and cause them to be checked against the schema description. Schemas are evaluated as part of the standard predicate phase.

Schemas are defined by extending the predefined template Schema, defined in the file /org/smartfrog/predicates.sf:

mySchema extends Schema {

// schema entries

}

Each of the schema entries are attributes whose names are to be found in the template to be validated. Each of these entries must extend a description that defines certain properties about the attribute. The properties are

• optional: a Boolean that states whether the attribute is optional or compulsory

• binding: a string which defines whether the attribute must be lazy "lazy"), must be eager ("eager"), or may be either (“anyBinding”) – this controls whether a link may exist instead of a value of the correct class

• class: a string which defines the name of the class which should be found as the value of the attribute (e.g. “java.lang.Integer”), or any class (“anyClass”).

Thus entries in a schema for a web server component may be

WebServerSchema extends Schema {

port extends {

optional false;

binding "anyBinding";

class "java.lang.Integer"; }

directory extends {

optional true;

binding "anyBinding";

class "java.lang.String"; }

}

However this is rather cumbersome, so some helper templates are defined in the include file. These are defined as follows, with the obvious meanings.

Compulsory extends {

optional false;

binding "anyBinding";

class "anyClass";

}

Optional extends {

optional true;

binding "anyBinding";

class "anyClass";

}

OptionalBoolean extends Optional {

class "java.lang.Boolean";

}

Boolean extends Compulsory {

class "java.lang.Boolean";

}

OptionalInteger extends Optional {

class "java.lang.Integer";

}

Integer extends Compulsory {

class "java.lang.Integer";

}

OptionalDouble extends Optional {

class "java.lang.Double";

}

Double extends Compulsory {

class "java.lang.Double";

}

OptionalLong extends Optional {

class "java.lang.Long";

}

Long extends Compulsory {

class "java.lang.Long";

}

OptionalFloat extends Optional {

class "java.lang.Float";

}

Float extends Compulsory {

class "java.lang.Float";

}

OptionalString extends Optional {

class "java.lang.String";

}

String extends Compulsory {

class "java.lang.String";

}

OptionalVector extends Optional {

class "java.lang.Vector";

}

Vector extends Compulsory {

class "java.lang.Vector";

}

OptionalReference extends Optional {

class "org.smartfrog.sfcore.reference.Reference";

}

Reference extends Compulsory {

class "org.smartfrog.sfcore.reference.Reference";

}

OptionalCD extends Optional {

class "org.smartfrog.sfcore.componentdescription.ComponentDescription";

}

CD extends Compulsory {

class "org.smartfrog.sfcore.componentdescription.ComponentDescription";

}

These templates allow for a neater definition of the schema given above;

WebServerSchema extends Schema {

port extends Integer;

directory extends OptionalString;

}

To attach a schema to a description, the schema need only be an attribute within the description to which it applies. Thus we can complete the above example as follows:

// the definition of schemas

#include "/org/smartfrog/predicates.sf"

WebServerSchema extends Schema {

port extends Integer;

directory extends OptionalString;

}

WebServerTemplate extends Prim {

schema extends WebServerSchema;

// default value

port 80;

}

Note that the name for the attribute linking the template to its schema need not be, as in this case, schema. Indeed, a template may have more than one schema attached as attributes, in which case the uses of the template are checked against all schemas attached.

Schemas may be extended in the same way as other templates, and their uses may easily be extended through placement as illustrated in the following examples.

#include "/org/smartfrog/predicates.sf" // the definition of schemas

ThreadedWebServerSchema extends WebServerSchema {

minimumThreads extends Integer;

}

ThreadedWebServerTemplate extends WebServerTemplate {

//overwrite with extended schema

Schema extends ThreadedWebServerSchema;

minimumThreads 7;

}

AlternativeThreadedWebServerTemplate extends WebServerTemplate {

// add to existing schema

schema:minimumThreads extends Integer;

minimumThreads 7;

}

Note that schemas are entirely optional and need be used only if desired. They carry no overhead during deployment and at run-time, but they can be expensive at language processing time.

Prim and Compound templates both have schemas associated with them. This can be useful to locate errors, but they also have two other effects. Firstly, when the descriptions are printed in any expanded form the schemas occupy a rather large amount of the overall description and can hide the structure of a description. Secondly, they can make a large description very expensive to process. Consequently, an attribute can be set at the top level to control whether these schemas should be included. Thus the following would switch off the use of the Prim and Compound schemas:

sfSchema false;

Comparing the SmartFrog Notation with XML

In this appendix, a basic and partial comparison is provided between the SmartFrog notation and XML. This is intended to explain the rationale behind the design choices made in the definition of the SmartFrog notation, and to provide an explanation as to why XML has not been substituted for the SmartFrog notation (it also happens to predate XML by a couple of years). It should be noted, however, that XML can be considered as a syntactic framework within which the SmartFrog language semantics could be encoded. This is not considered below, and only a direct comparison of the core notions is made.

The main areas examined in detail are:

1. Naming: how attributes are identified and referenced

2. Composition: how attributes are composed into hierarchies or collections

3. Inheritance: how collections of attributes are built from other collections

4. Schemas, typing and validation: how collections are checked for correctness, and the correctness constraints defined

5. Templates and parameterization: how the notations provide a degree of end-user customization of given configuration templates

These are not the only differences, but they are some of the major ones, and have a particular importance when it comes to examining the requirements for a rationally designed configuration notation.

Naming and Composition

Perhaps the most striking difference between the two notations is that in SmartFrog, every attribute is of the same kind (there is no distinction between attributes and elements), and every attribute is named. In XML, there are named attributes (given as an addition to the tag) and elements that are un-named in the parent element.

The tag itself cannot be considered a name for two reasons. First, it is not required to be unique within the parent element, and secondly it is a property of the tagged element – the infoset model supported by XML has the tag belonging to the tagged element rather than merely the name it has in the parent. If the element were copied elsewhere, it would maintain the same tag.

XML has three types of data that might be considered “attributes” (ignoring comments and processing instructions which are now obsolete). These are:

• elements – a grouping and composition notion delineated by tags, in which the order in the parent is important;

• attributes – named values associated with an element;

• character data - strings of text associated with an element, unnamed but ordered.

Not providing naming has one severe consequence – attributes cannot be uniquely identified within a structure. This is can be a real problem, though naming can be replaced by a somewhat heavy query mechanism returning a (possibly empty) set of elements. This is the Xpath standard in XML.

SmartFrog, on the other hand, has but one notion – that of attributes. These are values that may be named within an attribute collection (known as a ComponentDescription), which is in turn an attribute. This provides a clean model but perhaps without the richness (or complexity) of XML – however this simplicity is a great advantage when describing and explaining the notation, and the extra notions present in XML are of uncertain benefit.

To represent the notion of an element tag, SmartFrog would need to encode that somehow into the attribute structure, perhaps by the inclusion of an extra attribute in a component description to indicate a tag – for example sfTag. This, plus any schemas, predicates and default attribute values associated with that tag, could be encoded as a template to be inherited and used as required.

Inheritance

Inheritance in this context is the ability to take a description of a configuration (as a collection of attributes) and extend or redefine some of these to provide a modified configuration. Its role is identical to inheritance in an object-oriented language. It is perhaps not a strictly necessary feature in a world where we have editors, cut and paste operations, and so on – but experience with OO languages is that without this capability the results are unmaintainable

SmartFrog provides a model of single inheritance, where one definition may extend another. This use of single inheritance solves the issue of what happens when two attributes with the same name are inherited through two paths in the inheritance hierarchy. However, the model can easily be extended to provide multiple inheritance if a specific model of attribute precedence is provided or some name disambiguation scheme is used. The only other form of combinator provided in SmartFrog is aggregation, where an attribute set may be a named attribute within another attribute set.

Unfortunately, XML does not provide any form of inheritance. It does provide a model of aggregation, where an existing document can be embedded into a new document, but it becomes a single element within that document.

Functions and Phases vs. XSLT

SmartFrog provides a small selection of functions and operators that can be used within the definition of a description, and a set of user creatable phases for manipulating the description. XML does not have this notion in quite the same way. Instead it has a transformational framework, XSLT, which allows users to provide very general transform rules over the description, sufficiently general to even encode the notion of extends. Although powerful and expressive, there are two issues with this. The first is that it can be rather hard for end users to understand the consequences of using a transform – they are typically not robust to misuse – and secondly the syntax can be rather obscure.

Schemas, Typing and Validation

SmartFrog is untyped – the attributes may be any value representable within the notation, currently various numeric types, strings and attribute sets. Under inheritance, the type may be changed to another. This has not proved to be a significant problem, however statically typed descriptions do have distinct advantages in certain areas. Finally, there is no way of specifying which attributes should, or should not, be present in a configuration description. SmartFrog takes a different approach to the problem. There is an optional way of describing predicates and attaching them as attributes to a configuration description. Being attributes, they are inherited in the same way as other attributes are inherited, and hence apply to the new description (unless explicitly overridden). The predicates may test the existence or type of any attribute, and may potentially execute arbitrarily complex tests over single or any combination of attributes taken from the collection of attributes as a whole. This provides a very powerful mechanism for defining the correctness of descriptions, without the need for a parallel schema mechanism.

XML documents are basically untyped, or more specifically they only contain the single basic type “character data” on top of the basic notion of element. To extend this, a schema mechanism is provided which allows users to specify which attributes must be present and how the character data should be interpreted. In addition a small set of simple predicates are provided for defining such things as sub-ranges of numbers.

XML schemas do provide inheritance in that one schema may be an extension of another, however this is unmatched in the document space.

Templates

There is a clear requirement to provide templates for configurations that may be parameterized. We should be able to define a configuration that defines:

1. A set of “parameter” attributes which must be provided to complete a configuration; this set is itself a configuration.

2. A configuration template which, when instantiated with this set of attributes, produces a complete configuration.

3. It must be possible to constrain this set of parameter attributes in a way that guarantees a valid configuration as the result of the instantiation.

Put simply, the ability to write functions is required, producing a configuration from a configuration.

XML provides no support for the notion of template. The very fixed notion of a document, without inheritance, limits the ability to even provide an approximation.

SmartFrog implicitly supports templates through a combination of capabilities – inheritance and linking. A requirement for an attribute can be expressed by providing an un-resolved link to a top-level attribute that may then be instantiated via inheritance. Although in many ways a satisfactory solution, it does not provide clear expression of the requirements for instantiation, as the parameter attributes are never clearly identified. However, even in the rather limited form provided in SmartFrog, the benefits of providing a template notion is clear.

Introduction

Components

Having written the description of a service in a notation, and having had this processed by the appropriate language processor (done in most instances by the SmartFrog command-line tools), the simplified models are passed to SmartFrog to carry out the deployment process. This process consists of the hierarchical deployment of components as defined by the description. At this point, ignoring how the description defines which components should run where, it is worth defining the very notion of a component.

A component is an instance of a Java class that implements a specific Java interface: the Prim interface. The classes for components are normally implemented by extending one of the predefined classes that already provide most of the required semantics, adding to or modifying this behaviour as appropriate. This interface is quite complex and extensive, however most of it is for SmartFrog's internal operation. Programmers really only need to use a subset of these.

The most important aspect, and that which is core to the notion of a component, is that of the lifecycle and the methods associated with it. A component has a simple lifecycle to bring it into being, to start its activity, and eventually to terminate it. This lifecycle is implemented as a set of methods provided by the programmer (or through inheritance, by one of the classes provided as part of SmartFrog).

As illustrated by the diagram above, the lifecycle of a component is generally thought of as the phased start-up of first component creation, then component initialization (also know as deployment) and finally component initiation: starting the active parts of the component. Termination and clean-up can happen asynchronously at any time after the initial creation, triggered internally by the component, externally by the environment, for example in response to some form of failure.

In addition to considering a single component and its lifecycle, it is necessary to consider the lifecycles of collections of components. These collections might define the set of components in a specific application or service. As such, the sum of the lifecycles of these components effectively determine the lifecycle of the service itself. As an example, consider a service that manages web servers and creates or terminates web servers according to loading . Any single component has a simple start/stop lifecycle as determined by the lifecycle model given above, but the collection has a much more complex lifecycle involving the dynamic modification of the collection of web server components that are included in the service – sometimes adding and sometimes removing these web server components.

Now the lifecycle of services, or more generally the service of some collection of components, are mediated by other components – known as compound components – that make use of SmartFrog's APIs to interact with the SmartFrog framework to instantiate and terminate the components for which it is responsible.

Most such compound components are very specific to a service – for example the component that monitors the response-time of the web servers to decide whether to deploy or terminate a web server is a specific component to that service. However there are some generic types of collection that can also be defined and which can usefully be implemented as part of a core framework such as SmartFrog.

The most important such specific collection is the Compound (as opposed to the more general concept of a compound component). The Compound component is one where every member of the collection shares fate with the others in the group. Thus they are all created or none are, they are all initialized or none are, they all start together or none start, and they all terminate or fail together.

An example of where this Compound behaviour is appropriate might be where the collection consists of the component to collect service response data and the component that uses this data to decide how many web servers to be running. It would not be appropriate to tie the different web servers together in such a group so that terminating one web server would terminate them all.

All of these group behaviours are mediated by components that can be defined and then made available for use by other services wherever this is appropriate. In addition to the pre-defined compound components, such as Compound, users may define their own specific to their service.

Defining Components in the Language

The SmartFrog Component Model

This section describes how a set of attributes is interpreted by the SmartFrog framework as a collection of components distributed across a number of hosts in a network. There are several aspects to consider:

• how attributes indicate type of component.

• how attributes are used to indicate the location of components.

• how the system carries out the creation of components.

• how components are started and stopped.

• how components may use the framework APIs to access configuration values.

• how components may dynamically interact with the framework to dynamically modify the sets of components that are created.

First, however, it is worth a general discussion of the concepts behind the SmartFrog application model.

Applications As Component Collections

SmartFrog considers a whole system to be a collection of applications running over a distributed collection of compute resources. This collection of applications may be dynamic, generated on demand by a variety of external and internal events, such as a user request or a new resource being started.

Each application is, in turn, a collection of components defined statically via an application description or generated dynamically at run-time according to the requirements determined at that time. The components of an application may be dynamic, changing over time to adjust for circumstances.

These terms, namely system, application and component, deserve better definition to highlight their respective roles and the ways in which SmartFrog manages them. It is easier to consider them in reverse order.

Component

A component is defined as a single Java object which implements a specific API (defined in the Prim interface) and which consequently implements the specific lifecycle as defined by the SmartFrog component model.

Since the component is implemented as a single object, it resides entirely within one JVM on a single host. The component may, behind the scenes, create and manage other objects including other processes and programs written in other languages. However, for the purpose of SmartFrog, the management view of component is entirely defined by the Prim object.

Application

An application consists of a collection of components, and consequently it is not an atomic object as seen by SmartFrog. This means that the lifecycle is not viewed through a single interface, complicating the handling of the lifecycle of the whole application. An application has two characteristics that characterize the notion:

• Each component is tightly bound to the others via a parent-child relationship, each may be a parent to others and each (unless it is the root component) has a parent. This transitive closure of the parent-child relationship is the scope of the application.

• The lifecycle of each component in an application is tied to the lifecycle of the others via this parent-child relationship – parents are notified of child death, and vice-versa, and the parents are entirely responsible for the lifecycle of their children. The order of component start-up and termination is well defined and may be relied on for the simplification of component coding.

Components of an application may locate each other, thereby enabling communication, using the built-in SmartFrog naming capabilities by following the parent-child relationships. These links between components are specified via LAZY links.

System

A system is simply a collection of applications, loosely grouped over the distributed resources. Applications within a system do not have direct links between their components, nor any direct responsibility for, nor notification of, their respective lifecycle (though these may be implemented within specific component behaviour).

Typically, applications locate each other through naming or discovery services, and they must be able to cope appropriately with the non-existence of applications on which they depend, both at start-up and during operation.

Applications and Component Descriptions

As described above, each application is a collection of components connected via a parent-child hierarchy. Thus an application is a tree structure similar to the tree structure present in a component description hierarchy, as described in section 11. Indeed, it is the role of the SmartFrog system to convert a description into the equivalent running application according to an appropriate interpretation of attributes present in the description.

The process of taking a description and converting it into a running application is defined roughly as follows:

• A description in SmartFrog notation is parsed and resolved.

• The specific interesting component description is selected, namely the sfConfig component description.

• The result is a hierarchy of component descriptions (each extending NULL, the empty description).

• The role of SmartFrog is to create an equivalent hierarchy of objects, as determined by the attributes present in the component description hierarchy.

Consider the following small example:

StatusMonitor extends Prim {

sfClass "org.smartfrog.examples.StatusMonitor";

}

LogMonitor extends Prim {

sfClass "org.smartfrog.examples.LogMonitor";

}

sfConfig extends Compound {

log extends LogMonitor;

status extends StatusMonitor;

}

Prim and Compound are described in detail in section 2.3.4, but in effect define collections of attributes that describe leaves (Prim) and nodes (Compound) of a component hierarchy. Prim is a collection that defines some basic attributes relevant to all components (such as default code base), and Compound contains at least the additional attribute:

sfClass "org.smartfrog.sfcore.compound.CompoundImpl";

After all the resolution steps, and after selection of the sfConfig attribute, the following description is obtained:

sfConfig extends {

sfClass "org.smartfrog.sfcore.compound.CompoundImpl";

// plus Prim attributes

log extends {

sfClass "org.smartfrog.examples.LogMonitor";

// plus Prim attributes

}

status extends {

sfClass "org.smartfrog.examples.StatusMonitor";

// plus Prim attributes

}

}

This is a hierarchy of component descriptions. The SmartFrog framework takes this description and turns it into the following running application: a component parent-child hierarchy using the sfClass attribute to determine the class of object to instantiate.

In both cases, the described and running, a Context (section 11) is used to store the attributes. In the described case, the values are the component descriptions of the next level in the tree, whereas in the running case the same attributes hold the RMI references to the running components represented by the equivalent component description.

Consequently, the application forms a naming structure mirroring the one in the description and indeed, references may be used to traverse the structure in exactly the same way as with the descriptions. The difference is that once the application is deployed, references to components return the RMI object reference to the component rather than a copy of the attributes in the referenced description.

Representing Components With Attributes

Applications are described using the SmartFrog notation. To create an application, its description is parsed, resolved and the sfConfig attribute extracted. The ComponentDescription obtained is given to the framework to create and manage the components associated with that description. The means by which the SmartFrog framework does so depends on the attributes that are present within the ComponentDescriptions.

Each description that is intended to represent a running component has two types of attributes, though these are indistinguishable within the notation. These are

• the template attributes that define the Java class of the component, where it should be created, and certain other management aspects; these are all identified by their “well-known” names. All template attributes start with the letters sf.

• the component configuration attributes containing configuration information to control the component behaviour; these are all the attributes other than the template attributes, and their interpretation is component specific.

This section is concerned only with the template attributes. The template attributes may be split into several categories, the most important of which are described in the following sub-sections.

Defining the Component Class

Each component created by SmartFrog is an instance of a class that implements the Prim interface (and usually extends the class PrimImpl). The key attribute of any component is therefore defining the identity of that component class. This is done by providing the well-known attribute sfClass holding a string representing the full package and class name of the component. Thus the description:

sfConfig extends {

sfClass "org.smartfrog.example.AClass";

}

defines an application with a single component, an instance of the class AClass in the package org.smartfrog.examples.

The code for the component class is normally loaded from the codebase defined at the point of launching the daemon. However it is also possible to define a codebase for a specific component, or a whole sub-tree of a description, by setting the sfCodeBase attribute within the definition. This also affects all sub-components, which may reset the codebase to the default by setting the sfCodeBase attribute to the string "default":

sfConfig extends {

sfcodebase "default";

}

Note that the sfCodeBase defines an additional location where a class may be found – the default codebase is still searched for the definition.

Controlling Deployment

In addition to specifying the class of the component, it is necessary to define where the component is to be created. Components are created in SmartFrog processes known as daemons. Daemons have a variety of flavours, however all provide the ability for the SmartFrog infrastructure to request the creation of new components. The two primary flavours are the root daemon that must run on each host, started manually or automatically at boot time, and named sub-daemons that may be created on demand by the SmartFrog framework.

SmartFrog provides a mechanism by which daemons may be identified via a combination of host and process names, however it also provides the means by which other location identification mechanisms may be implemented; an example of this being the use of SLP.

The two key attributes for identifying locations are the sfProcessHost and the sfProcessName attributes, both strings identifying the host and process respectively. If neither attribute is present, the current process is assumed; the current process being whichever is currently carrying out the deployment. If the sfProcessHost attribute is present, but the sfProcessName is not, the root daemon is assumed. Clearly any such use of the sfProcessHost attribute assumes that a root SmartFrog daemon is running on the identified host, and that security settings of the remote host permit the local system to deploy to it.

Thus, the following example will create the example component in the root process on the specified host.

sfConfig extends {

sfClass "org.smartfrog.example.AClass";

sfProcessHost "15.144.56.243";

}

and the following will place the component into a sub-process named ExampleProcess, creating it if it does not already exist.

sfConfig extends {

sfClass "org.smartfrog.example.AClass";

sfProcessHost "15.144.56.243";

sfProcessName "ExampleProcess";

}

RMI

SmartFrog relies on Java/RMI to provide the transport layer for carrying out management tasks, such as initiating the creation of a component. SmartFrog also makes it easy to use RMI to provide inter-component communication for application purposes. In particular, the RMI reference of a component is returned whenever a LAZY link is dereferenced to point to a component. However, it is an overhead for all components to be full-blown RMI servers; it is only necessary that the first component of any hierarchy of components in a daemon be a server, though others may be so if required.

If it is unnecessary for a component to be exported as an RMI server, the sfExport attribute may be set:

sfConfig extends {

sfClass "org.smartfrog.example.AClass";

sfExport "false";

}

It is equally possible to set it to true to ensure that the RMI export is done, although this is also the default if the attribute is not explicitly set. If the export is set to "false", only components in the same JVM will be able to access the component.

Prim and Compound

In the notation, the root of all extensions is the NULL attribute set; this is normally indicated by extending nothing. However, when components are being described (as opposed to arbitrary structured data) it is conventional to use the description Prim as the root of all descriptions. This provides a single place where default-valued template attributes may be placed, thus guaranteeing that all components inherit them.

Prim is defined in the file org/smartfrog/components.sf. Thus, this must effectively be included in every description, as in the following example.

#include "/org/smartfrog/components.sf"

sfConfig extends Prim {

sfClass "org.smartfrog.example.AClass";

}

In addition to Prim, there is a standard component called Compound defined in the same components.sf file. This provides the definition of the most commonly used application-grouping component, which creates a set of components as part of an atomic collection. Thus, most applications are similar to the following example in their use of Prim and Compound:

#include "org/smartforg/components.sf"

Service extends Prim {

sfClass "org.smartfrog.example.Service";

// and other attributes

}

ServicePair extends Compound {

service1 extends Service{ … }

service2 extends Service { … }

}

sfConfig extends ServicePair {

sfProcessHost "localhost";

}

The Compound description is unlike Prim, in that it contains an sfClass attribute defining the compound implementation class, which means that it is directly instantiable in a deployment. Otherwise, it is empty but may be used to define default template attributes for all compounds.

Compound extends Prim {

sfClass “org.smartfrog.sfcore.compound.CompoundImpl” ;

}

Compounds are described further in section 4.

Lifecycles

An important aspect of the SmartFrog component model is the lifecycle. The lifecycle is implemented as a simple state machine, shown in the following diagram.

The transitions in the state machine are associated with actions implemented (if required) by the components. The transition actions are implemented by the invocation of methods on the component, during which the component may take any appropriate action. These transition methods, also known as the template methods, are indicated along side the transition in the diagram.

These methods are described more fully in section 3. Default minimal actions are provided if the component has no specific action to carry out during the transition.

In an application with multiple components, the lifecycle of the whole system is defined by the combination of component lifecycles in some order. It is not completely defined within SmartFrog as to how these lifecycles are composed as it depends on the specific components used, and specifically those that are compound components.

The root parent of the parent-child hierarchy is controlled by the framework, and it is responsible for triggering the lifecycle transitions. It is the responsibility of that component to transition each of its children. The most common component used as root, and as intermediate nodes of the hierarchy is the Compound and this defines a simple combined lifecycle aimed at providing the notion of a single atomic composition with a shared lifecycle:

• on transition to the instantiated state, all children are similarly instantiated;

• on transition to the initialized state, all children are initialized;

• on transition to running, all children are started;

• on transition of any component to terminated, all children and the parent are terminated, and thereby the whole hierarchy is terminated.

This specific semantics has some important properties. The entire application is stepped through the lifecycle in a synchronous way: all components are created before any are initialized and all components are initialized before any are started. Termination is rather different as its occurrence is asynchronous with respect to the other transitions and any component in the hierarchy may be the first to transition to the terminated state (i.e. unlike the other transitions, it is not only the root which may initiate it). Nevertheless, the semantics are such that the whole application will terminate if any component terminates.

There are other possible semantics for parent-child lifecycle combination other than Compound, and some are provided by the framework. In particular, the workflow package provides a number of combinations such as parallel composition and sequential composition. The workflow package, documented in an accompanying manual, is designed to provide a simple lightweight workflow-style capability to the SmartFrog framework.

Only Compound, and its close relative DetachingCompound, will be considered within this manual, as it is the core implementation of component composition and is used by all others.

The SmartFrog API

The SmartFrog API is in essence the interface that each component must implement. This is the Prim interface (not to be confused with the Prim description) in the package org.smartfrog.sfcore.prim. All the SmartFrog capabilities are defined through the methods defined in this interface. To provide default implementations of these methods, all components should extend the class PrimImpl to define their default behaviour and enable access to the framework services.

The API is divided into two parts:

• The template methods that a component may wish to redefine; these are primarily the lifecycle methods.

• The utility methods that provide the component with the ability to access the SmartFrog framework capabilities, which includes access to the component’s configuration attributes. Equally they provide the SmartFrog system or other components access to the same capabilities on that component.

Both of these parts are defined in the Prim interface and are fully documented in the accompanying Javadoc; a partial description of the more important aspects is given in section 3.

Primitives

Primitives are the core of the SmartFrog component model, they are the basis for all the components that are created to implement application behaviour. Implementing a prim component in Java is the essence of SmartFrog programming.

Creating a prim class involves the following steps:

• Inheriting from the PrimImpl class and implementing the Prim interface (note that although PrimImpl itself implements Prim, it is necessary for RMI that the component directly declares that it implements it or another Remote interface.

• Providing an empty default constructor throwing the RMI remote exception, this constructor is used by SmartFrog to create the component instance.

• Providing the appropriate lifecycle template methods if they are required – default methods are provided by the PrimImpl class should no action be required at a specific point in the lifecycle.

• Implementing any component-specific interface required by the component.

The component-specific behaviour defined in the template methods or in the component-specific interfaces may use the utility methods.

Template Methods

The template methods are defined to be the ones shown in the lifecycle diagram, namely:

• sfDeployWith(…) is the method that is used alongside the default constructor for the basic creation of a component. It should not be overridden with application specific behaviour.

• sfDeploy() is the method that the system invokes to transition the component from the created state to the initialized state. Users may override this method to provide component specific initialization code, though care should be taken to invoke the superclass implementation to maintain correct behaviour. After this method, components should be ready to receive requests from other components in the application. Thus, listener threads must be started but threads that invoke other components may not. The component may assume that all other components in the same application have been created, but may not assume that they have been initialized.

• sfStart() is the method that the system invokes to transition the component from the initialized state to the running state. Users may override this method to provide component specific initialization code, though care should be taken to invoke the superclass implementation to maintain correct behaviour. The method will start any required active threads. The component may assume that all other components in the application have been initialized, but may not assume that they have been started.

• sfTerminateWith(TerminationRecord tr) is the method that may be called at any time to transition from any state to the terminated state. The termination record contains details of the reason for termination. All threads must be stopped, all resources must be released, etc. No assumptions may be made as to the state of any other component in the application. The superclass termination method must be invoked to ensure correct behaviour.

Thus, a typical SmartFrog primitive has the following shape:

import org.smartfrog.sfcore.prim.Prim;

import org.smartfrog.sfcore.prim.PrimImpl;

import org.smartfrog.sfcore.prim.common.SmartFrogException;

import java.rmi.RemoteException;

public class MyPrim extends PrimImpl implements Prim, … {

/* any component specific declarations */

public MyPrim() throws RemoteException {

}

public synchronized void sfDeploy()

throws RemoteException, SmartFrogException {

super.sfDeploy();

/* any component specific initialization code */

}

public synchronized void sfStart()

throws RemoteException, SmartFrogException {

super.sfStart();

/* any component specific start-up code */

}

public synchronized void sfTerminateWith(TerminationRecored tr) {

/* any component specific termination code */

super.sfTerminateWith(tr);

}

/* any component specific methods */

}

This template is absolutely the key to programming SmartFrog components. The three template methods may be left out if they are not required. Note the fact that all the three methods call the super-class method, and note that in sfDeploy and sfStart this is done before the component specific part, whilst with sfTerminateWith this is done at the end. The sfTerminateMethod must also be written so as not to fail if called before the component was actually started.

Utility Methods

The utility methods are defined in PrimImpl designed to provide a component programmer with the key utilities to enable interaction with the SmartFrog system. These are the ability to find and manipulate attributes, locate application components and to terminate itself and hence (under normal circumstances) the whole application.

The utility methods are primarily defined in Prim, however other methods are defined in the RemoteReferenceResolver interface that Prim extends. A brief overview of these methods is now given, however a more complete definition is provided with the Javadoc. The precise signature of these methods should also be checked in the Javadoc,

Object sfResolve(Reference r)

Locate an attribute when given a reference. This method resolves from the point in the component hierarchy that is the Prim on which it is called. References may be constructed using the class constructor (recommended) or the fromString(“…”) method defined on Reference to provide a simple, though inefficient, way of creating complex references. The class Reference is documented in section 11.4.

void sfTerminate(TerminationRecord tr)

Terminate the component and hence (under normal circumstances) the application. This will cause the termination template method to be called at some point in the future, though not necessarily immediately.

void sfDetach()

Split the application tree into two independent components; the parent-child link becomes severed and thus the lifecycles of the parent and the child components become separated. The child component becomes a root component, i.e. one with no parent.

sfAddAttribute/sfReplaceAttribute/sfRemoveAttribute

These are a collection of methods that allow the modification or removal of the component’s attributes.

Prim sfParent()

Returns the parent of the component, or null if it is a root component.

Context sfContext()

Each component contains the context that was originally given to the component as its description. This method returns that context.

Reference sfCompleteName() throws RemoteException

Returns the complete name of the component from the root of the application. Throws exception if unable to trace the component hierarchy starting from the root.

Reference sfCompleteNameSafe()

Returns the complete name of the component from the root of the application and does not throw any exception.

These methods form the basis of the programming API for components providing interactions with the SmartFrog system. Other methods, to do with dynamically modifying the structure of the application hierarchy, such as deploying new branches of the tree, are covered in the section on Compounds, section 4.

Compounds

SmartFrog provides the ability to create collections of components with certain semantic guarantees. These collections – whatever their semantics – are known as compounds. SmartFrog provides a small number of these collections; others may be defined as required. The primary collection component is simply known as “Compound” and is the root of all other collections.

There are three aspects to consider with Compound

• The component description for compound – contained in the file "/org/smartfrog/components.sf".

• The interface Compound.java.

• the implementation of the base class of all compounds: CompoundImpl.java.

Compound Component Descriptions

The component description is an extension of the component description Prim and adds attributes relevant to the liveness mechanisms and, importantly, it adds the sfClass attribute to ensure that the compound component description generates an instance of the CompoundImpl class when deployed.

In order to define a collection of components, the description must extend Compound and add any Prims or Compounds that are required. For example

#include "org/smartfrog/components.sf"

sfConfig extends Compound {

component1 extends Comopund {

foo extends Prim { sfClass "…"; }

bar extends Prim { sfClass "…"; }

}

baz extends Prim {sfClass "…"; }

}

defines the system to be a collection containing a sub-collection and a primitive. The sub-collection component1 contains two primitives. Note that the extension of the Compound does not contain an sfClass attribute as this is supplied by the definition of Compound in the include file.

The Compound Interface

The Compound interface, in the package org.smartfrog.sfcore.compound, is an extension of the Prim interface, hence any compound is by definition also a primitive component. It also defines a number of additional features:

• Compound extends the interface ChildMinder that provides methods for registering and removing child components. These methods will rarely be used by SmartFrog programmers as they are automatically invoked as required by the underlying system. These methods are documented in the Javadoc for the interface.

• Compound directly defines a number of methods to allow applications, should they desire it, to deploy additional application descriptions. Although in the normal course of events this will not be used directly by SmartFrog programmers, it is sufficiently frequently used to merit some further explanation here. There are three methods:

public Prim sfDeployComponentDescription(Object name,

Compound parent,

ComponentDescription cmp,

Context parms)

throws RemoteException, SmartFrogException;

public Prim sfCreateNewChild(Object name,

ComponentDescription cmp,

Context parms)

throws RemoteException, SmartFrogException;

public Prim sfCreateNewApp(ComponentDescription cmp,

Context parms)

throws RemoteException, SmartFrogException;

The first of these is primarily intended to be an internal method for use by the framework and should be used cautiously by programmers. The method is given a component description that has already been parsed and resolved and deploys it (i.e., creates and calls sfDeployWith(...)) with the following additional aspects:

• A parent, an existing Compound including the one implementing the called method, may be provided, or null if no parent is required. If no parent is provided, a wholly new application is created.

• The name by which this component is to be known within that parent compound – again may be null if a parent is not required

• A set of additional attributes that should be placed into the top-level component description before deployment.

• The name by which this component is to be known within that parent compound – again may be null if a parent is not required

• A set of additional attributes that should be placed into the top-level component description before deployment.

The error handling during the lifecycle is defined to be to detach and terminate the new child, if necessary, and to throw an appropriate exception in the calling thread.

The second of these methods, sfCreateNewApp, is used to create a new application which is its own root – it has not parent. Consequently, no parent is required, and no name for it to be associated with within a parent. Consequently, in addition to the description, the only additional data required is

• A set of additional attributes that should be placed into the top-level component description before deployment.

The error handling during the lifecycle is defined to be to terminate the application (there is no need to detach – it is a new root), and to throw an appropriate exception in the calling thread.

CompoundImpl

The class CompoundImpl in package org.smartfrog.sfcore.compound is the implementation of the core capabilities of SmartFrog collections. In addition to being the class which is extended whenever a new type of Compound is defined, it provides all the semantics for one of the most common type of grouping – the shared-lifecycle component collection.

The purpose of the CompoundImpl is to provide a collection with the following semantics:

• The Compound provides a notion of child component – one for whose lifecycle it is responsible. A child is an attribute of the Compound, but also has a specific relationship relative to lifecycle management and failure monitoring.

• The Compound and its sub-components share an identical lifecycle, i.e.

• whenever the compound is created, so are the sub-components in order of definition

• whenever the compound is initialized, so are the sub-components in order of definition

• whenever the compound is started, so are the sub-components in order of definition

• whenever the compound fails or is terminated, so are the sub-components; this may be done asynchronously or synchronously in order of definition

1. This phasing of the lifecycle is of extreme importance to the SmartFrog system. The phasing carries through the hierarchy of the compounds and effectively provides a top-down, depth-first traversal of the compound parent-child containment tree. Thus, each child is guaranteed that each of the other children (and indeed the entire tree) has been stepped through the previous phase of the lifecycle before it will step through the next. Thus, all components will be initialized before any are started, and so on. The only exception to this is termination, which may occur at any time and may be completely asynchronous.

• The Compound monitors its children for their status using the child’s sfPing template method. If there is any failure of this method (either an exception or the method returning false), the child is assumed to have died and the compound propagates this as defined by the shared lifecycle principle. Note that other semantics for failure recovery, or other semantics for lifecycles, may be implemented and many are provided within the accompanying workflow package

Component Template

The following text is the typical outline for a primtive component in SmartFrog

import org.smartfrog.sfcore.prim.Prim;

import org.smartfrog.sfcore.prim.PrimImpl;

import org.smartfrog.sfcore.prim.common.SmartFrogException;

import java.rmi.RemoteException;

public class MyPrim extends PrimImpl implements Prim {

/* any component specific declarations */

public MyPrim() throws RemoteException {

}

public synchronized void sfDeploy()

throws RemoteException, SmartFrogException {

super.sfDeploy();

/* any component specific initialization code */

}

public synchronized void sfStart()

throws RemoteException, SmartFrogException {

super.sfStart();

/* any component specific start-up code */

}

public synchronized void sfTerminateWith(TerminationRecored tr) {

/* any component specific termination code */

super.sfTerminateWith(tr);

}

/* any component-specific helper or public methods go below here */

}

The template to write a basic modified compound is similar, but using Compound and CompoundImpl in place of Prim and PrimImpl.

Well-Known Attributes

SmartFrog defines a number of predefined component templates, such as the ones for ProcessCompound, which require specific attributes to be defined. These attributes are captured in the interface:

org.smartfrog.sfcore.common.SmartFrogCoreKeys

The special attributes used in the framework are:

• sfProcessHost: Attribute used to determine the host to use to locate the root process compound on that host.

• sfProcess: Attribute used to determine the process/subprocess name where a component runs.

• sfProcessName: Attribute used to name a process/subprocess

• sfProcessComponentName: Attribute used to name a component

• sfRootLocatorPort: Registry port used by the rootProcess daemon

• sfSubprocessGCTimeout: Attribute with garbage collection time out for subprocesses

• sfHost: attribute used to determine the host address where a component runs

• sfProcessAllow: Attribute used to define if subprocesses can be used

• sfProcessTimeout: Attribute with subprocess deployment timeout

• sfProcessJava: Attribute that holds the process java start command

• sfProcessClass: Attribute that holds the class name for subprocesses

• sfDeployerClass: Attribute that holds the class name for deployer

• sfSyncTerminate: Attribute that determines asynchronous or synchronous termination of compound

• sfClass: Attribute that holds the class that implements a component

• sfConfig: Attribute that determines the resolution root of a SmartFrog description

• sfSchemaDescription: Attribute that determines the definition of a schema

• sfCodeBase: Attribute that defines the codebase for a component

• sfLivenessDelay: Attribute that defines how often to send liveness in seconds.

• sfLivenessFactor: Attribute that defines how many multiples of the liveness delay to wait till a liveness failure of the parent is declared

• sfExport: Attribute that defines if a component has to accept remote method calls

• sfRootLocatorClass: Attribute that defines the root locator class

• sfBootDate: Attribute that hold the boot time of the root process daemon

Some special names used in the framework:

• rootProcess: Name used to name root process

• ROOT: Name used to refer to the root reference in a particular hierarchy of components or description

• sfRunProcess: Name used to name a root process deployed without registry

• unnamed_: Prefix use to name unnamed deployments

• sfDeployFailure: Name used to name a deploy phase failure

• sfStartFailure: Name used to name a start phase failure

Deployment In Detail

In SmartFrog, the term deployment is used to indicate the creation of the components in response to the system being given a suitable parsed and resolved description. A parsed component description may be deployed in two primary ways:

• by invoking the sfDeployComponentDescription method on a compound, and passing the parsed component to it – this would on the whole be restricted to those cases when the compound is to be the parent, or when it is simply desired to have the initial deployment occur near the compound (i.e. in the same JVM).

• by invoking the same method on the ProcessCompound of the SmartFrog JVM in which the top-level component should be created, though any use of deployment control attributes, such as sfProcessHost, may still cause the component to be deployed elsewhere.

The SmartFrog system uses the second of these two mechanisms for deploying the descriptions that are given to it on the command-line.

In either case, the deployment semantics are identical and proceeds as follows:

• The compound passes the top description to the deployer.

• The deployer, if it is the right one, examines the description for the attribute sfClass, extracts the class-name and creates an instance of the component class using the default constructor. If it is not the correct deployer, it locates and passes on the request – see section 2.1 for a description of how this is done.

• The instance is initialized using the sfDeployWith method, being passed its component description as a parameter.

• If the component is a prim, it carries out some basic initialization and returns.

• If the component is a compound, it examines the attributes of the description and extracts those that are non-lazy component descriptions. It passes these descriptions to the deployer to create an instance of its subcomponents.

• The object reference (possibly remote) is returned as the result of the sfDeployComponentDescription method call. The receiving object is then responsible for invoking the remaining lifecycle methods in the correct order.

A number of facts should be noted. Firstly, that any component tagged LAZY is not deployed by its parent compound – this leaves the description to be used as structured data or for later programmatic deployment. Secondly, the semantics of what is considered a component in a description is not defined by the framework; rather it is defined by the compound component. This alternation of responsibility between the deployer to create a single instance of a component and the component to decide the next steps is an important feature of the deployment mechanisms within the SmartFrog framework. The last fact is that the follow-up initialize/start lifecycle is entirely up to the components that control the deployment – they are responsible for invoking these methods. It is true that the framework will invoke a specific lifecycle on the initial descriptions that are passed on the command-line – however, any other component that initiates a deployment may chose to impose a different lifecycle if so desired.

Selecting Deployers

A deployer is a Java object that is capable of creating an instance of a component in a specific place, as described by attributes in the component description. SmartFrog comes with three deployers, each providing a little more capability than the previous. Some of the services supplied with SmartFrog may add additional deployers. There are two aspects to cover

• how a deployer is chosen

• how each deployer interprets the attributes to select a location for deployment.

The first of these is either the class defined by the sfDeployerClass attribute of the component description to be deployed or the default deployer if this is not provided. The default is an instance of the PrimProcessDeployerImpl class. Thus in the example

sfConfig extends Compound {

sfDeployerClass "org.smartfrog.example.NewDeployer";

…

}

The component will be deployed with the NewDeployer class. If this attribute had not been present, the default would be used. Each component in a hierarchy may be deployed with a different deployer.

The three deployers provided by the core SmartFrog system are as follows:

• PrimDeployerImpl – deploys the component in the current JVM, ignoring any other attributes

• PrimHostDeployerImpl - deploys the component in the root process compound of the host whose name is provided in by the sfProcessHost attribute. If no name is provided, the current JVM is used.

• PrimProcessDeployerImpl – deploys the component in the named process (or the root process if not provided) of the specified host. If neither is provided, the deployer uses the current JVM. The process name is defined using the sfProcessName attribute, the host using the sfProcessHost attribute. This is the default deployer.

Thus, the following description will be deployed on the appropriate host and process, given that the default deployer will be selected.

sfConfig extends Compound {

sfProcessHost “sfhost.smartfrog.org”;

sfProcessName “example”;

…

}

A description of processes, root processes and named sub-processes, is given in section 7.

Termination

Termination may be initiated on a component in one of three ways:

• By a call of the API method sfTerminate(…) by a thread started by the component, or from an external source such as a management system.

• From the parent of the component.

• From a child of the component, indicating that the child has terminated – although depending on the semantics of the compound, this may trigger an action other than the propagation of termination. The standard CompoundImpl, though, will propagate this termination to itself, the other children and its parent.

Whereas for most of the component lifecycle stages, the compounds and primitives are stepped synchronously through the cycle – each is created before any are initialized, and each is initialized before any is started – this is not necessarily a good model for termination. In many cases, termination may be carried out asynchronously. Indeed, in the case of failure of a component or the network, this is an absolute necessity. In other cases, it may be desirable to terminate in a specified order whenever possible.

SmartFrog allows a degree of control over the termination process. The default model is to terminate asynchronously, notifying parent component and child components in arbitrary order, and calling the termination template method on the component – the sfTerminateWith(…) method – at some point in that process.

If a component must be terminated synchronously, the attribute sfSyncTerminate must be set to the boolean value true. If not set, or set to false, the component will terminate asynchronously.

Synchronous Termination

Synchronous termination causes a component to carry out the following steps in order:

• Inform the children to terminate, waiting for each to be complete until the next is invoked. This is done in the order of definition in the component description.

• Terminate self by calling the sfTerminateWith template method.

• Notify parent that termination has occurred, if not initially told by the parent to terminate.

Note that a child component need not terminate itself synchronously, this is set by the attributes within the child. If it is desired that all components terminate synchronously, the definition of Prim should be modified in the file components.sf to include the appropriate attribute.

Asynchronous Termination

This is the default mode of termination. The same three steps are undertaken, though the children are all informed of the need to terminate in separate threads that are created for this purpose. The implications of this are that there may be some delay in the termination process, that resources may not be freed immediately. However, it also means that long termination sequences are scheduled independently of notifying its termination to its parent. This fast notification may be important for fault tolerance purposes, for example.

Terminator Thread

As mentioned in section 9.2.2, asynchronous termination is done using a separate thread. Terminator thread provides a standard way of asynchronously terminating SmartFrog components. This class is defined in org.smartfrog.sfcore.common package. For example:

// create an instance of terminator thread by passing component reference and termination record

TerminatorThread terminator = new TerminatorThread(compRef, terminationRec);

// Start the thread

terminator.start();

// Terminator thread object can also be constucted using another constuctor

public TerminatorThread(Prim target, Throwable t, Reference compId);

Some utility methods are available in the TerminatorThread class:

// returns a TerminatorThread object which does not notify

// components parent while terminating the component

public TerminatorThread quietly();

// returns a TerminatorThread object which

// detaches the component from parent before terminating

// the component

public TerminatorThread detach();

// returns a TerminatorThread object which

// does not terminate the underlying component

public TerminatorThread dontTerminate();

Attributes, LAZY Links and RMI Object References

Accessing Attributes At Runtime

Attributes of a component are stored in a Context, as defined by the ComponentDescription used to create the component. These attributes may be accessed using the sfResolve method of the Prim interface. This method takes a reference and, relative to the component on which it the method is initially invoked, de-references it to obtain the desired attribute. This method may equally be used by any component, one that has an RMI reference to the object or by the component itself. If called within the body of component, the reference is de-referenced starting with the component itself.

The sfResolve method is defined as follows.

Object sfResolve(Reference r)

Locate an attribute when given a reference. This method resolves from the point in the component hierarchy that is the Prim on which it is called. References may be constructed using the class constructor (recommended) or the fromString("…") method defined on Reference to provide a simple, though inefficient, way of creating complex references.

This method will return an Object that may be inspected using reflection or caste to an appropriate class. For most attributes it is clear from the context what the value returned should be.

Although the above method is the primary one for accessing attributes, there are a number of other methods with a number of semantics – they are all documented in the Javadoc.

LAZY links And RMI

If the description contained a LAZY link, the link resolution phase would not have resolved it in advance – rather it would have left the reference to be the value of the attribute. There are now two possibilities when sfResolve is invoked:

• That the reference is de-referenced at this point, returning the appropriate value.

• That the reference itself is returned as the attribute value.

The first of these semantics is that implemented by sfResolve (though not by all methods in the Prim interface). Consequently, if an attribute is a LAZY link, it is silently de-referenced at the time it is accessed, following the parent-child component hierarchy in exactly the same way as is done with ComponentDescription declarations during link resolution.

Note that the value returned by the resolution is not cached locally. It will be fetched each time the attribute is accessed. The value should be cached by the component itself if this is required.

There are several reasons why a LAZY link might be used instead of a normal link, in spite of the fact that delaying access in this way is much less efficient – particularly so when the attribute referenced may be remote. These reasons are all related to the property that the value of the attribute is not available at time of link resolution.

• The attribute referenced may be set at run-time by the component, or may continually change its value and must be read many times. This variability is the reason for not automatically caching values locally.

• The link may reference a component with the intention of obtaining the RMI object reference of the component

Consider the following example:

sfConfig extends {

comp1 extends Prim { … }

comp2 extends Prim { …

otherComp LAZY comp1;

}

}

The link to comp1 from within comp2 is defined as LAZY. This is because the link, if it de-referenced at the time of link resolution, results in a copy of the ComponentDescription that is comp1 being placed within that of comp2 under the name otherComp. Thus, if it were not LAZY, after resolution the result would be the resolved equivalent of

sfConfig extends {

comp1 extends Prim { … }

comp2 extends Prim { …

otherComp extends Prim { … };

}

}

which is not the intended behaviour. If the link is made lazy, on the other hand, the resolution is carried out at run-time by the Compound/Prim hierarchy and instead of the ComponentDescription being copied, the RMI object reference of the component referenced is returned. Under these circumstances, comp2 would be able to call the exported methods of comp1.

The Moving ROOT

The root reference part appears simple, but it can have some unexpected side effects. The meaning of the ROOT reference part is that, at the time of resolution, the ComponentDescription or Compound/Prim hierarchy is traversed upwards from the point of initial resolution until there is no parent. This top ComponentDescription or Component is the root.

Therefore, when a link is defined using the ROOT reference part, the ROOT refers to the file itself, the virtual ComponentDescription containing all the attributes at the top level.

At run-time, however, this ComponentDescription no longer exists – the component called sfConfig from that file is normally the new run-time root. Consequently, LAZY links with ROOT resolve differently from the way they would have done if they had not been lazy and had been resolved at the time of link resolution.

The situation is even more confused, though, since the structure of the tree can change during the life of an application. For example, the use of sfDetach changes the structure of the tree during its execution. This difficulty in pinning the meaning of ROOT can cause great confusion and therefore the use of ROOT should be limited to occasions where the semantics are clear.

Modifying Attributes Values

Given the use of LAZY links, with the ability to read attribute values many times, they may be used as a way of passing information between components. To provide the ability to add, remove and replace attribute values from the context, three methods are provided as part of the Prim interface. Note that these descriptions of these methods do not accurately define the signature. These should be found from the Javadoc.

public Object sfAddAttribute(Object name, Object value)

Add an attribute to the component's context. Values should be marshallable types if they are to be referenced remotely at run-time. If an attribute with this name already exists it is not replaced.

public Object sfRemoveAttribute(Object name)

Remove named attribute from component context. Non present attribute names are ignored.

public Object sfReplaceAttribute(Object name, Object value)

Replace named attribute in component context. If attribute is not present it is added to the context.

Trapping Accesses And Reference Adaptors

Attributes of a component are stored in a Context, as defined by the ComponentDescription, and made available through the Prim interface using the sfResolve method. However, it can be useful for a component to provide an attribute that is not stored within its context but is evaluated on demand, for example an attribute representing the current load on the component.

For this, the request for an attribute value must be trapped, diverted from the lookup in the context, and the correct value returned. This is possible by understanding how a reference is de-referenced.

The sfResolve method is almost immediately converted into a request to resolve the reference from a specific index of the reference parts using the method

Object sfResolve(Reference r, int index)

This method resolves one part to get to another ComponentDescription, and forwards the request adding one to the index. In this way the reference is eventually fully de-referenced and the attribute value returned.

This method is implemented by PrimImpl, and its default implementation is to look up in the context and forward if the reference is not yet completely de-referenced.

The accepted way of trapping a request is to override the default implementation in the appropriate component class, checking to see if the attribute is one that needs special handling and if not, invoking the super-class method to continue as normal. If it has to be handled specially, carry out the appropriate processing and return the evaluated attribute value.

This mechanism has some interesting possibilities. For example, it is possible to intercept a reference early on in its de-referencing process, within a component that may then use the rest of the reference parts to be a parameter to the component to evaluate its attribute value. This may be used, for example, to provide an adaptor into other naming and discovery technologies.

Consider a component that uses a database to extract a value for an attribute given a table name, a key and the column name required. Assume further that this component has been deployed on a specific host, say db.smartfrog.org, with the sfProcessComponentName set to dbAccess. Descriptions may now use references of the form

foo LAZY HOST db.smartfrog.org:dbAccess:ttt:kkk:ccc

where ttt, kkk, ccc are the table name, key and column respectively. This is again an example of how a component which is accessing the attribute foo need know little of how the attribute’s value is obtained. This is an issue for the configuration not for the component code.

sfHost and sfProcess Attributes

Every component, when created, adds two attributes to its collection of attributes. These are the hostname and process name in which the component is running. These are defined in the very earliest phase of the lifecycle, during construction. Consequently they are present during all other phases.

Attribute Serialization

At run time, when components are scattered across a number of hosts, components exchange the values of attributes through the use of link resolution. As these values are passed around, they traverse a number of hosts. For example, consider the diagram below:

In this diagram, component A requests the attribute “foo” from component B whenever the attribute “bar” is accessed. This is passed back in response to the request to resolve the bar attribute. The attribute has to be of a class that is known to both hosts containing the components A and B, otherwise some form of exception will occur. However, what is not so obvious is that the attribute value is also passed through the compound component containing A and B.

It should be noted that although in the language a very limited number of classes can be used for attribute values, and these are known everywhere, at run-time attributes can have other values than these simple ones, indeed any serializable class, and in particular this will include all the RMI stub classes for the various components.

Early versions of SmartFrog simply insisted that the classes of attributes that were passed through intermediate nodes in response to a resolution request had to be known to those intermediate nodes. This was not an adequate solution, so an alternative model had to be found.

An attempt at solving this automatically wrapped all attribute values within a well-known wrapper class (SFMarshalledObject) for passing around. This wrapper held the actual value as a byte array containing the serialized value. In this way only the wrapper class needed to be known at every host and the serialization and deserialization simply occurred transparently at each end of a link resolution (in this case at the hosts containing A and B). At the intermediate node, only the wrapper would be handled and the real attribute value would be held as bytes within it.

This solved many of the problems associated with the need for attribute classes to be known everywhere, but it introduced others. Consider if two components are always together on the same host. They locate each other using a link in the normal way, and use local interfaces to communicate (this is often done for security reasons, not to expose to remote calls sensitive interfaces). As the objects are serialized and deserialized within any link resolution, the result is always a remote object reference even though the object is local. This cannot be caste to a local-only interface, so it became impossible to ever call a local interface on a component located through link resolution. As it is impossible to extract the real object from an RMI Object Reference, so there is no easy way out of this problem.

The final solution, and that in the current version, is for the wrapper to be a little more intelligent about the serialization of the value it wraps. It only does so if it is being serialized itself – so if the wrapper is never passed between hosts, the value it wraps is not serialized either.

This fixes most of the problems associated with local interfaces, as well as solving the issue of classes having to be known everywhere. However it is not perfect. If a value is ever passed remotely, even if it ends up on the original host, it will have become serialized. A classic example of this is shown in the following diagram.

Although A and B are on the same host, the compound is not and the attribute referenced will therefore pass through the remote host. So if A and B need to communicate using local interfaces, this is not a suitable description. An additional container compound will need to be added on the same host as A and B to act as the local container and so ensure that the link resolution never traverses a host boundary. This is also more efficient for deployment and for liveness, so is anyway to be encouraged.

Note that in the discussion above, for host one should really read process. Crossing process boundaries, even on the same host, introduces the same problems.

Liveness

A feature of SmartFrog is its all-or-nothing semantics for Compounds, its shared lifecycle for all components. This must be done in a context where any host within the network or, indeed, the network itself, may fail at any time. These failures may result in applications losing some components, or perhaps being partitioned into two or more parts if the network itself fails.

The guarantees that SmartFrog attempts to provide, such as all-or-nothing deployment, cannot be achieved by purely passive means. The system needs to monitor the various components to ensure that they are still active, can be accessed and are in good shape. Detection of the failure of an application component, or of the network between such components, must be notified to the application in an appropriate way. Note that the difference between failure of a node and the failure of the network between nodes cannot normally be diagnosed from the perspective of another node.

The hierarchical nature of an application provides a natural chain of responsibility – a parent is responsible for the checking and monitoring of its children, and vice-versa. Notification of failures will be made to these components and it is the responsibility of these components to take appropriate action.

SmartFrog provides a default liveness checking mechanism based on a parent regularly heart-beating its children. Each component provides a method sfPing as part of the Prim interface. This method either returns or throws an exception, and it may be overwritten by the component implementer to carry out any appropriate checks.

There are two possibilities when this method is called:

• The method succeeds, and returns normally – this is considered as confirmation that the component is alive.

• The method fails by throwing an exception, implying one of the following:

• that the remote host has failed.

• that the network has failed and the two components are out of communication.

• that the component is indicating that an error has occurred and it should be considered as dead.

1. The default Compound semantics is to terminate itself under any of these circumstances and hence the whole application follows suit – again alternative semantics may be defined.

On the child side, it monitors how frequently the parent is checking its state. If the parent misses more than a specified number of heart-beats, the child assumes that the parent has failed, or there is a network failure, and handles the failure – again the default is to terminate itself (and hence any part of the application below it in the hierarchy).

Should the parent or network recover, the child will respond with an exception to any request for status, and the parent and hence the rest of the application will terminate by default.

If the default behaviour needs to be changed, the report to the component is done via a call to the method sfLivenessFailure. This method, in the default implementation provided by PrimImpl, simply terminates. This method may be over-ridden if required.

To try to be a little more efficient, the SmartFrog system does not create a liveness thread for each component. Rather, it creates one for every component that is in a different process. This thread is responsible for checking all the components in the hierarchy downwards until it has checked a single level of remote components. Equally, it is only the “first” component that monitors whether it is being checked on a regular basis. Consequently, overheads for large numbers of components on one host are minimal. The following diagram makes this clear.

The frequency of liveness checking can be modified to balance responsiveness requirements and overheads in any particular system. Two attributes may be set:

• sfLivenessDelay: (default 15 seconds) how frequently in seconds should the liveness be checked. Zero indicates no checking.

• sfLivenessFactor: (default 2 misses) how many missed heart-beats should be considered a parent failure.

These may be modified for the entire system by setting them in prim.sf, or for all components rooted in a process by setting the attribute definitions in the process compound for that process, or it can be set for the sub-tree of a component by setting it in that component.

Each component searches for the current value of these attributes up the containment tree up to the root (using an ATTRIB reference part). If it does not find them anywhere up the tree, including in the root component, the default is taken from the local process compound (this defines the defaults of 15 seconds and 2 misses given above).

Hooks

At times, it is necessary to carry out an action for some or all of the phases of the lifecycle of every object – a typical use being to trace or log the components that come and go. To enable this, a set of lifecycle hooks have been provided, such that at every component lifecycle phase the appropriate hook is called, parameterized by the component.

The hooks are called in the PrimImpl base class, as part of the default implementation of the template lifecycle methods. For the hooks to be called, therefore, it is essential that the superclass template methods are called in any derived class, as indicated in the primitive template in Appendix E.

To create a new hook class, the class should implement the interface PrimHook in package org.smartfrog.sfcore.prim. This interface defines a single method, sfHookAction, which is called when the hook is invoked. This method is documented in the Javadoc.

To register a hook, an instance of the hook class should be added to one of the four HookSet members available in Prim (using the addHook method), namely sfDeployWithHooks, sfDeployHooks, sfStartHooks, and sfTerminateWithHooks; one for each of the template methods. Note that the effect of the hook is local to a specific process.

Removal of hooks is also possible, so creating a component that adds a hook on start, and removes it on termination, is feasible – so for example a management component could, on demand, capture all the lifecycles for a period of time at a process and send the information back to some central console.

The mechanism is primarily intended for debugging and management.

Processes and Java Virtual Machines

The SmartFrog system is designed to form the basis for a fully distributed configuration and programming environment. As such, the system must be able to deal with deploying components into many processes (Java Virtual Machines) on many different hosts. This section covers the subject of these processes, how they may be identified and located, and how they may be started and managed. It also deals to a degree with the APIs provided to allow applications to interact with them, however, as usual, the full API is described in the accompanying Javadoc.

There are two concepts to understand as part of the underlying control of the SmartFrog system. The first of these is the SmartFrog Resource Reference (SFREF). This is a URL to a description file to deploy or otherwise use. The second is an SmartFrog Action Descriptor (SFACT), which is used to indicate to SmartFrog an action to take. These are now described in more detail.

SmartFrog Resource References

Throughout the SmartFrog system, including on the command line, references to SmartFrog resources (i.e. files) may be given in a number of ways:

• as a URL to the file;

• as a relative or absolute path name to a file;

• as a path to a resource in a jar file on the classpath or code base.

In this last case, the reference should be given as a path relative to the root of the package structure within the jar file, i.e. without the leading "/". In most cases this leading "/"is removed by the code, but there may be some instances where this is not so.

In the following descriptions of the scripts, a reference to such a resource is referred to as an SFREF.

SmartFrog Action Descriptor

An action descriptor is used on the command line to describe a certain type of action that will be carried on by the daemon. An action has a number of “:” separated fields, most of which may be left blank in many cases.

The format is:

NAME:ACTION:SFREF:SUBREF:HOST:PROCESS

The semantics of the fields are defined as follows:

NAME

The name is a single word, or a SmartFrog reference in which case it must be surrounded by quotes. The name has one of two interpretations depending on the action to be taken (see next field).

In TERMINATE, DETACH, DETaTERM, PING, DIAGNOSTICS the name is a reference to the component on which to apply the action.

In DEPLOY, the name is treated like a placement and the name is split into two: all but the last part is a reference to another component and the last (or only) part is the name which will be given to the deployed component within that referenced component. If the component is not a ProcessCompound, the component is also made the parent of the deployment.

In all cases, the NAME is resolved relative to the process compound of the HOST and PROCESS specified by the appropriate fields.

When a name is not provided, it indicates the process compound of the host and process defined in the HOST and PROCESS fields. Also, in DEPLOY no name means use the sfProcessComponentName from the description if available or generate a random name to name the deployed description.

Examples:

foo

"HOST localhost:foo"

ACTION

This field defines the action to be taken on the named component

• DEPLOY a component or application.

• TERMINATE a component or application.

• DETACH a component from its parent.

• DETaTERM detach and terminate a component from its parent.

• PING a component.

• PARSE a description and generates report.

• DIAGNOSTICS a component and generate report.

SFREF

The SmartFrog description (if needed) to be used by ACTION. It is a SmartFrog Resource Reference see 3.1. It needs to use quotes (“ or ') when the reference is using “:”. Currently this is only required for a DEPLOY and PARSE actions and is ignored otherwise.

Examples:

/home/sf/foo.sf

"c:\sf\foo.sf"

'c:\sf\foo.sf'

SUBREF

When the SFREF is parsed and resolved, the result is a component description containing a number of attributes. In the “sf” language, this is the contents of the sfConfig definition. Under normal circumstances, it is this whole definition that is used for the deployment, but occasionally, for testing purposes perhaps, it is useful to specify some single subcomponent. Under these circumstances, the name of this attribute, or a reference to a deeply nested application, may be provided. This is the SUBREF.

Examples:

foo

"fist:foo"

'fist:foo'

HOST

host name or IP from where to resolve the name. If HOST is not present, the process name is ignored and the process executing is used. If you want to refer to another process, other than the executing one, on the local host, "localhost" should be used and the appropriate PROCESS name used.

Examples:

foo.hpl.hp.com

127.0.0.1

PROCESS

process name from where to resolve the name. When empty it defaults to “rootProcess”.

Examples

These examples show the use of the action descriptors for different purposes.

Example 1: Deploy a description in the local daemon

Ex1:DEPLOY:org/smartfrog/examples/counter/example.sf::localhost:

Example 2. Terminate the local sfDaemon

rootProcess:TERMINATE:::localhost:

or

:TERMINATE:::localhost:

Example 3: Deploy "counterToSucceed" from counter/example2.sf

counterEx3:DEPLOY:org/smartfrog/examples/counter/example2.sf:"testLevel1:counterToSucceed":localhost:

Example 4: Get diagnostics report for "sfDefault" component running in localhost

sfDefault:DIAGNOSTICS:::localhost:

SFSystem And Command-Line Parameters

The main loop of a SmartFrog process is provided by the class SFSystem. When this class is invoked, it reads various command line parameters and system properties to generate the appropriate type of SmartFrog process and to trigger the desired configuration actions. The general form of the command line for SmartFrog is

java [–D properties] org.smartfrog.sfcore.SFSystem [parameters]

In general the command line will be triggered by a script setting up the properties as defined and described below, and arranging for the command line parameters to be correctly defined. The parameters are:

• -a SFACT: SmartFrog Action Descriptors (SFACT). There can be more than one of this. See format for SFACT in 13.2

• -f SFREF: file that contains a set of file that contains a set of SmartFrog Action Descriptors (SFACT). There can be more than one of this. See format for SFREF in 13.1.

• -e : exit after deployment of the configurations is complete.

• -?: usage and help information.

The system properties are rather more complex in their effect. To understand their effect on SmartFrog behaviour, the concept of a process compound must be explained. Of course, additional properties may be defined to parameterize specific applications that require it (this is particularly useful in conjunction with the PROPERTY and IPROPERTY links).

Process Compounds

Every process contains a specialized component known as a ProcessCompound created by SFSystem at start-up. It is a modified Compound providing the full component creation interface offered by all compounds. As such, it is the means by which all top-level components within the process are created. As it is a compound, it also contains attributes and sub-components:

• It contains attributes that affect the behaviour of the process, such as security settings, classpath information, and so on.

• It contains an attribute referencing each application that is running in the process. These applications are all children of the ProcessCompound, in the sense that the ProcessCompound does monitor the children for status. This is so they may be removed from the ProcessCompound on termination. The applications, however, do not consider the ProcessCompound as their parent.

• It contains, as a child, the ProcessCompound of any sub-process that may be created during the deployment process: it is possible to have a simple two-level tree of processes controlled by attributes within the ComponentDescription to be deployed. A sub-process does consider the root process as its parent and will terminate if the root dies.

However, to interact with the ProcessCompound in a process, for example to initiate a deployment, it is necessary to obtain the RMI object reference to that ProcessCompound. SmartFrog defines a core mechanism using the RMI registry for this purpose, though other mechanisms may additionally be defined. One such is that provided with the SLP discovery infrastructure described in a separate document. Only the registry-based mechanism is described here.

Types Of Processes

Processes come in three main types:

• Root Processes that create an instance of an RMI registry and register themselves within it so that they may be located. Note that a host may have several root processes so long as they use different ports for the registry. However there are restrictions in the current release that limit the degree of interactions of processes on different ports and hence this should only be done if the two belong to different SmartFrog systems. A SmartFrog system is designed so that a single root process exists on each host, and that these are on the same port number.

• Sub-processes are processes that register with the root process and become children of the root process, and hence are named within the ProcessCompound of the root process. This is the mechanism by which the sub-processes are located and their RMI references obtained. Subprocesses may be created in two ways – by explicit launching from a command line or by defining that an application should be deployed in a sub-process that is created if it does not already exist.

• Basic processes are processes that do not use the core SmartFrog mechanisms to advertise their existence – either because they are not required to be accessed or because they use other mechanisms for this purpose.

The normal model for a SmartFrog system is that a root process compound is created as a service or daemon on each host, possibly at boot time, and that applications are created (in dynamically created sub-processes if so desired) as defined by applications that are launched from basic processes that exit when the application has been successfully deployed.

Process Attributes

Each process, when it is started, customizes its behaviour dependent on attributes that are given it in two distinct ways:

• As attributes defined in the processcompound.sf file that is read at the start-up of any process.

• System properties set on the command line that override the defaults provided in the above start-up file.

These attributes control aspects of process behaviour such as type of process it should be, what port is used for the registry within the system, whether security should be enabled and where remote code is available for remote downloading. Most of the attributes may be defined in either or both the above ways, however a few of the attributes must be passed as system properties. In particular, the security properties need to be set before the processcompound.sf file is read.

When the processcompound.sf file is parsed, all system properties with the prefix org.smartfrog.sfcore.processCompound are added to the context before deployment, thereby creating the process compound itself. Consequently, using the command line option

-Dorg.smartfrog.sfcore.processCompound.sfRootLocatorPort=2000

when starting Java overrides the default value defined in processcompound.sf. The attributes from processcompound.sf that are considered user-modifiable are as follows:

• sfProcessName
  (no default value preset in processcompound.sf)

• sfLivenessDelay 15;

how frequently (in seconds) to check children processes and applications to status

• sfLivenessFactor 5;

how many missed checks from parent process before assuming that parent process is dead

• sfProcessAllow true;

allow sub-processes – only used with the root process compound

• sfProcessTimeout 60;

how long to wait (in seconds) for a child process to be created before failure is assumed

• sfRootLocatorPort 3800;

the registry port for the root process compound

Accessing Process Compounds And Attributes

Attributes may be accessed via the process compound itself. Indeed, the attributes should never be accessed by requesting the system properties - they may not have been defined through that mechanism. To access the attributes, first obtain the process compound of that process then simply lookup the attribute using the normal SmartFrog supplied methods (it is, after all, a Prim) as in:

SFProcess.getProcessCompound()
.sfResolve(Reference.fromString("sfRootLocatorPort"));

Note that for efficiency, using the string reference parser should be limited. Overall, it is better to build and use a reference to avoid the expensive step of parsing the string as a reference.

If the process compound required is on a different host, or is in a different sub-process, the mechanism is to first locate the root process compound on that host, then to find the process of that name as an attribute within the process compound. This can be done using the following invocation

SFProcess.getRootLocator().getRootProcessCompound("hostname");

Or by making use of the host reference

sfResolve(Reference.fromString("HOST hostname"));

Attributes may then be queried as before. Once the root process compound is obtained, sub-processes may be obtained by looking up the process name in the compound as with any other attribute.

Creating And Naming Sub-Processes

Sub-processes are created in two ways. A user may create a sub-process of an existing root process by setting the system property sfProcessName as follows on the command line to start the daemon:

-Dorg.smartfrog.sfcore.ProcessComopund.sfProcessName=name

By doing so, the process registers with the root under the supplied name. If the root does not exist, the process will fail and terminate. Note that if the reserved name “rootProcess” is used, a root process is created.

The second mechanism is via the attribute “sfProcessName” being given in a component to be deployed as in the following snippet

Foo extends Compound {

sfProcessName "fooProcess";

// ...

}

On deployment, the compound Foo will be deployed in a sub-process fooProcess, the host depending on the provision of the attribute sfProcessHost and local if not provided. If fooProcess already exists, this will be used, otherwise a new process will be created using that name.

It should be noted that in automatically creating this subprocess, the command line is defined to pass on all system properties of the root, apart from the org.smartfrog.sfcore.processCompound.sfProcessName property that set to the appropriate name for the process.

Naming Applications

As applications are deployed into the various processes, the process compounds are involved in the creation of the first component of an application tree in that process (this may not be the root component of the application tree, just the first in that process). As it does so, the process compound will keep an attribute referencing this component, the name of the component being either obtained from the component’s sfProcessComponentName attribute if it exists, or a random unique name generated on demand.

This component is not a child of the process compound, though the process compound does monitor it for status using the liveness mechanism, removing it if liveness shows that the component has terminated. However, the provision of this attribute does enable a management system to locate processes (using the registry and named sub-processes) and then to find all applications that are resident completely or partially within that process. By following the hierarchy of parent/child relationships, every component across all processes of every application can be located.

Using the –n command-line parameter of SFSystem command ensures that the root component of the associated configuration is given the name provided on the command line. It is exactly as though application were being given a name by using the sfProcessComponentName in the application itself.

HOST and PROCESS Links

Note that the naming of components in this way also provides a convenient naming service across a SmartFrog system that could be used by components to locate each other – for example, a component may wish to log events at the logging service on a specific host. The normal way in which components locate each other is, of course, by using a link. To support the use of the naming capability, a HOST link is provided which will, as it is resolved, use the root location mechanism to access remote process compounds, allowing the remainder of the link to be de-referenced in that context, to a subprocess compound or an application component. The syntax for HOST references is given in section 2.2.4.

Another link that may be used in a similar way is the PROCESS link that refers to the process compound of process in which the component is deployed.

The SmartFrog Security Model

Introduction

This section describes the SmartFrog security model. Its design deliberately had a very simple usage model in mind, so it may not suit all uses. Future releases of SmartFrog will address more complex scenarios by enhancing some of the current mechanisms, so even if it does not fully meet your requirements, some experimentation may still be worthwhile. In addition, although some penetration testing has been done, security mechanisms are difficult to get right first time, so unwanted "features" may well appear.

Several aspects are covered in this section. Firstly, the threats that are considered in the model are described. Secondly, the policies that must be enforced in order to protect the system from these threats are defined. Thirdly, the specific mechanisms that have been implemented, and the assumptions regarding the rest of the system are listed. Finally, a discussion about the current limitations that will motivate enhancements in future releases.

Threat Model

SmartFrog could be an extremely powerful tool to spread viruses. The ease of deploying and managing the life cycle of a distributed application plays against you when the application deployed is malign. Therefore, it is critical that the system controls who can deploy what and where.

In particular, the communication channels between SmartFrog daemons are not necessarily secure, e.g., they could be Internet connections. An attacker could modify deployment configurations sent from legitimate daemons, or just pretend that he is a valid participant and send his own. In addition, he can obtain critical configuration information, such as passwords, by snooping on the communication, that he can later use to attack the system.

An important feature of SmartFrog is to dynamically load resources from web servers while deploying. These resources could be additional configuration descriptions, Java classes, scripts, executable files and so on. New exploits on web servers appear every month, and the concern is that these could be used to modify resources that SmartFrog will download later on.

Making a node "SmartFrog aware" implies installing a permanent service in this node, opening up a port for incoming requests, creating a special account to run the service, and so on. This can make that platform more vulnerable if someone can compromise the service itself, e.g., by exploiting a buffer overflow.

Security Policy

The ultimate goal is that a distributed application configured, deployed and managed by SmartFrog is not more vulnerable than the same application configured, deployed and managed manually using a local secure procedure. This means that SmartFrog is not trying to "fix" the security problems of the application itself by, e.g., constraining what the application can do. However, the application could indirectly benefit by having a more flexible mechanism to bootstrap its security, or it could even use directly SmartFrog security services.

Another desired goal is not to introduce new vulnerabilities in the hosting node because SmartFrog has been activated. The problem is that the interactions of SmartFrog with the rest of the system make this goal platform dependent. Again, SmartFrog cannot "fix" problems in the underlying OS security; it is simply that SmartFrog must not make them more "visible".

To help clarify the target security model in this release, the concept of a SmartFrog Trusted Community (SFTC) has been introduced. A SFTC is a set of principals, typically composed of SmartFrog daemons and administrators that fully trust each other, i.e., they will do anything that another valid member requests, and they do not trust anybody else. There is a single authority that defines who is initially in the community, but a current member could later on add new members based on its own criteria. A member of an SFTC should only use resources that are trusted, where trust in this context means that they were created or authorized by another member of the community, and nobody has modified them since. In some cases, we want to enforce confidentiality as well as integrity on these resources, always within the scope of an SFTC.

As mentioned in the discussion on threats, SmartFrog uses web servers to dynamically load resources. Web servers are not part of the SFTC, although they host resources that are trusted by members of the SFTC. Mechanisms described in the next section will enforce the integrity (but currently not the confidentiality) of these resources.

There is a strong requirement for knowing what is coming from inside or outside the community, but it is less valuable to know from whom within the community the original request came from. This means that accountability of individual members in the community is very limited, something not surprising when members fully delegate each other actions, and there are no members with "special" privileges. Nevertheless, SmartFrog still give different identities to each principal in an SFTC to make future enhancements easier.

Special care should be taken before authorizing that a particular application might be deployed in the platform. Deployed components share the same environment and privileges of the SmartFrog infrastructure and they could take control of it. The problem is not only deploying a virus, but also deploying an application with an exploitable vulnerability, that will allow an attacker to get control of the application, then the SmartFrog local daemon, and finally all the nodes in the SFTC. In general, ensure that daemons have the minimal privileges required to carry out the tasks required by the applications.

Security Mechanisms

In this section, the security mechanisms that have been implemented for SmartFrog to support SFTCs are described. This implies knowing when a principal is a valid member of the community, ensuring that interactions over an insecure network of valid members are safe, and satisfying the integrity (and authentication) requirements of trusted resources, possibly hosted by non-members of the SFTC. In addition, other mechanisms that have not been implemented as part of SmartFrog, but are assumed to be available to make the model work, are discussed.

Built-in security mechanisms

SmartFrog uses PKI (Public Key Infrastructure) based on X509 certificates to provide principals of the SFTC with credentials that justify they are valid members of the community. The centralized CA (Certificate Authority) is based on openSSL and fully integrated with the rest of the release installation process (by using Ant). Keys for individual members are actually generated using Sun's keytool, and Sun's implementation of a keystore (with random passwords) is used to keep node credentials safe (we assume though that the centralized CA is in a safe, isolated environment).

Java 2 security mechanisms are leveraged to create a SFTC. In particular, the Java security policy is set so that, when enforced by a SecurityManager, gives full privileges to classes loaded from signed jars (using a trusted key for the SFTC), and none otherwise. These mechanisms are extended to other resources apart from classes, such as configuration descriptions, so that they are only loaded if they are in a signed jar (same signing key as before). This is particularly useful when the jars are hosted by web servers that are not part of the SFTC. Jars containing classes and configurations of this release are signed as part of the installation process. Jars with your classes/configurations could be easily signed with a similar process (i.e., using Ant).

RMI calls are tunnelled over SSL using the JSSE API and Sun's reference implementation. SmartFrog forces mutual authentication in the SSL sessions based on the 1024 bits RSA public/private keys that are discussed above. Only the SFTC CA keys are part of our trust assumptions so, by validating the other partner's X509 certificate chain, SmartFrog knows that it belongs to the community. However, the authenticated credentials of the other session peer are currently propagated to the application layer, but this will only be useful in future releases. Current SSL session settings include triple DES encryption, with HMAC SHA-1 for message authentication. In addition, a similar mechanism is used to protect access to the RMI registry.

SmartFrog supports dynamic loading of stubs during RMI calls, or arbitrary resources in signed jar files using an enhanced RMI class loader. In both cases the loading sources is restricted to a configured codebase, and use the Java 2 mechanisms for signed JAR files described above.

All SmartFrog core classes use the security hooks described above for loading resources and communicating with other peers. In addition, the set-up of the security mechanisms is done as early as possible in the initialization of daemons to minimize exposure.

Assumptions

The process of setting up the critical components of SmartFrog is safe. For example, the Zip file containing the release has not been tampered with or, if keys are centrally created, their confidentiality has not been compromised when they were ship to the target host.

Similarly, SmartFrog critical components are protected by the underlying OS after they have been set-up. For example, everything under the directory "private/" can only be read/modified by the authorized user account that is using SmartFrog (the security credentials are in that directory). Moreover, an attacker cannot modify the SF core classes, basic scripts, or the JVM itself. In general, SmartFrog is not trying to solve vulnerabilities of the platform, just hoping that things do not get worse.

The current SmartFrog infrastructure is only as secure as the applications that it deploys, unless significant application customization is done (e.g., spawn processes under a different user with less privileges).

The implementation of the security mechanisms in SmartFrog uses many third-party packages to quickly implement the security mechanisms. This implies that if vulnerabilities are found in one of these components, e.g., Sun's secure random number generator, this will have a devastating effect on the implementation.

Known limitations and future enhancements

Some of the security assumptions will be relaxed in future releases of SmartFrog. The next logical enhancement is to isolate the SmartFrog infrastructure from the applications that it deploys and manages. Since restrictions should not be placed on what SmartFrog can deploy, OS support will be required for that.

Once isolation can be guaranteed, a common platform can be used to deploy applications for non-mutually trusted clients. This will require virtualizing the deployment service so that clients just can view and interact with their own applications.

The next step will be deploying applications that spawn multiple domains of trust, i.e., multiple SFTC, in which a non-hierarchical trust model and more accountability is required. This federated model, together with a new revocation mechanism, will limit the exposure when the security credentials of a valid member are compromised.

Finally, denial of service attacks are not prevented, and some of third party packages that are used are particularly exposed to these attacks. Re-engineering SmartFrog to be more robust against these attacks is a significant effort.

Properties

SmartFrog may have a number of properties defined that alter the behaviour of the daemon. Some of these are required to be defined on the command line (typically those that affect the way Java itself works), others may be defined in a file (often default.ini) and passed to the daemon on the command line.

These properties are captured in the class:

org.smartfrog.sfcore.common.SmartFrogCoreProperty

Some interesting properties are:

• org.smartfrog.sfcore.common.Logger.logStackTrace: Optional boolean property to include stack trace with error message. Default=false.

• org.smartfrog.sfcore.processcompound.sfRootLocatorPort: SmartFrog daemon connection port. Default=3800.

• org.smartfrog.sfcore.processcompound.sfLivenessDelay: Liveness check period (in seconds). Default=15.

• org.smartfrog.sfcore.processcompound.sfLivenessFactor. Liveness check retries. Default = 5.

• org.smartfrog.sfcore.processcompound.sfProcessAllow: Allow spawning of subprocess. Default=true.

• org.smartfrog.sfcore.processcompound.sfProcessTimeout: Subprocess creation/failure timeout. Default=60. Slower machines might need longer periods to start a new subprocess.

Properties that can only be defined using Java -D command line:

org.smartfrog.sfcore.processcompound.sfProcessName: A user may create a sub-process of an existing root process by setting this system property. To start a SmartFrog daemon this property has to be set to “rootProcess”.

org.smartfrog.iniFile: to load a file with properties to define JVM system properties. This file may not contain properties that affect security or code loading – it is read after these are already initialized.

org.smartfrog.sfcore.processcompound.sfDefault.REGISTRATION_NAME: This property is used to describe some descriptions that should be deployed in all daemons and sub-processes. Each description will be registered in the process using REGISTRATION_NAME as name. It is possible to have many of this properties but the REGISTRATION_NAME cannot be repeated.

org.smartfrog.codebase: Property used to define the codebase, a list of space separated URLs of JAR files used by the daemon.

org.smartfrog.sfcore.security.keyStoreName: to load the private keys for the daemon.

org.smartfrog.sfcore.security.propFile= to load security properties.

org.smartfrog.sfcore.security.debug: to enable security debug information. Default=false;

java.security.policy: Property to define Java security properties.

java.security.debug: Property to define debug level in Java security.

java.rmi.server.logCalls: Property to enable RMI logging. Default=false;

sun.rmi.loader.logLevel: Property to define debug RMI level. Example: .VERBOSE

java.security.manager: Property to define the Java security manager.

Example

This section creates a simple example, covering all aspects including the creation of the SmartFrog description and the necessary Java classes.

The example is a variant on the normal “hello world” that is so favoured by books on programming. There will be two components – a printer and a message generator. The message generator will locate the printer, send a series of messages to it, and terminate. The printer, on the other hand, will wait until it is given a message, and then print it. It will terminate when told to do so.

The example is supplied with the SmartFrog distribution and is in package org.smartfrog.examples.helloworld.

For each of the two components, two aspects must be considered:

• How the description for the component is defined, what attributes must be provided and what the default attribute values are

• How the implementation is to be written, what is to be done in the various lifecycle phases and what threading is required

Following the definition of the components, consideration must be given as to how they are to be combined into a single application.

The Printer

The printer is a very simple component. It offers a single method that must be accessible via RMI, namely printIt, which must take a string and print it on the standard output. It has no specific initialization requirements, no threading and no specific need to clean up on termination.

To make it a little more interesting, the printer will prefix the output message with an identifier - either the name attribute if it is provided, or the name the component has in the application tree if it is not.

The Description – file “printer.sf”

The description needs only to link the component type to the implementing class. It does not need to provide a default value for the name of the printer, since the use of no name will cause a default to the name within the component tree.

Printer extends Prim {

sfClass "org.smartfrog.examples.helloworld.PrinterImpl";

name ""; //name – if empty, uses sfCompleteName

}

This fully describes the basic printer component. Host bindings will be done as part of the complete application description.

The Interface – file “Printer.java”

Since the printer has an RMI accessible method, that method must be described in an interface that extends Remote, and the method must throw the appropriate RemoteException. The interface is very simple:

package org.smartfrog.examples.helloworld;

import java.rmi.*;

public interface Printer extends Remote {

public void printIt(String message) throws RemoteException;

}

This interface will be used by RMIC to generate the appropriate stub for the PrinterImpl class enabling other components to remotely call this method.

The Implementation – file “PrinterImpl.java”

The PrinterImpl class is a primitive component and so must follow the outline template provided in section . In addition, it must implement the Printer interface.

Note that since there are no specific start or termination actions to carry out, there is no need to provide the sfStart or sfTerminateWith methods.

package org.smartfrog.examples.helloworld;

import org.smartfrog.sfcore.prim.*;

import java.rmi.*;

import org.smartfrog.sfcore.Reference.*;

import org.smartfrog.sfcore.prim.*;

public class PrinterImpl extends PrimImpl implements Prim, Printer {

/* any component specific declarations */

// my name

String name = "";

// reference to my name attribute

Reference nameRef = new Reference(ReferencePart.here("name"));

// standard remotable constructor – must be provided

public PrinterImpl() throws RemoteException {

}

// initialization template method

public synchronized void sfDeploy()

throws RemoteException, SmartFrogException {

super.sfDeploy();

/* any component specific initialization code */

name = sfResolve(nameRef, "", false);

if (name.equals("") name = sfCompleteName().toString();

}

/* component specific methods */

public void printIt(String message) throws RemoteException {

system.out.println(name + ": " + message);

}

}

The Generator

The generator is a slightly more complex component. In particular, it must locate the printer, and call it with a sequence set of messages. This must be done in a separate thread started during the sfStart lifecycle method. The thread must be terminated during the termination template method if it has not already done so.

In addition to locating the printer, the component must obtain its set of messages and a frequency with which it should print these messages (defaulting to, say, one every 10 seconds).

The Description – file “generator.sf”

The description needs to link to the component type to the implementing class and define the other attributes with their default values. The link to the printer component, which will be LAZY because the RMI reference is required, cannot be provided until the whole application is created.

Generator extends Prim {

sfClass "org.smartfrog.examples.helloworld.GeneratorImpl";

frequency 10; // default value set to 10 seconds

messasges []; // a vector of messages

printer; // link to the printer component

}

The Implementation – file “GeneratorImpl.java”

Unlike the printer, the generator has no externally accessed methods apart from those related to being a primitive component. Consequently, no additional interface needs to be defined. The generator, though, requires a thread – this may be done in many ways but extending Runnable and creating a thread from itself is the easiest.

package org.smartfrog.examples.helloworld;

import org.smartfrog.sfcore.Reference.*;

import org.smartfrog.sfcore.prim.*;

import java.util.*;

import java.rmi.*;

public class GeneratorImpl extends PrimImpl implements Prim,Runnable {

/* any component specific declarations */

Reference messagesRef = new Reference(

ReferencePart.here("messages"));

Reference printerRef = new Reference(

ReferencePart.here("printer"));

Reference frequencyRef = new Reference(

ReferencePart.here("frequency"));

Printer printer;

Vector messages;

int frequency;

Thread sender;

boolean terminated = false; // notify the thread to terminate

public GeneratorImpl() throws RemoteException {

}

public synchronized void sfDeploy()

throws RemoteException, SmartFrogException {

super.sfDeploy();

printer = (Printer) sfResolve(printerRef, true);

messages = sfResolve(messagesRef), [], true);

frequency = sfResolve(frequencyRef,0,true)* 1000;

}

public synchronized void sfStart()

throws RemoteException, SmartFrogException {

super.sfStart();

// create and start the thread

sender = new Thread(this);

sender.start();

}

public synchronized void sfTerminateWith(TerminationRecord tr) {

// terminate the thread nicely if needed, can wait for it to

// awake to actually do so

terminated = true;

super.sfTerminateWith(tr);

}

public void run() {

// the body of the thread

try {

for (Enumeration en = messages.elements();

en.hasMoreElements() & !terminated; ) {

printer.printIt(en.nextElement().toString());

if (frequency > 0) {

Thread.sleep (frequency)

}

}

} finally {

// it doesn't matter calling this many times

sfTerminate(TerminationRecord.normal(null));

}

}

}

Compiling the Components

The two classes and the interface all need to be compiled using the javac compiler. This has been done in the distribution and the classes are in the sfExamples.jar file.

Further, the two classes must be prepared for RMI by creating and compiling the stubs and skeletons. This is done using the rmic compiler that is invoked on the class:

rmic org.smartfrog.examples.helloworld.PrinterImpl

rmic org.smartfrog.examples.helloworld.GeneratorImpl

Both the classes must be rmic’ed because although the generator has no interface as part of its own specialized behaviour, it does implement the Prim interface which may be access remotely – for example in the case of remote deployment.

If either of the classes will never be remotely accessed – for example if the entire application will always be run in a single JVM – the rmic steps may be skipped. In this case, the attribute sfExport must be set to false in the appropriate component descriptions. If this is not done, the SmartFrog system will attempt to load the stubs and skeletons for the classes and make the objects accessible remotely.

Again, the stubs and skeletons have been pre-compiled and are in the jar file.

The Combined Application

Now that the various parts are defined and compiled it is possible to define a few applications, consisting of one or more generators and printers, deployed on one or more hosts, as required.

Starting with the simplest, one of each on the same host:

#include "org/smartfrog/components.sf"

#include "org/smartfrog/examples/helloworld/printer.sf"

#include "org/smartfrog/examples/helloworld/generator.sf"

// the application must be called sfConfig

// a compound is a collection of components

sfConfig extends Compound {

g extends Generator {

messages ["hello", "world"];

printer LAZY ATTRIB p; // link to the instance of the printer

}

p extends Printer {

name "myPrinter";

}

}

Note that the printer is given a name rather than using the name derived from the tree (which would be “p” as the name is from, but not including, “sfConfig”). The generator uses the default frequency of 10.

The next example shows that it’s possible to have more than one generator using the same printer:

#include "org/smartfrog/components.sf"

#include "org/smartfrog/examples/helloworld/printer.sf"

#include "org/smartfrog/examples/helloworld/generator.sf"

// the application must be called sfConfig

// a compound is a collection of components

sfConfig extends Compound {

g1 extends Generator {

messages ["hello", "world"];

// link to the instance of the printer

printer LAZY ATTRIB p;

}

g2 extends Generator {

messages ["hello", "world", "again"];

frequency 5;

// link to the instance of the printer

printer LAZY ATTRIB p;

}

p extends Printer {

name "myPrinter";

}

}

(In this case, there is a problem with termination that will be explained later.)

A more complex example is one where we define a new component that consists of a printer and generator pair. This pair will print to each other, and the application consists of two of these pairs.

#include "org/smartfrog/components.sf"

#include "org/smartfrog/examples/helloworld/printer.sf"

#include "org/smartfrog/examples/helloworld/generator.sf"

// a compound is a collection of components

Pair extends Compound {

g extends Generator {

messages ["hello", "world"];

// link to the instance of the printer

printer LAZY ATTRIB p;

}

p extends Printer;

}

sfConfig extends Compound {

pair1 extends Pair;

pair2 extends Pair;

}

In this case the printer component has not been given a name, so will use the one derived from the tree – pair1:p and pair2:p respectively. Each generator will find the associated printer, even though they are using the same link, because these links are resolved in different contexts.

There is, however, a problem with this example and the previous one. Since the lifecycle of the whole application is connected, the first of the two generators to finish will cause the whole application to terminate even though the other may not have finished yet. This may be solved by the use of the workflow components, but this is not covered in this reference manual.

Enriching the example a little further, the pair should optionally be parameterizable with the set of messages and the frequency, but without requiring the user to know the structure of the components contained within it.

#include "org/smartfrog/components.sf"

#include "org/smartfrog/examples/helloworld/printer.sf"

#include "org/smartfrog/examples/helloworld/generator.sf"

Pair extends Compound {

// default value for param

messages ["this is a", "boring”, "set of strings"];

// ditto

frequency 10;

g extends Generator {

// link to container’s value

messages PARENT:messages;

// link to container’s value

frequency PARENT:frequency;

// link to the instance of the printer

printer LAZY p;

}

p extends Printer;

}

sfConfig extends Compound {

pair1 extends Pair {

messages ["hello", "world”, "again"];

}

pair2 extends Pair {

frequency 5;

}

}

This example shows the use of the parameterization pattern described in section 5.

The final example is one where the hosts on which various components are to be deployed may be modified. Taking the last example, we will enrich the notion of pair with two hosts – the printer host and the generator host, both defaulting to the local host.

In the combination, we will put the printer of one pair and the generator of the other pair on one host, and the reverse on a second host.

#include "org/smartfrog/components.sf"

#include "org/smartfrog/examples/helloworld/printer.sf"

#include "org/smartfrog/examples/helloworld/generator.sf"

Pair extends Compound { // a compound is a collection of components

printerHost "localhost";

generatorHost "localhost";

// default value for parameter messages

messages ["this is a", "boring”, "set of strings"];

// ditto

frequency 10;

g extends Generator {

sfProcessHost generatorHost;

messages PARENT:messages; // link to container’s value

frequency PARENT:frequency; // ditto

printer LAZY p; // link to the instance of the printer

}

p extends Printer {

sfProcessHost printerHost;

}

}

sfConfig extends Compound {

hostA "foo.smartfrog.org";

hostB "bar.smartfrog.org";

pair1 extends Pair {

printerHost hostA;

generatorHost hostB;

messages ["hello", "world”, "again"];

}

pair2 extends Pair {

printerHost hostB;

generatorHost hostA;

frequency 5;

}

}

By modifying the hostA and hostB attributes, different deployments may be obtained.

Exit Codes for SmartFrog Scripts

0 - Success

1 - General Errors

69 - Bad Arguments input to Script

130 - Script terminated by Control-C