Streaming Test Results
The previous post featured the Subunit test framework which supports streaming of test results. The output is expressed in a structured format which can be filtered into other formats such as pyunit for example.
The objective of this post is to explore the requirements on formats to support streaming of test results. The motivation is that the streams produced by diverse test suites could then be filtered into a consistent output which is essential in the context of integration.
Basically, test results are nothing more than structured data. In its simplest form, this consists of representing information in such a way that it can be written and then read in a consistent way. In the widest interpretation of the definition, this can be encountered when defining data types in the source code of a programming language and having it parsed by the corresponding compiler. In a narrower interpretation, test results should be expressed in such a way that they could be easily be understood by a variety of programming languages.
The point is that data should be written in a structured format to be read by other programs rather than human beings. This might seem obvious, but it is worth mentioning explicitly because there are several test suites which don’t apply this principle by default.
The format of data consists of its intermediate representation between the time it is written and later read. XML is a popular example often used in the context of Web Services, where it represents the message format when exchanging data between a requester and a provider. This context is particularly relevant in respect to integration because the requestor and the provider can be written in different programming languages. In theory, integration is seemingly simple because the format has been standardized, so a parser is readily available in most contemporary programming languages. In practice, integration is rarely simple.
In addition to standardized formats, it is also worth mentionning some of the benefits of specialized formats. For example, the Subunit test framework defines its own output format for test results which happens to be trivial to read and write in just about any programming language. Even in a shell script, the output can be parsed using the most basic regular expressions.
So, even though there are benefits to adopting standardized formats, there are also good reasons to use specialized formats. The relevant part is that they should be trivial to parse.
In addition to the raw data, a format can also specify meta information which is essentially “data about the data”. Here are the most common approaches to specifying such information:
- Some data formats do not contain any meta information and simply express all raw data as strings. For example, this is the case for current version of the Stomp protocol. It is therefore the responsibility of the application layer to explicitly coerce everything into the corresponding types and data structures.
- Other formats only encode type information within the data itself. For example, it is possible to consistently parse the YAML format while preserving type information regardless of the programming language. However, it is still the responsibility of the application layer to validate the structure of the data.
- Finally, another approach consists of decoupling the raw data from the meta information. For example, Web Services typically expose an XML schema which can define extensive information about the data such as type and structure. So, when the application layer finally receives the data, it can reasonably assume that the data has been validated for most intents and purposes.
A priori, it would seem that encoding type information within the data might be desirable. However, if the structure of the data still needs to be validated by the application layer, this actually provide little gain. In other words, there is little effort in validating the type of the data if the structure has to be validated anyways.
The IEEE defines interoperability as “the ability of two or more systems or components to exchange information and to use the information that has been exchanged.” In the context where the components consist of test suites and the information exchanged consist of test results, the implication is that the output should adhere to a common information exchange reference model. In other words: what is sent should the same as what is understood.
The most obvious approach would require that all test suites output the same data format as defined by the integration framework. This does not necessarily imply that this format is specific to a particular programming language. However, it does imply that every component must adhere to the same format.
Another approach would lower the requirements further to the semantics of the information rather than the format. This could be accomplished by supporting multiple formats which could be converted between each other without losing any meaning. This implies that if type information is part of the semantics, a format such as YAML could not be converted to Stomp which expresses everything as strings. However, the other way around would be possible assuming that the YAML filter understood how to cast the information contained in the Stomp data.
The problem with supporting multiple formats is that this could result in an explosion of filters. To convert data between each format would require implementing a filter for each permutation, this is a many to many relationship. However, to convert from a single format to and from each other format, this becomes a one to many relationship.
In the most general sense, streaming refers to making a sequence of data elements available over time. In this particular context, the data elements consist of test results and they should be made available in a pipeline on the command line.
- Some data formats already support streaming messages on a pipeline. For example, the Stomp protocol was designed in the context of messaging systems where multiple messages are frequently exchanged between the sender and the receiver. The protocol clearly defines a separator to indicate the end of each message which can be demultiplexed by the receiver.
- Other data formats were designed to serialize individual objects rather than multiple objects in a stream. For example, JSON expects to interpret a complete message all at once by eval’ing the content. So, there is essentially no concept of a separator to identify the beginning or end of individual messages within a stream.
The solution for these other data formats is to encapsulate individual messages into a container format which actually does support streaming. For example, a JSON message could be encapsulated into the body of a Stomp message which could then be streamed.
This post has only identified the single requirement that test results should be structured to be understood by other programs. The rest mostly consisted of considerations for which there are known solutions with varying impact. However, none of these considerations seem to be absolute blockers.