• Synopsis
• Web Service
• Project Outline
• People
• Related Publications

The possibility to reproduce and compare results of other researchers is essential for scientific progress. In the field of information retrieval, however, it is often impossible to specify the complete experiment setting; e.g., in the scope of a scientific publication. As a consequence, a  reliable comparison becomes difficult, if not even impossible. TIRA is our approach to address this shortcoming.

TIRA—short for "Testbed for Information Retrieval Algorithms"—provides a means to embed experiments directly into a web page. The developer of an experiment (the author of a paper) uses TIRA to publish an experimentation interface on the web, where other researchers can play around with the author's original experiment setting. The following video (1'06) illustrates the need for experiment reproduction and result sharing in an entertaining manner.

Visit the TIRA Web service.

The TIRA platform consists of several web services that are delivered as a single web application. The web services are integrated in a ressource-oriented way, following the REST paradigm. All data is represented as XML. A graphical user interface allows users to configure, execute, save, and inspect experiments in an intuitive manner. As shown in the figure below, experiment storage is provided by eXist, a native XML database which offers a RESTful HTTP interface. Experiment executions are distributed among the nodes of a computer cluster. These nodes can be accessed via RESTful HTTP as well. The worker nodes take advantage of AiTools, a toolkit that is developed at the Web Technology and Information Systems Group.

TIRA introduces an abstraction layer for experiment construction and configuration. This layer allows users to model the possible experiment variants as an extensible graph composed of XML fragments (see figure below, representative excerpt). This graph can be accessed via the user interface, and the experimenter can intuitively explore the space of experiment variants.

Implementation  The most salient property of TIRA is that it communicates its experimentation interface over the web. Thus, its system architecture comprises the classical components of a web service. These are a web browser that runs the TIRA JavaScript and renders the user interface, a web server that provides static content, and a service provider that we refer to as the TIRA handler. The TIRA handler delegates requests either to the experiment retrieval component that hosts all executed experiments in a database, or the experiment execution component that manages the execution of experiments over the command line of the underlying operating system. Below, a sequence diagram is given that illustrates the interaction of the TIRA components in the course of the configuration, execution, and evaluation phase of an experimentation session.

Configuration Phase  The interaction process is initiated by a request to a web page that includes the TIRA JavaScript, together with an anchor element containing the URL of a TIRA experiment description. The experiment description is an XML file that contains all the parameters of an experiment, an element for results that may be empty, as well as the command to execute the experiment. When the web browser interprets the TIRA JavaScript, it requests the experiment description from the web server, and replaces the anchor element of the web page with the experimentation interface, containing the parameters of the experiment as text input fields and drop down boxes. The snippet of an example configuration is shown below. Whenever the user selects a parameter value that requires further specification, the associated parameter record is loaded from the web server and added to the experiment description (see (1) in the sequence diagram above). It is possible to specify multiple values for a parameter, either by entering a comma separated list of values into text input fields, or by clicking on the plus button next to drop down boxes to duplicate the respective parameter.

Execution Phase  Once all parameters of the experiment have been specified, the user can request the execution of the experiment. When the TIRA handler receives a request for execution, it proceeds in the following way: If the received experiment description has not been processed before, a unique URL is created for it and inserted into the description as an attribute. The description is handed over to and saved by the experiment retrieval component. If an identical experiment description has already been processed, the version stored by the experiment retrieval component, which contains all available results and the unique URL, is retrieved (see (2) in the sequence diagram above). If the experiment description includes multiple values for some parameters, it is decomposed into all possible experiment descriptions that contain only single values for each parameter. For every generated experiment description, a reference is added to the result element of the multi-valued description, and a new request for execution is raised, which is then processed canonically. If the experiment description received by the TIRA handler is single-valued, it is handed over to the experiment execution component, which executes the experiment with the given configuration on the command line---precisely, for variants of the Unix operating system, on POSIX level (see (3) in the sequence diagram above). Once the results for an experiment are computed, the experiment program registers them in the associated experiment description hosted by the experiment retrieval component. Notice that generally, the experiment retrieval component does not need to operate on the web server that handles the request. A cluster of web servers can be set up, where one of the servers hosts a shared retrieval component that all other instances use to synchronize a distributed execution of experiments. During the execution, the web client requests and displays the execution progress asynchronously (see (4) in the sequence diagram above).

Evaluation Phase  When the first results for an experiment are available, the experimentation interface displays them in one of the available result viewers. Currently, three result viewers are provided: a table viewer that organizes the results in a table, a chart viewer to plot result values in a diagram, and an export viewer that outputs the results as comma separated values.

• Tim Gollub
• Benno Stein
• Steven Burrows
• Dennis Hoppe
• Sven Meyer zu Eissen

Students: Clement Welsch, Arnd Oberländer, Johannes Kiesel