EXPERIMonitor – Experiment Content
Experiment content is produced and consumed during experimentation with and testing of multimedia systems. Such content provides insight into the structure, behaviour and performance of systems under test including user experience. System configuration, dependency graphs, input/output data sets, testing procedures, Quality of Service/Quality of Experience metric models and associated measurement sets all characterise experiment content.
EXPERIMonitor is a software tool focused on the management of experiment content that allows developers to explore the relationship between QoS and QoE in complex distributed multimedia systems. The tool is specifically designed to support the observation of systems where user-centricity, mobility, ad hoc participation and real-time access to information are critical to success.
EXPERIMonitor uses a hybrid data model that combines formal low level metric reporting with semantic provenance information. The hybrid approach provides the ability to collect large quantities of measurement data (e.g. service response times, network latency, user satisfaction, etc) whilst allowing for exploration of causation between observations within such data (e.g. user satisfaction in relation to service response time).
Using an extensible metric model, developers can define QoS and QoE metric models for parameters of interest in specific trial. QoE is provided by “in-application” or post trial online questionnaires. QoS data is reported by instrumented applications, services and infrastructures. In both cases EXPERIMonitor provides client APIs (Java, C++, Android, Ruby and IoS) to report measurements. The semantic provenance model is based on the W3C PROV standard allowing for reporting of interaction and activities between users, applications and services.
EXPERIMonitor offers a dashboard that supports real-time observation and historical data exploration of data sets. A system and data exploration view is provided. The system view is important for verifying setup, execution and tear-down phases of experimentation by providing access to status information about connected clients and metrics being reported. Data exploration is the interface used to derive insights from results data. Data exploration focuses on the participants, activities and interactions with application and services within the system under test. Developers can explore the information space from the perspective of a user or service depending on their interests. For example, if a developer observes degradation in service performance (QoS) within a time period they can automatically find all of the users (QoE) affected by the event. Alternatively, if a set of participants report poor levels of satisfaction (QoE), EXPERIMonitor can provide a full list of QoS contributing to their experience.
The ability to efficiently traverse experiment content between QoS and QoE is an essential capability for evaluation of complex socio-technical systems. Typically, the exploration of data cannot provide direct evidence of causation but can provide an indication of factors that influence each other which can lead to further investigation and analysis.