Building a Web-Based Information System to Evaluate IT-Artefacts in Online Experiments - The Case of Intentional Forgetting

Riehle Dennis M, Fleischer Stefan, Becker Jörg

Zusammenfassung

Online experiments are a common research method across a variety of academic disciplines. For example, online questionnaires are used in social sciences to collect empirical data from a large number of participants. In business information systems, questionnaires are, for instance, used to evaluate an IT artefact as part of a design science research cycle. Consequently, there are numerous application systems available, some also for free and/or as open source products, which enable researchers to create, design and conduct online questionnaires. Some tools may even support the statistical analysis of participants' replies.

However, traditional questionnaires are based on a question-answer principle. A question or other mostly static content is shown to the user on their screen and the answer must be provided as text inputs, options to be chosen or similar methods known from web forms. While this is sufficient for many use cases, it may become problematic when evaluating an IT artefact, more precisely, a prototype of a software. In Information Systems, a prototypical software may be the outcome of a design science-oriented research approach, which requires evaluation from both a user and a technical perspective. While a questionnaire for sure provides insights on the perceived functionality of a user, technical aspects of the software are hard to gasp. This problem increases, if the actual functionality starts to differ from the perceived functionality users experience. For example, users may think that they solved a task adequately, while in reality they did not as they failed to recognize all technical functionalities provided by the software.

We have developed a web-based information system which integrates both a web-based prototype with a questionnaire system. This allows us to present the prototype on screen to the user, while simultaneously instructing the user to solve a task and collect responses with traditional questionnaire form elements. To make our information system applicable to different use cases, the used prototype can be configured. Therefore, different prototypes can be evaluated in one integrated platform, as long as they are (1) web-based and (2) implement certain interfaces provided by our platform. We demonstrate our information system with the case of intentional forgetting, where we continuously develop new versions of a prototype and where each version is evaluated in the same platform.

Schlüsselwörter
Evaluation; Online Experiments; Versioning; Intentional Forgetting