DIRECT: Applying the DIKW Hierarchy to
Large-Scale Evaluation Campaigns
Information Management Systems (IMS) Research Group
Department of Information Engineering
University of Padova
We describe the effort of designing and developing a digital library system able to manage the different types of information resources produced during a large-scale evaluation campaign and to support the different stages of it. In this context, we present Distributed Information Retrieval Evaluation Campaign Tool (DIRECT), the system which has been adopted to manage the CLEF evaluation campaigns since 2005.
Scientific data, their management, access and reuse through citation, curation, enrichment, and preservation are essential components of scientific research. We consider Information Retrieval (IR) experimental evaluation as a source of valuable scientific data and we have proposed a data curation approach in order to better manage, preserve, interpret, and enrich the scientific data produced . The result of our work is Distributed Information Retrieval Evaluation Campaign Tool (DIRECT), a Digital Library System (DLS) which has been developed, adopted and tested in the Cross-Language Evaluation Forum (CLEF) campaigns since 2005 [2, 3]. In designing the system, we have adopted the Data, Information, Knowledge, Wisdom (DIKW) hierarchy  in order to structure the information space involved in an evaluation campaign and to model the interaction of the different actors with the created information resources. We provided a demontration of the functioning of the DIRECT system and of the different tasks that can be accomplished with it.
Architecture of the User Interface
According to the investigation of user requirements and needs, the DIRECT user interface is designed to meet the following goals:
- to be cross-platform and easily deployable to end users;
- to be as modular as possible, clearly separating the application logic from the interface logic;
- to be intuitive and capable of providing support for the various user tasks described in the previous section, such as experiment submission, consultation of metrics and plots about experiment performances, relevance assessment, and so on;
- to support different types of users, i.e. participants, assessors, organizers, and visitors, who need to have access to different kinds of features and capabilities;
- to support internationalization and localization: the application needs to be able to adapt to the language of the user and his country or culturally-dependent data, such as dates and currencies.
The modularity of the components has enormous benefits when building interactive applications, since it helps the designer to better understand and develop each component and modify it without affecting the others. Therefore, we used the Model-View-Controller (MVC) approach as provided by the Apache STRUTS framework to clearly separate the following three layers:
- model layer: contains the underlying data structures of the application and keeps the state of the application;
- view layer: the way the model is presented to the user;
- controller layer: manages the interaction between the view and the input devices, such as the keyboard or the mouse, and updates the model accordingly.
For a larger view of Figure 1, click here. After the image opens, click on it to zoom.
DIKW Levels and User Interface Support
For a larger view of Figure 2, click here,
Presenting the demo of DIRECT, we can use Figure 2 as a map to present some of the interfaces developed at each level of the hierarchy. We highlight:
- at the data level, the interface developed to manage the experiment submission by the participants, and how a logical structure is suggested by the system to support the user's actions;
- at the information level, the interface enabling assessors to do relevance assessments, and how information can be created by the user over existing data;
- at the knowledge level, how the system helps the creation of relation between different information on the visitor's metrics and statistics page, and some plan for future investigations to allow the user to create his own scenario of knowledge;
- finally, at the wisdom level, we point out some possible future directions that suggest how the DIRECT system may support scientific production.
For a larger view of Figure 3, click here
The work reported has been partially supported by the TrebleCLEF Coordination Action, as part of the Seventh Framework Programme of the European Commission, Theme ICT-1-4-1 Digital libraries and technology-enhanced learning (Contract 215231).
 M. Agosti, G. M. Di Nunzio, and N. Ferro. The Importance of Scientific Data Curation for Evaluation Campaigns. In Proc. 1st Int. DELOS Conference, pages 157-166. LNCS 4877, Springer, 2007.
 G. M. Di Nunzio and N. Ferro. DIRECT: a System for Evaluating Information Access Components of Digital Libraries. In Proc. ECDL 2005, pages 483-484. LNCS 3652, Springer, 2005.
 M. Dussin and N. Ferro. Design of the User Interface of a Scientific Digital Library System for Large-Scale Evaluation Campaigns. In Second DELOS Conference - Working Notes. ISTI-CNR, 2007.