If digital libraries are to achieve their full potential, they need to be usable and used by people for whom information retrieval is not generally the main goal. In this paper, we outline various views of "usability" and how they apply specifically to digital libraries. There are great challenges to integrating user perspectives with technical developments, in terms of understanding those user perspectives, developing design processes that adequately accommodate them and ensuring adequate communications between all stakeholders in design. We discuss these issues.
Millions of dollars (euros, pounds, yen, etc.) are being invested in the provision of digital information resources that could be classified as digital libraries. The investment is in various directions, including digitisation projects, technical developments and formulation of necessary standards. The assumption presumably is that this investment is making documents readily and usefully available to all appropriate user communities. However, this assumption needs to be challenged. As one librarian at a recent workshop commented, "'build it and they'll come' is a lie". Not only may the intended users not "come" to the digital library; even if they do, they may well leave unsatisfied. The question of how users may have satisfying and productive experiences working with digital libraries is the focus for this paper.
The starting point for this paper involved work reported at a workshop on usability at JCDL'02 (Blandford and Buchanan, 2002), and a follow-up panel at ECDL'02 (Alexander, Anderson and Blandford, 2002), in which many of the same issues were further explored. The motivations for organizing a workshop on "Usability of Digital Libraries" at JCDL'02 were 1) a concern that digital libraries will only realise their potential when they are usable, useful and used by a broad cross-section of users, and 2) a desire to bring together a community of researchers and practitioners to share insights and experiences of addressing this challenge.
Digital libraries are potentially powerful tools, but their potential will be realized only if users are able to harness that power and take it in fruitful directions. As with all new technologies, early adopters will persevere with using systems almost regardless of the costs and benefits. However, to have an impact commensurate with current investment, it is essential that there be take-up by a much broader community of potential users.
We start from the position that digital libraries are required to enable users to access the information they need to perform their tasks effectively. Put another way: digital libraries need to be both useful and usable.
The term "useful" is generally taken to mean "supporting the required functionality"; in the case of digital libraries, the obvious use is making digital documents available to the appropriate user groups at the time they are needed and in appropriate formats. In practice, users of digital libraries include not just the end users of documents but also librarians and others responsible for the development and maintenance of digital collections, whose needs are centred on collection maintenance. Arguably, if digital libraries are not useful, there is little need to worry about their being usable; however, the converse also holds: if a library is not usable then it cannot be truly useful.
At its simplest, the term "usable" means "can be used". Within Human-Computer Interaction (HCI), "usable" can mean a range of things, including:
- How efficiently and effectively users can achieve their goals with a system (for which it may be possible to apply performance measures);
- How easily users can learn to use the system ("learnability");
- How well the system helps the user avoid making errors, or recover from errors;
- How much users enjoy working with the system the quality of the user experience or whether they find it frustrating; and
- How well the system fits within the context in which it is used.
We can consider how these different usability criteria apply specifically to the design and deployment of digital libraries.
Usability criteria for digital libraries
As yet there is no consensus on what the key criteria are for evaluating the usability of digital libraries. The challenge is made more complex by the variety of user types for libraries. For example, some users are the end users of information, finding documents that address their immediate information needs within the context of the task currently being performed, while others are expert intermediaries, performing a document search on behalf of someone else. Digital library users have varying levels of expertise, both in information seeking and in the topic on which information is sought. Also, the tasks of collection maintenance have very different requirements to those of document retrieval and management (from an individual's perspective). In this paper, we focus on the end user's perspective (i.e., the reader's rather than the librarian's perspective).
Usability performance measures
Within the broader Information Retrieval (IR) community, two particular performance measures have been used to assess the quality of retrieval algorithms: precision and recall. While the quality of retrieved documents is a key criterion for successful use of a digital library, it is not the only important performance indicator. Indeed, it may not be the best, from a user's perspective. Preliminary studies (Blandford, Stelmaszewska and Bryan-Kinns, 2001) suggest that users may not want to receive all the relevant documents on a topic after all, they probably will not have time to read them all but users want to receive a manageable number that can be easily distinguished in terms of the documents' content. While results so far are not conclusive, and it is likely that what is considered a "good" results set will vary with the task and with user expertise, it is already clear that technical metrics do not tell the whole story and that other criteria of quality need to be identified.
Other traditional HCI performance measures, such as average time to achieve goal or mean number of keystrokes, are likely to be difficult to apply meaningfully in digital libraries since time is highly dependent on network performance and many tasks depend on quality of fit between users' search terms and the terms used within the library, rather than on number of keystrokes. However, keystroke efficiency can be used meaningfully for some browse-based tasks.
Overall, there is potential for identifying further key performance indicators for usability, but the indicators will never tell the whole story about the quality of a digital library from the user's perspective.
Surprisingly little work has been done on how users learn to work with digital libraries, and consequently our understanding of how expertise is developed, and of how digital libraries and their interfaces can be better designed to support learning, is weak. A few studies (e.g., Vakkari, 2001; Hsieh-Yee, 1993) have compared novices' and experts' information seeking, but without discussion of the design implications of the findings. Informally, novice end users have been found to struggle with learning how to use digital libraries. Libraries demand more sophistication of query formulation than web search engines such as Google (http://www.google.com); skills acquired in one library environment are often not easily transferred to another, and interfaces are currently changing rapidly, such that learning can become largely obsolete within a few months. Even professional librarians can find it difficult to keep skills up-to-date.
Preliminary results suggest that end users treat the library system as a tool, not as an object of study. Winograd and Flores (1986) extend a discussion by Heidegger on the role of a hammer: skilled users of this tool are largely unaware of it while they are working, until something goes wrong (e.g., the nail bends, or they hit their thumb), at which point their focused engagement with the task is replaced by a focus on the tool and their use of it. In practice, users of digital libraries are not yet able to use them without focusing on the library itself. In our studies, novices have tried to focus on their information tasks, but their attention is persistently drawn to the use of the library itself. In contrast, expert intermediaries searching for information on behalf of end users have been found to engage directly with the library, exploring the effects of manipulating search terms on the number and kind of results returned, learning about particular libraries with respect to the current query.
There is clearly much further work to be done on how people learn to search effectively for information, and to use digital libraries in that search and hence on how libraries can be better designed to support learning.
Errors and error recovery
Arguably, errors are less a problem for digital libraries than they are for systems in which the user is changing data (often irretrievably). However, many user difficulties can be attributed to their making mistakes that they do not even recognise as such and, thus, failing to make progress in their interactions. The quality of feedback from system to user is very often insufficient to ensure users are aware of their actions (erroneous or otherwise sub-optimal) or how their performance could be improved. More critically, while successful searches are often completed quickly, failures are usually protracted, as users persist in trying to find information that either is not available or is not easily accessed. While these incidents are not classic "errors", they result in a poor user experience.
Even within mainstream HCI, work on user experience has only come to the fore very recently, addressing issues of user pleasure, perceived threats to security and privacy, sense of frustration, etc. Landauer (1995) discusses user experience in terms of competence the user's sense of increasing capability in using systems, which means that users can often gain satisfaction from learning to use inherently poorly designed systems and effectance the user's sense of satisfaction at having achieved an interesting effect.
In the context of digital libraries, competence is more often experienced as a negative as a sense of frustration. At the JCDL 2002 workshop, many participants raised this as a key aspect of user experience. At the subsequent ECDL 2002 panel, Wilma Alexander expressed this from the perspective of a professional librarian, drawing her inspiration from Kipling in structuring her observation of the kinds of questions she gets from users:
- What is this site for?
- Why do I have to login?
- When did they move the website?
- Where is the search button?
- How do I get fewer answers?
- Who is responsible for this site?
In terms of effectance, a good user experience does not seem to depend on always finding the perfect document; but it does require finding new, interesting, and possibly surprising material. The importance of serendipity in search should not be underestimated, although it is difficult to design explicitly to support it. Conversely, failure to find relevant material rapidly leads from frustration to disillusionment to giving up.
Context of use
The final aspect of usability we consider here is the fit of a library within the context of use organisational, physical, and task-oriented. The use of information resources in various contexts e.g., legal work (Kuhlthau and Tama, 2001), journalism (Attfield and Dowell 2003), Humanities research (Bates, 1995) have been studied by the information seeking community. However, relatively little of this work has related findings directly to the design and deployment of digital libraries.
Some of these studies have highlighted the fact that information seeking in context often has a pattern. Initially, users are typically unsure of exactly what their requirements are, and they are therefore unable to clearly assess the relevance of documents retrieved. The information problem is gradually resolved to a point where users can more clearly define search terms and assess document relevance. (It is at this stage, but not earlier, that expert intermediaries can provide effective support for search.) Various authors (e.g., Marchionini (1995), Sutcliffe and Ennis (1998), Kuhlthau (1993)) present models on how users select sources, formulate queries, assess results, etc., but not of the broader information task. Digital libraries typically provide poor support for the earliest problem formulation stage, or for re-finding documents seen but not immediately judged relevant during that stage. It seems likely that a better understanding of how information seeking with digital libraries forms part of the larger information task whether that be preparing a legal brief, diagnosing an illness or writing an article would result in the design of libraries that are better suited to the broader task context.
In some situations e.g., the provision of medical information (Adams and Blandford, 2002) the introduction of new ways of accessing information can create conflicts within the organisation. The democratisation of information can break down traditional power hierarchies by empowering junior staff while disadvantaging senior staff who have poorer information seeking skills and less time to acquire them. While recognition of these factors may not lead one to consider detailed design changes to mitigate against or accommodate such effects, the need for sensitive deployment and appropriate user training are clearly indicated.
Just as the roles and relationships of end users within an organisation are changed by the introduction of digital resources, so those of librarians are also going through a period of rapid change. As information becomes more readily available to end users from various locations, those users have less need to go to a physical library, risking leaving the traditional librarian "out of the loop". This demands changing working practices to ensure that communication between librarians (as information specialists) and end users (as consumers of information) remains effective. Arguably, the design of digital libraries should make more provision than is currently the case for end users to access local support for library use.
More generally, end users need information at the right time and in the right place. At present and probably for the foreseeable future paper still has an important role to play, as does people's ability to share, annotate and organise pertinent documents in ways that support their broader tasks. Effective digital libraries will be those that support users' information tasks. As one librarian at the JCDL workshop observed, use of any particular digital library is discretionary, and "what they need is information, not to use this library".
In this section, we have outlined a range of factors that contribute to usability from detailed technical factors that influence the reliability and basic functionality of a library, through requirements on the interaction, to factors that relate to the context of use. This list is by no means complete. For example, we have not even mentioned screen layout or detailed design of user navigation through data structures; these are aspects of interaction design for which the requirements for digital libraries are not substantially different from those of other systems, and for which general HCI techniques can be applied (see, e.g., Preece, Rogers and Sharp (2002).)
We close this section with a brief summary of some particular points raised at the JCDL 2002 workshop. Firstly, familiarisation is important: users need to be able to rapidly become familiar with the library structure, type of contents, and search mechanisms. Designers need to work towards greater consistency of interaction style: just as one can hire a car in a different country and rapidly master its features to be able to drive it safely, so digital library users should be able to switch between one digital library and another without needing to learn "from scratch" again, and it should be possible to rapidly assess what type of contents a new library contains.
Secondly, users need to be able to detect when they are leaving or entering a library; at present, boundaries are often poorly marked and cause great confusion. Different libraries often have local restrictions on their use (for example, depending on the subscription held by the user or their organisation) that are inadequately signalled at the interface. The user generally has to select what collection to search in without adequate information about the likelihood of particular documents (or documents satisfying their search criteria) being there. Overall, users need more information presented clearly within the context of use about the collections and their boundaries and limitations.
One of the hardest design challenges to address is that of design evolution. As users become familiar with particular systems, they typically adapt their behaviour often in ways the design team did not anticipate to make systems work effectively for them. Design changes, often completed with reference to a few users but not all, are introduced without sufficient notice, disrupting users' patterns of behaviour and forcing new learning of how to use tools again. There is an unavoidable tension between stability of systems that may be difficult for new users to learn and design changes that improve systems for new users but reduce the capabilities of existing users.
Design of the development and deployment process
The previous section considered the products of design; in this section we consider elements of the process. One of the major difficulties of digital library development is that disjointed teams of developers are typically responsible for different components: one group may be designing the core digital library system while another is building document collections, a third is providing a local portal through which to access those collections, and a fourth is delivering web browsing systems. What the user experiences is the overall effect of the interactions between these various system components that may be more or less well integrated.
To date, few studies have investigated the usability of publicly available (e.g., by subscription) digital libraries that are accessible over the Web. The majority of usability studies focus on the usability of one particular library, very often using pre-defined tasks (rather than the user's chosen in-context tasks) for the study. There has also been a greater focus on post hoc usability evaluation (after a system has been designed) than on user-oriented tools to support design.
Most evaluation studies have been empirical. Some apply quantitative techniques for example, comparing two versions of a system to establish which is better on some measurable criteria. Others focus more on qualitative techniques, typically identifying user difficulties when working with a new system. Such techniques generally deliver results that are highly pertinent to the design of the particular system being investigated, but that do not readily generalise to the design of other systems. One great challenge articulated by participants at the JCDL workshop was that of finding and incentivising typical users, and giving them meaningful tasks.
Others have sought to reduce the need for representative users by developing heuristics and checklists to enable librarians and other key staff to assess usability without empirical evaluation, or as a pre-cursor to such evaluation. For example, some have applied Nielsen's (1993) usability heuristics to the design of digital libraries. Taking a more focused approach, Sandusky (2002) presents a checklist of criteria that any digital library should satisfy.
Several librarians at the workshop expressed a desire to have a compendium of tools, together with guidance on what evaluation tools to apply when.
Rather than just post hoc evaluation, insightful though that can be, it is clearly preferable to build a user perspective into the design process. Various research projects are tackling this in different ways.
One approach, exemplified by the work of Bishop and Bruce (2002) is to include end users as members of the design team ("participatory design"). In this approach, the user perspective is represented throughout the design process. As well as the clear benefits of such an approach, there are some well-documented disadvantages, such as those particular users gradually being assimilated into the design team in a way that they lose touch with the perspective of the users they initially represented, and users not being particularly design-savvy (in terms of knowing what is technically feasible and what is not). Nevertheless, as an approach it represents a great step forward from non-consultative delivery of technical solutions that fail to match user needs.
Another approach, reported by Jones and Sumner (2002), focuses not on how to evaluate a particular system, but on how to evaluate and improve an organisational structure and development process a much less well defined problem.
Finally, a third approach being investigated by groups including Bolcini and Paolini (2002) and Keith et al. (2002) is the development of tools to support design. These include the use of Claims Analysis and scenarios, adapted from the work of Rosson and Carroll (2002), which involves members of the design team creating scenarios of use and developing a design rationale, accounting for how well the proposed design will support users in the scenarios. The effectiveness of this approach depends on the design team being open to critical reflection on their own work, and on the quality of the scenarios used, which are ideally based on extensive data gathering from representative users.
In summary, there is no "silver bullet" that ensures good user-centred design. The challenges of technical complexity are compounded by the complexities of understanding users' tasks and information seeking behaviours.
Great strides have been made over the past few years: progress has been made on a range of technical challenges, and digital libraries are making an impact on the ways people work in many situations (academia, medicine and other professions). Nevertheless, if such resources are to have maximum impact on the way people work with information, they must fit naturally with the ways people work.
There is a natural tension between technical developers who are creating new interaction possibilities and usability specialists who want products that work with users the way users think in the context of their current tasks. If this tension is used constructively, great progress can be made, whereas if usability specialists just harp on about problems while technical specialists create interesting, but ultimately useless, systems, progress will be extremely slow. There is a real need for effective communications between all stakeholder groups.
Even core services are not "usability neutral". That is: technical specialists cannot design and implement fundamental structures and algorithms (e.g., protocols) on which libraries are built, serenely assuming that their work is "neutral" with respect to the way in which the libraries are subsequently used. Every important design decision has an impact on what is subsequently possible, and usability cannot be "added on" at the end. User needs have to be taken into account from the earliest stages and the deepest levels of design.
In the short term, checklists and guidelines may help in the development and implementation of systems that work with users rather than against them. In the longer term, a deeper understanding of user behaviours and user needs, and user-oriented design techniques will be necessary. Arguably this will demand a paradigm shift in the ways digital libraries are designed and deployed.
Adams, A. and Blandford, A. (2002) Acceptability of Medical Digital Libraries. Health Informatics Journal. 8(2), 58-66. Sheffield Academic Press. ISSN 1460-4582.
Alexander, W., Anderson, T. and Blandford, A. (2002) Presentation at ECDL panel on usability of digital libraries. Available from <http://www.uclic.ucl.ac.uk/annb/DLUsability/ECDLpanel.html>.
Attfield, S.J. and Dowell, J. (2003) Information seeking and use by newspaper journalists. Journal of Documentation Vol. 59, No. 2. 187-204.
Bates, M. J. (1995) The Getty End-User Online Searching Project in the Humanities: Report No. 6: Overview and Conclusions. College & Research Libraries 57. 514-523.
Bishop, A. P. and Bruce, B. (2002) Usability research as participative inquiry. In A. Blandford and G. Buchanan (Eds.) Proc. Workshop on Usability of Digital Libraries at JCDL'02. 26-28.
Blandford, A. and Buchanan, G. (2002) (Eds.) Proc. Workshop on Usability of Digital Libraries at JCDL'02. Available from <http://www.uclic.ucl.ac.uk/annb/DLUsability/JCDL02.html>.
Blandford, A. E., Stelmaszewska, H. and Bryan-Kinns, N. (2001) Use of multiple digital libraries: a case study. In Proc. JCDL 2001. ACM Press. pp 179-188.
Bolcini, D. and Paolini, P. (2002) Goal-Oriented Requirements Specification for Digital Libraries. In Proc. ECDL 2002, LNCS2458, pp107-117, Springer Verlag, Berlin/Heidelberg
Hsieh-Yee, I. (1993) Effects of search experience and subject knowledge on the search tactics of novice and experienced searchers. Journal of the American Society for Information Science. 44(3). 161-174.
Jones, C. and Sumner, T. (2002) Evaluation of the National Science Digital Library. In A. Blandford and G. Buchanan (Eds.) Proc. Workshop on Usability of Digital Libraries at JCDL'02. 5-6.
Keith, S., Blandford, A., Fields, R. and Theng, Y. (2002) An investigation into the application of Claims Analysis to evaluate usability of a digital library interface In A. Blandford and G. Buchanan (Eds.) Proc. Workshop on Usability of Digital Libraries at JCDL'02. 15-20.
Kuhlthau, C.C. and Tama, S.L., (2001). Information search process of lawyers, a call for 'just for me' information services. Journal of Documentation, Vol. 57, No. 1, pp. 25-43.
Kuhlthau, C.C., (1993). A principle of uncertainty for information seeking. Journal of Documentation, Vol. 49, No. 4, pp. 339-355.
Landauer, T. K. (1995) The trouble with computers; usefulness, usability and productivity. MIT Press.
Marchionini, G. (1995) Information seeking in electronic environments. Cambridge University Press
Nielsen, J. (1994) Heuristic Evaluation. In J. Nielsen and R. Mack (Eds.), Usability Inspection Methods (pp. 25-62) New York: John Wiley.
Preece, J. Rogers, Y. and Sharp, H. Interaction Design. Wiley, New York (2002)
Rosson, M. B. and Carroll, J. M. (2002) Usability Engineering. San Francisco: Morgan Kaufmann.
Sandusky, R. J. (2002) Digital Library Attributes: Framing usability research. In A. Blandford and G. Buchanan (Eds.) Proc. Workshop on Usability of Digital Libraries at JCDL'02. 35-38.
Sutcliffe A., and Ennis M. (1998) Towards a cognitive theory of information retrieval. Interacting with computers 10. 321-351
Vakkari, P. (2001) A theory of the task-based information retrieval process: a summary and generalisation of a longitudinal study. Journal of Documentation. 57(1). 46-60.
Winograd, T. and Flores, F. (1986) Understanding Computers and Cognition, Reading MA: Addison-Wesley.
Ann Blandford and George Buchanan