* * * * * * Draft of 3/14/97 AM * * * * * *

The Alexandria Digital Library (ADL) is one of the six digital library projects funded by NSF, DARPA, and NASA. ADL’s collection and services focus on geospatial information: maps, images, georeferenced data sets and text, and other information sources with links to geographic locations. Throughout the project, user feedback has been collected through various formal and informal methods. These include online surveys, beta tester registration, the tracking of user sessions through the web interface, ethnographic studies of ADL users and potential users, target user group focus sessions, user feedback comments while using the interfaces, and a formal Design Review Panel. This paper briefly describes the evaluation studies conducted and what was learned about user characteristics and the study approaches themselves. User reactions to the ADL interface and to the functionality and content of ADL are summarized. Finally, the value of these findings to design and implementation decisions is considered.

Work in the area of digital libraries has gained momentum over the last decade (Fox, 1995). Much of this work, perhaps related to the term digital is involved with computer science issues such as databases, networking, storage, scalability, and image processing. However, it is not mere coincidence that the term library is included in the phrase digital library. Digital libraries are expected, eventually, to provide at least those services provided by traditional libraries, and much more. Thus, the study of digital libraries also includes such areas as economics, security, ethnographic and sociological studies, and usability. Regardless of the power of the system, it is successful only to the degree to which the vast majority of its intended users are easily able to use its intended functionality. Therefore, the six projects in the NSF/DARPA/NASA Digital Library Initiative (CACM, 1995; National Science Foundation, 1993) have put serious effort into evaluating emerging prototypes in relation to users. This paper discusses some of the evaluation work within the Alexandria Digital Library (ADL) Project.

ADL focuses on geospatial information and data within the context of a digital library. Beginning with the first Rapid Prototype (Alexandria Digital Library Prototype CD, 1995), and the current Web Prototype (Andresen et al, 1995), ADL has attempted to provide a system which facilitates the search and retrieval of any information that is geographically referenced. This includes maps, aerial photographs, and satellite images, but also encompasses text and other traditional items found in library collections. Most of the development to date has been driven by the need to demonstrate key functionality. Regardless of the "klunkiness" of the interface or overall system design, this function-driven approach served the purpose of either proving that the system could perform requisite tasks or bringing to the surface problematic components or points of integration. Throughout this process, evaluative tasks were being performed. The system is currently being redesigned with the hope of fully incorporating the perspective of the intended users into a "user-centered design" (Sugar, 1995).

Digital libraries go beyond what might be considered standard information retrieval (IR) in that they may include, for example, post-retrieval activities such as long-term storage of user profiles and results, integration of results with processing, and a mixing of collections and processes in a user’s work space and the central library. User-centered design, for a digital library must include not only systems evaluation but also an understanding the process of information seeking and use (Marchionini, 1995)

Of course, the system changes over time, and there are obvious problems with trying to evaluate such a system. However, it is straightforward to deal with this via version control, careful planning, etc. What is perhaps more interesting is the interplay between the design and the user feedback. Clearly with new software systems there is a degree of education involved; the users must understand the capabilities in order to evaluate them. This would be considered part of the standard "usability techniques." However, as a research project, we are attempting to tailor the design based on the needs of these educated users. Thus the process might be thought of as a cycle; one in which the implementers build certain functionality, the evaluation team tries to educate users about the potential, the users realize new potential, some of which may not be part of the system, and the implementers modify the design (perhaps with a more generalized). How do we evaluate a system whose functionality is determined in part by user feedback, while the feedback is used to measure, in part, how well the system helps users perform the functionality? We make the comparison to the "self-evident door handle" (Norman, 1988, pp.87-92), which, when correctly designed, makes its use obvious. We are not building something as simple, nor as functionally static, as a door handle. For a digital library – specifically, the interface to a digital library – we need to find the conceptual models and affordances (Norman, 1988, pp.9-12) that work for users.

Alexandria is not primarily an evaluation research project, per se, nor does it necessarily attempt to develop new models of IR behavior. Its focus is more on: 1) incorporation of geospatial searching into existing IR models as an extension of those models; 2) computer science research topics; and 3) the implications of user studies on the design and implementation of (spatial) digital libraries.

There are many evaluation methods available. Evaluation studies range from more specific information about a few users, as in videotaping, to less specific information about a lot of users, such as the beta test registration information.

We chose methods based on:

• Having people who could perform the method
• Working within cost and time limits
• Maximizing the expected Return on Investment (ROI) of participants and staff (Weedman, 1996)
• Using methods that complement one another

members of the User Evaluation and User Needs Analysis Teams come from diverse backgrounds including computer science, information science, education, sociology, and library science. As is common in such a group, it has taken some time for us to understand each other's academic background and orientation, as well as the problems we are each attempting to address. As this understanding has grown, the geographical topics, evaluation methods, specifications and requirements, and even cognitive IR models have grown into an integrated group-wide conception of whatever it is we're building. This allows us to do a better job of incorporating the evaluation results into the implementation, directing evaluation to answer specific questions, such as integrating GIS concepts into IR models.

We chose the following methods:

• Surveys
• Target user groups
• Ethnographic studies
• Demographic analysis of beta tester registration data
• Log analysis

The first four of these studies are being done at the University of California, Santa Barbara, and are reported on here. Dr. Barbara Buttenfield (Buttenfield, babs@colorado.edu) is doing the log analysis work at the University of Colorado, Boulder. She has nine months of transaction logs incorporating over 134,000 transactions and is using them to illustrate graphically the navigation paths users choose through ADL (Buttenfield & Kumler, 1996). It is part of our plan to integrate the direct user involvement studies with the log analysis studies in the next stage of analysis.

We view our choice of methods to be complementary to each other. The demographics from the beta tester registration provided the background for the online survey data and told us something about the people who showed an interest in using our system. The survey obtained more detailed information from a subset of those users about their reactions to the interface. The analysis of the user comments from the survey and other sources supported the findings of the videotaped and audiotaped sessions. The Target User Groups, which focused on user needs as expressed through user scenarios and user requirements, gave us an idea of how the task environment affects user expectations of such a system.

Beta Tester Demographics

The ADL Beta Tester program began during the spring/summer of 1996. At that time, interested persons were able to sign-up with ADL to gain access to the library. Each person had to fill out a short Web Access Request Form in order to receive the username and password, which enabled them to access the full functionality of ADL on the Web. The original intent of the Web Access Request Form was to provide the information needed to decide who would be chosen to be a beta tester. Therefore, the responses to the questions were free-form; that is, not bounded by a limited set of choices and not verified. It was subsequently decided to let everyone become a beta tester who requested access.

The Request Form contained five mandatory fields and a set of optional fields. The required fields were Name, Email, Organization, Occupation, and Referral. None of these was controlled in any way by a domain or accuracy check. Everyone who submitted the form with a valid email address received a confirmation notice with the username and password.

There were 2287 beta tester registrations available to us when we analyzed the data. The responses were analyzed manually to see what could be learned about the set of interested users who wanted to test the ADL web interface. An analysis of the access logs indicate that beta testers came in from 906 different IP addresses, with a total of 1,340 sessionsbetween August 12, 1996 and February 4, 1997. Since the usage logs before August 12^th were lost as well as the logs for fourteen days after August 12^th, the full count of the number of accesses by beta testers is not possible. A smaller number of beta testers (109) submitted the online survey form; see description of the survey below.

A high-level summary of the results of the analysis of the beta tester data from the Access Form is that

• Most of them discovered ADL randomly through the web.
• Presentations, publications, and personal visits to UCSB were cited as the referral source by approximately 25% of the beta testers.
• There was worldwide participation from 49 countries, with 65% from the U.S.
• The Email domain analysis was inconclusive because many users came through commercial network vendors.
• Scientists comprised 42% of the group by occupation - geographers alone made up 9% of the total.

The ADL User Feedback Survey is a web-based online survey

(http://alexandria.sdc.ucsb.edu:3366/ADL/FillOutSurvey). It is a six-part questionnaire that combines open-ended with Likert-scale forced-choice questions. This survey was developed from previous surveys and interface literature results. In addition, we added items specifically oriented toward our particular system. Questions about the user's background were also included, such as his or her familiarity with computers and with the type of information ADL is designed to retrieve.

There were three primary goals of the survey. The first was to provide a mechanism for acquiring detailed and directed feedback about users' experiences with the interface. In particular, it provided quantitative data to complement the qualitative data generated by some of our other evaluation methodologies. The second was to learn something about the ADL community of users. Finally, it was hoped that the survey would allow us to study relationships between users' experiences with the system and their background characteristics

The resulting survey consisted of a set of multiple choice questions and open-ended questions. Most of the multiple-choice questions were formatted with five possible answers: strongly agree, agree, neutral, disagree, strongly disagree, no opinion. These questions were randomly formatted in a positive or negative sense to remove the tendency of users to give all answers the same and encourage a more careful reading of the statements. Each major section of the survey included an open-ended question with room for a free-text response. These provided opportunities for qualitative feedback in respondents’ own terms, and opportunities for respondents to make specific recommendations and point out areas of the system that the survey might not sufficiently cover.

There were 96 usable surveys (completely or nearly completely filled out) from 109 surveys submitted. Demographically, the group is:

• 80% male
• 87% have college degrees
• 63% have a Master’s or Ph.D. degree
• the average age is 35-36 years old
• 24% said that English is not their native language

They are, as a group, frequent users of computers, libraries, geographical data, the web, and online catalogs.

These statistics paint a profile of the ADL user community that is more male and more highly educated than Internet users in general. For example, the Yankelovich "Cybercitizen" survey of 1995 (http://www.yankelovich.com/cyber/FINDINGS.HTM, 2/27/97) found that cybercitizens

• are more likely to be men -- 57% versus 48% for adults in general;

• are better educated -- 33% with a college degree versus 19% adults in general; 57% with some college versus 43% for adults in general: and
• are younger -- an average age of 35 versus 43 for adults general.

Using the incomplete usage logs that are available, it appears that between 40% and 70% of the beta testers actually used ADL. If it is therefore assumed that between 1000 and 1600 beta testers actually used the system at least once, the survey responses that were submitted represent approximately 6% to10% of the active beta testers and approximately 4% of the total number who signed up to be beta testers. The low response rate and uncertain extent of any non-response bias suggest caution in generalizing the results. Also, most respondents filled out this survey after no more than one or two sessions of exposure to ADL.

Factor analysis using multivariate data reduction techniques (e.g., Mulaik, 1972) produced six factors that account for 58.5% of the variance in the original data. These are shown here, in order by their effect on the variance of responses, with the questions that contributed the most to each factor. The questions are randomly stated in either a negative or positive sense.

(Variance accounted for: 19.7%; weak but statistically significant approval)

The sequence of screens is confusing

Learning to operate the ADL is difficult

The ADL is difficult to use

The needs of inexperienced users are taken into consideration

Tasks can be performed in a straightforward manner

I do not like the Gazetteer section

Remembering names and use of commands is difficult

The screen layouts are helpful

(Variance accounted for: 9.0%; strong and statistically significant approval)

The ADL is wonderful

The ADL is stimulating

I like the Map Browser section

Exploration of features by trial and error is encouraged

(Variance accounted for: 8.6%; strong and statistically significant approval)

I understand most of the terms used throughout the ADL

The ADL terminology does not relate well to the work I do

I like the exit procedure and session-saving capability

Help messages on the screen are unclear

(Variance accounted for: 7.7%; not statistically significant different from neutral)

The ADL could frequently help me do work I need or want to do

I do not plan on using the ADL very often

(Variance accounted for: 6.9%; not statistically significant different from neutral)

The ADL performs too slowly

The ADL has adequate power

The ADL is frustrating

(Variance accounted for: 6.6%; strong and statistically significant approval)

Boundaries between the main ADL sections are clearly marked

The ADL keeps me informed about what it is doing

The ADL commands were well depicted by the buttons and symbols

These interpretations accord quite well with the summary of reactions to the individual items. The results paint a mixed picture: they point to the existence of important difficulties with the current implementation but also provide support for some of the directions that have been taken. Average reaction to ADL may be described as "mildly approving." In particular, the largest approval was found for the clarity and organization of the terminology and component sections. Particular problems were identified for the speed and performance of ADL. It is unclear to what degree the latter problems are due to characteristics of the web or Internet network. These reactions were based on relatively little experience with ADL; it will be critical in the future to consider the nature of the learning or experience curve when using ADL.

Virtually no relationships were found between any user characteristics and reactions to ADL. It is critical to realize, however, that our sample of respondents is very non-representative of the population, whether that is the entire population of people in the country or just the eventual population of users of systems like ADL. An excellent example of this is web use. Over 80% of our sample reported working with the web at least "daily." That is probably not representative of all people who might one day use ADL.

The only significant relationship is to the respondents’ sex, with females being less approving of ADL than are males. Female respondents were found to have higher rates of library use, fewer computer accounts, and used ADL a bit more often before filling out the survey, but none of these background variables were significantly related to their overall approval index.. Given the small number of females (19), this finding should be considered very provisional in any case.

. The comments to the open-ended questions paint a somewhat more negative picture of the ADL than do the Likert-scale responses; the great majority of the comments express problems or difficulties. This is consistent with the idea that people are more motivated to comment when they encounter problems than when something works. The comments are being evaluated as part of the ethnographic studies.

A team from the UCSB Graduate School of Education conducted ethnographic studies to inform the ongoing development of the web interface and its underlying library development. The Education Team decided to describe and analyze user activities and interactions both in the physical Map and Imagery Laboratory (MIL) in the UCSB library and in the virtual workspace of the ADL web interface, noting the similarities in use patterns and expectations.

The Map and Imagery Laboratory is renowned for its collection of geospatially referenced data, including aerial photographs, remotely sensed images, maps and digital cartographic data. The goal of the Alexandria Digital Library (ADL) is to create the electronic equivalent of MIL's services and collections. Because the design of ADL originated in the actual practices of MIL, one of the studies is from audiotaped recordings of reference interviews conducted by MIL staff. Visitors to MIL were informed of the study and invited to participate. If they agreed, the staff person tape-recorded the reference interview.

The holdings of the MIL are in closed stacks, accessible to the public only through the services of the library staff. This means that the reference librarians operate both as guides and as gatekeepers. To study user interactions in the MIL, reference staff solicited participation in the study from those who asked them for assistance during the study period. When users agreed to participate, audiotapes were made of the reference interviews. This resulted in thirteen recorded and transcribed sessions.

Two distinct modes of interaction between user and reference staff person were noted initially in the thirteen transcribed interviews. One mode was for the user to pose a question and depend on the reference person to expand and develop the question and suggest or recommend appropriate resources. The other mode was for the user to direct the reference interview, tapping the knowledge of the staff person as needed. On analysis, these modes were expanded into four different patterns (Table 1) based on two domains of knowledge: task knowledge (along the continuum of familiarity with spatial information and its uses) and system knowledge (along the continuum of experience with the Map and Imagery Laboratory). We found examples from the MIL reference interviews depicting all four of these patterns. These patterns reflect the interrelated proficiency with both system and task.

• Pattern 1 lacks both system and task knowledge (limited proficiency);
• Pattern 2 has system knowledge but lacks task knowledge (system proficiency);
• Pattern 3 lacks system knowledge but has task knowledge (task proficiency); and
• Pattern 4 has both system and task knowledge (full proficiency).

Reference assistance adjusted to each of the patterns as the examples below illustrate. The example of Pattern 1, limited proficiency, indicates an interaction typical of what happens when a user asks for something general, using vague language, thus requiring more from the reference staff who must second guess both what the user wants and for what purpose.

Example of Pattern 1: Lacking system and task knowledge (limited proficiency)

In this example, the user seems unaware of the way a map is used to present information (in this case, country boundaries because the request was for a map of the world, where world was defined as "all the countries.") It is up to the reference staff to suggest that the country names should be listed as well, and that the user may wish to have a copy of the end result. The user clearly depends on the reference staff for assistance. The assumptions of the reference staff are clear, too: 1) the user has a reason for being here, a purpose with an expected outcome (which often includes being able to walk away with a product); and 2) asking the right questions can determine this information.

In Pattern 2, system proficiency, the user is more aware of his own purpose and familiar with the services MIL provides. This does not, however, include task knowledge, which means that the user does not know what particular artifact to request, or how it may be used.

Example of Pattern 2: System knowledge without task knowledge (system proficient)

Interestingly, this search didn't pan out. The reference librarian returned with an alternative:

The user knows that the MIL provides services that allow people access to artifacts that contain spatial information, and that this information can answer questions. Because the user lacks familiarity with spatial information, he explains the nature of the project on which he is working and the particular information he needs (distance to the coast), but he depends on the reference staff to recommend useful artifacts. The staff person suggests different kinds of artifacts that may include the information requested, but the user is unfamiliar with their various uses and depends on the reference staff for guidance, taking the time to explain further what he needs. In this particular case, the reference staff discovers that the MIL lacks the necessary artifacts, but he is able to suggest an alternative. The user expects to receive assistance from the reference staff and MIL. Once again we see the staff assuming that the user has a specific purpose and an expected outcome, and using these assumptions to negotiate an understanding with the user.

When the knowledge shifts from system to task (task proficiency) we see a different pattern of interaction. In Pattern 3, the user is inexperienced with the Map and Imagery Laboratory, but is familiar with spatial information and its uses. As a result the user comes to the reference interview with both a specific purpose and a clear question, guiding the direction of the discussion.

Example of Pattern 3: Lacking system knowledge but having task knowledge (task proficient)

Because the user knew exactly what to request, the reference staff person is able to move directly to an explanation of how the system works. He hands over an index and suggests that the user be comfortable while perusing it for the necessary "quads" (the short form of the word for quadrants itself suggests an implied understanding that the user already knew the particular spatial information artifacts needed and was aware of how to use them.) The user expects the reference staff to facilitate his search and retrieval of particular artifacts. The reference staff assumes that the user knows what he wants and will be able to select it from an indexed list.

Pattern 4, full proficiency, describes interactions where the user has both system and task knowledge. At such times, the user enters the Map and Imagery Laboratory knowledgeable about the system and familiar with spatial information and its uses.

Example of Pattern 4: Knowledgeable of both system and task (full proficiency)

The user is able to discuss procedures, costs, and equipment for processing, causing the reference staff person to comment on his extensive knowledge. The expectations of the user are that the staff will understand his task, the processing equipment needed, and the ways to charge for the services. The reference staff assumes that the user is fully aware of search, retrieval, processing, and the related charges.

Concurrent with the collection and analysis of the MIL reference interviews, users interacting with Alexandria Digital Library's web interface prototype were videotaped. As with the users of the MIL, there were significant differences in how people interacted with ADL depending on the user's background knowledge in several areas. Unlike the MIL experience, however, users did not have the equivalent of a reference librarian to whom they could turn to reshape a question or to be guided toward a successful outcome.

Table 2 presents the range of background knowledge required of ADL users in order for them to be most likely to feel successful in their interactions with the web interface to the Alexandria Digital Library. Only one group of users could be said to have all of the knowledge required and this was a group of the three people most responsible for developing the ADL web interface prototype. [Can we put a number in the cells instead of the Xs to indicate the RELATIVE depth of knowledge that these subjects have along these dimensions? It appears that all subjects have the same knowledge of library search strategies, for example. The inference is that the degree to which they have library search strategy knowledge doesn’t matter – I don’t think this is true]

Eight Different Videotapes of Use

Table 3 shows the characteristics of the videotaped sessions and gives some indication of how successful or confident the participants felt in their interactions with ADL.

[Can you now relate the knowledge held to the success they had? Are any of the knowledge areas more important than others? Or can you make links between the lack of knowledge in these areas and the types of problems they had?]

The Education Team also conducted a domain analysis of the feedback comments made by beta testers while they were using the web prototype and of the comments that were submitted as part of the online survey of beta testers. These comments were categorized by the section headings of the online survey of the beta web interface so that they could be easily correlated with those results. The Team is in the process of clustering the data in other ways to characterize user reactions and expectations.

Many problems were identified from the patterns seen in the data. Users expressed impatience with the introductory material and with the slowness of the system itself. However, they recognized the problem ADL was battling to make a complex system understandable--that is, useful and efficient while still providing sophisticated functionality. It is a design issue to determine how transparent or opaque to make that functionality, to decide how much users need to know as they use the system and how much can occur hidden from their sight (which may also mean removed from their control).

Studies of users of spatial data in libraries are not exactly thick upon the ground. Examples of what does exist are Deckelbaum, 1989; Larsgaard, 1987; Leavey & Moore, 1988; Leeuwenburg, 1982; Mittra & Ghatak, 1993; Perry, 1992; Seavey & Rex, 1992; Treude, 1975; and Treude, 1981. The studies tend to be based on questionnaires to elicit user evaluations of the library services provided. The reason for this relative sparseness has to do with several matters. First may be that the vast majority of all map collections, and especially those of any size, are held by university libraries. Overall, the users of one university library have needs much the same as those of another university library, although within any one university, these users’ skills go from novice to expert. Thus, for example, the article by Leeuwenberg, which discusses users in a university library in Australia, holds no surprises for a map librarian in charge of a collection in a university in the United States.

Yet another reason for the lack of studies in the past is that it has been difficult to design studies that do not take up user time, which for university users is always at a shortage. If user studies could be done by having computer software "follow" a user through a question - as is done with transaction logging on the Web - that would be ideal. Otherwise, users look upon user studies with disfavor unless these studies take up less than five minutes - and preferably no minutes - of the users’ time. Map librarians have responded to this fact of life by doing studies that do not directly involve the user: counting the number of items checked out and refiled by, for example, geographic area, by type of borrower (e.g., student, faculty, staff), and so forth, rather than by having the users fill out forms or undergo lengthy analyses of user needs.

The ADL User Needs Analysis Team decided to use a focus group technique to gather information from potential users of ADL – largely the same group that currently uses the Map and Imagery Laboratory. The scope of the potential user base, according to the NSF Cooperative Agreement, is any user who is searching for georeferenced information. To provide focus to this open-ended charge, three groups were chosen for Target User Groups:

• Earth Scientists: the primary users of the current web interface
• Information Specialists: a group that includes librarians but is not limited to the traditional reference librarian in a traditional library
• Educators: a group that does not currently use ADL and for whom the web interface is not specifically designed

Two Target User Group (TUG) meetings were organized and held during the year, one on August 23, 1996 (full day; prefaced by two-hour introductory meeting on August 12) and the second on January 13, 1997 (half day). At least three people from the local area were recruited for each TUG. They were asked to represent their user groups rather than focus exclusively on their personal task environments. They had varying degrees of acquaintance with the current ADL web interface. The purpose of the sessions was not to critique the current interface but rather to find out what these users would do with a system like ADL if it were all they would like it to be.

Each TUG session began with an overview of what was to be accomplished in the session, followed by discussions in the individual focus groups, and then by a plenary session to compare and contrast the focus groups’ work. For the all-day meeting in August, each group identified user requirements in the areas of Content, Search, Retrieval, Processing, and Interface Design. They also selected (or added) user scenarios from a list that best represented the information tasks of their group. Following the meeting, team members categorized the requirement statements into groupings related to implementation. During this process, the statements were edited for clarity and statements were added to cover more fully the categories that were created. The second TUG meeting, which included many people who were not at the previous meeting, validated many of the points made during the first TUG meeting.

Three major results came from the TUG sessions: (1) characterizations of the three target user groups, (2) design issues, user requirement statements, and scenarios and (3) a model TUG session plan for others to use. These are described next.

The following characterizations include some of the scenarios that each group selected as representing how they would use ADL.

The Earth Scientist (ES) Group will look to ADL as a source of data for their research and development activities. They are interested in access to better technology as well as access to available data sets so that they can do better science. ES works in a high-tech environment and is interested in speed and the capability of handling large data sets. They want to perform more complex analyses and integrate disparate data sets. They want to develop models of complex systems and be able to visualize the results. ES will have very specific search criteria in terms of level of detail, source and nature of the data, and data quality. They are likely to need to obtain copies of the data set for use with their own software.

The ES Group would like for ADL to provide a working environment where they can find and manipulate spatial data sets and related georeferenced data and information that are related to particular research problems. They chose the following scenarios as best representing their activities:

• Basemaps: The most important use that I have had for spatial data has been in the acquisition of digital basemaps. For example, DLG files or Census tiger files for various areas about the United States.
• Aerial photography: From recent searches in preparation for pending litigation stemming from the La Conchita landslide, Ventura Co., CA. All available coverage from aircraft platforms of La Conchita regardless of scale, film type, date, or perspective (vertical and oblique). Stereo coverage preferred. Since site is located near Ventura/Santa Barbara counties border, it is necessary to search existing analog record (organized by county) under both counties since coverage often slops over county boundaries. A subset of this would be to search for best (most detailed) stereo coverage of site at as close to 5 year intervals as possible from earliest to present.
• Aerial photos and remote sensing: Researcher wants to know the available air photo and remote sensing coverage for a region. And what is available in the library. More specifically, the researcher might want to find out what the earliest photo available is and the latest photo and what photos are available with scales greater than xxx.
• Environmental monitoring over time: Environmental change research tied to specific locations. Examples: habitat loss, landscape changes following development, flood and fire events, regional biodiversity.
• GIS analysis: Find a map, at a scale no smaller than 1 in 200,000 that: covers any part of the Mississippi Valley; shows a town with a population of over 500,000 people; shows roads; is within a 30 minute drive of some Indian burial sites for which the library has nineteenth century photographs in digital form.

What sets apart the Information Specialist (IS) Group in their approach to ADL is that they are more likely to seek information for others rather than for themselves and, because they spend more time with the search and retrieval system itself, learn the system more thoroughly, use more complicated retrieval functions, and have more complex retrieval requirements. In any one session with ADL, they are likely to handle multiple queries for different people. For example, using ADL to do a search for a student and then starting a completely new search for a faculty member.

ISs see a wide range of information needs and have to know at least a little about many information contexts in order to guide others in the use of the systems and the collections available. In the context of a geospatial library, they need to have at least rudimentary skills with map-based systems (e.g., Arc/Info) and networked computer functionality (e.g., ftp). They need to know what can and can’t be done with the systems and tools available to them and about collections of materials both online and off-line. This group does not usually produce a final product, unlike the Earth Science group, and will not necessarily walk users through the process from start to finish (finding the information, collecting it, manipulating it, producing a report) like the Education group will with their students. This group may, depending on their mission, offer training in the use of systems like ADL. The IS group would like ADL to help them do their job of identifying and making available information resources that meet the needs of their clientele for spatial and georeferenced information.

The Information Specialist Group chose the following scenarios as best representing their activities:

• Locate named place: Student wants to know location of river, or land feature, described in a paper.
• Location of place with uncertain spelling: I'm looking for a small town in Germany, I don't know if it's east or west Germany. It's called Heidebring, or Hiedebring, or something like that. I'm not sure of the spelling.
• Locate spatial location and find associated terminology: An indexer (cataloger) has a document about a lake in East Africa. The name of the lake cannot be found in the list of valid (authorized) terms for indexing (cataloging) and the author of the document includes only the latitude and longitude coordinates and the country in which the lake is located, but not the region or geological provinces of which it is a part. The indexer needs to find by spatial reference (the lat-long coordinates available) the appropriate regional terms to add to the indexing (cataloging) record.
• Text and image search: At ten year or less intervals, find all documents describing Mascalitan island and images with a resolution more than 2 meters during summer periods and maps prior to 1930 at a scale greater than 1:24,000. Mascalitan island is a landlocked feature in Santa Barbara County.
• Country-related information: What do you have about the political regions of Brazil as well as data you have for concerning the country's infrastructure in all the main built-up areas of Brazil's municpios.
• Aerial photography: From Blue Oak Project. This project required identifying available historical aerial coverage between 1938 and 1950 at scales of either 1:20,000 or 1:24,000 for 150 site locations throughout California's vast Blue Oak Woodland habitat areas. Sites were identified by lat./long. coordinates. Also, roll and frame identification for the same 150 sites from recent NAPP coverage at 1:40,000 was needed.

The Educators Group is interested in incorporating technology and resources into their classroom activities. They put a high value on creating environments where students have the opportunity to think critically and where they have access to powerful information resources they can use to address real-world concerns. They would like for students to be able to use resources to get data they can manipulate to generate their own "products" - charts, maps, and essays based on what they learn. The work environment for education is significantly different from that of the other target groups. If there is a computer in the classroom it may be the only one, and very likely it does not have a fast processor or a lot of memory. The majority of schools still have most of their computers in computer labs and teachers are still working out ways to develop curriculum to use them effectively. There are classrooms with several computer workstations where the curriculum is focused on collaborative learning. In such cases, teachers may want to know how ADL can be a resource for group activities. Connectivity at the T1 level is coming to public schools in the Santa Barbara area but is not currently present in all locations. Primarily, education is going to be interested in knowing how to teach the system itself and what it offers and then how to use ADL as a resource to teach other concepts.

The Educator Group chose the following scenarios as best representing their activities:

County Connections Project: Students using historical maps and photographs for research projects on their own communities creating a web page with pictures, interviews, etc.

• Science Classroom: Students would be looking for weather patterns, erosion patterns, geologic formations, etc.
• Earth Science Classroom: A computer tells us there is an earthquake today in Lodi: find emergency services, roads, population density, evacuation centers .
• Family Trees: Tracing "roots" and culture.

Presentations on the system design and collection development issues for ADL were prepared for the August TUG session. This was the first attempt by ADL staff to focus on the main design issues in search, retrieval, and processing functionality and collection development from a user’s point-of-view. The issue framework was used to categorize the user requirement statements that were created by the TUGs and to structure the discussions at ADL Design Review (ADR) Panel Workshop on February 19-21, 1997. We now have five sets of user requirement statements covering search, retrieval, processing, interface and navigation and collection development (content). These statements have been formatted as web-based forms. We can ask various groups of users to evaluate the statements on the basis of their importance or on the frequency that they would use the features. The requirement lists will continue to grow and change as more user input is acquired.

We have a list of 58 user scenarios that indicate the types of questions that could be handled by ADL. These have proved to be very useful for system design and for characterizing types of user groups.

Another outcome of the TUG activities is the creation of a TUG session model that can be adapted and used by other groups who are willing to cooperate with us and collect user information from other groups. Because the emphasis of the session is to find out about georeferenced information needs – rather than evaluate any particular system or interface design – this session can be used in any location. The collected user requirement statements and scenarios, when integrated with existing information, will paint an ever richer picture to support the development of ADL and similar systems.

The challenge now is both to identify what we have learned about ADL users (potential and current) that is useful for design and implementation and what we have learned about the evaluation techniques themselves.

First, we look at four questions:

1. What have we learned about our users?
2. What have we learned about the evaluation and user study approaches?
3. What have we learned about the ADL interface?
4. What have we learned about the functionality and content of ADL?

Most of the "early adopters" (the beta tester group) of ADL on the web are a highly educated, information-savvy, worldwide group of Internet users, as indicated by the beta tester demographics and the survey responses. The survey responses indicate a range of ages from 17 to 70 years, with a mean of 35-36 years and almost a 4-to-1 ratio of male to female. We are handicapped by drawing any inferences from this survey data to our larger user group since surveys were only returned by approximately 4% of the beta user population. Likewise, we cannot find significant correlations between user characteristics and survey responses because (1) the number of surveys returned was small and (2) the survey population lacked sufficient representation of user characteristics such as age, formal education, and library use. The exception to this is that there is a significant difference in the overall negative/positive responses to survey questions based on sex; females were more negative. But this is based on only 19 female, so we can only note it as interesting; we cannot make more general inferences.

Our Target User Group (TUG) activities gave us a better picture of the characteristics of three user groups in relation to the use of geospatial information and data. The three TUGs - earth scientists, information specialists, and educators - indicated through their choice of the user scenarios that best represented them and their statements of user requirements that they have very different task environments and expectations of a system such as ADL in terms of functionality and content.

The groups can be clearly distinguished from one another.

• Earth Scientists are seeking data sets, work in a high-tech environment, have very specific search criteria, and want to have an ADL working environment in which to manipulate data sets.
• Information Specialists will spend more time learning the broad capabilities of ADL in order to serve a variety of needs brought to them by the end users. They are more likely to handle difficult search and retrieval tasks; less likely to process the data itself. The information specialists, to the degree that they work with earth scientists, reflected this commonality in their orientations; to the degree that they work with other groups (in the general university community, with the general public, or in special library situations) their interests are broader.
• Educators need to work ADL into their classroom activities. They are therefore interested in the potential for group activities and in content and functionality that supports their education goals. They are currently limited by sparse computer resources.

Whereas the TUG activities allowed us to understand user expectations, the ethnographic studies focused on users in action - while they were using either the traditional library or the ADL web interface. From the MIL reference interviews and the videotaped interactions with the web interface to ADL, we learned that the experience and background knowledge users have affect the way they interact within either the physical workspace or the virtual one. In the physical workspace, users are able to situate themselves within any part of the environment that they find familiar, such as the reference desk and sign-in book they see upon entering the facility. From this point, at least in the physical workspace of the MIL, users are able to interact with a reference staff person who facilitates the searching and retrieval process. The data show that all the users, regardless of where they are on the task/knowledge proficiency continuum, work well with this facilitated model.

In the virtual workspace of the web interface to the ADL, there is no person with whom the user may interact. At this point there is not even a virtual one, although there are the options to select the comment button or to email the system administrator. The users' background and experience are even more important to the interaction with ADL, since it is up to the users to determine how to search the system within the given structure and to use that structure to select what they want to retrieve. Moreover, successful interaction with the web interface to ADL assumes that the users have knowledge of a broad spectrum of background information and skills: multiple computer platforms, web and Netscape Navigator browser, library search strategies, maps as representations of geographic information, and geo-spatially referenced data and their uses. We also found that knowledge of programming and interface design and previous experience with ADL supported successful interaction immensely, though it was not assumed initially that users would come to the system with such knowledge.

In both the physical and virtual workspaces users expected to learn from the interaction, to be educated about the process, and to learn additional strategies for future interactions. In the physical workspace of MIL, this process was implicit in the interaction, with the reference staff working with the user to find the best way to accomplish the task. In the virtual workspace of ADL, the learning process was not as completely supported and users expressed their frustration at various points throughout their interaction with the system as a result. Commonly users would wonder aloud, or write in their comments, what they were doing "wrong" or what they didn't understand, suggesting that they felt the difficulty resided more within themselves than it did within the system. More proficient users tended to identify problems with the system.

The language of the interface (i.e., the terminology used in the ADL interface) was not a problem for the ‘early adopters’ who filled out the beta tester survey. It was, however, identified as a problem through the ethnographic studies, both in the videotaped sessions and in the user feedback through the web interface. This difference can probably be explained by the computer/system skills of the beta testers and their knowledge of geospatial information and data versus the comparatively uninitiated subjects of the ethnographic studies. [Specific language problems examples]

The effect of personal characteristics on reactions to ADL could not be derived from the studies undertaken, except for the slight indication of a sex-related reaction cited earlier.

We are interested at this point in our evaluations whether or not we received sufficient return on our investment (ROI) in the various studies of users and their expectations of and reactions to ADL. Did the studies yield a commensurate quantity and quality of information that can be used for digital library system modeling and development? Could we use our evaluation resources, which are limited, in better, more effective ways? What basis of understanding do we have now from the studies we have done that should guide what we do next? A related concern is how to structure incentives in such as way to get and keep user participation. In other words, the ROI for users who participate in our studies must also be a concern.

Dr. Judy Weedman’s study on the process of involving clients in application-oriented research (Weedman, 1996) found evidence for a basic instability in the collaboration of users and designers stemming from the incentives for participation of the two groups. From the system designer point-of-view, the investment in user studies must be beneficial to the design process. In order to get needed feedback for requirements analysis "[d]esigners must convey what it is they need to know about users’ work; users must identify the relevant dimensions and communicate them in ways that give direction to technological innovation. Users are led by the constraints of their experiences to conceptualize improvements in terms of current work; designers are led by their knowledge of possibilities and the desire to contribute to advances in their field to seek more fundamental changes. Consideration of any significant change requires users to extrapolate from current work practice to an imaginary future work practice. The time required to converge on mutual understandings from these two different starting points is extensive, and therefore costly. These costs are particularly difficult for users because the investment does not lead directly to scientific gain in their own discipline."

Weedman says that "[a] second stage when resource investment is heavy and asymmetrical is that of testing. Users have difficulty with the need to repeatedly stress the system when the reward is identifying problems…."

This leads us to evaluate our study methods on the basis of the investments that were made by both the designers and the users and on the perceived benefits received. The following discussion of each of the study methodologies looks at the ROI perspective and changes that could be made to maximize the benefits on both sides.

When the beta tester registration process was conceived, it was not anticipated that so many beta testers would be accepted; the analysis of the data from so many testers was not considered when forming the open-ended questions. If this type of registration is implemented again, the analysis of the data will be considered when the questions are created and only those questions that will yield useful information for ADL will be asked. Registration information would be much more useful also if individual use patterns on the ADL system could be linked to the registration information. Under these circumstances, the self description of the user’s country would give us a picture of the worldwide distribution of users - but we should investigate whether there are automatic ways of collecting this information based on analysis of web traffic patterns. It would be useful to ask about language proficiency to give us an indication of the strength of need for a multi-lingual interface. It would be useful to see if user occupation has a strong correlation to patterns of use. This information would allow us to expand our understanding of the characteristics of different groups of users that we are getting only from our Target User Group activities now. Finding out from first-time users how they discovered ADL (the referral source) would be useful for targeting outreach and judging the effectiveness of various avenues for the presentation of ADL in terms of attracting users. It also satisfies a curiosity that ADL staff has about how word of their project is spreading within the potential user community.

To make the registration data useful, however, multiple choice questions must be provided. The list of valid choices must be carefully chosen to provide the level of detail of the most practical value to ADL design. For example, high-level geographic regions could be listed for selection with a box to enter the county name if we continue to collect geographic location by self-description. The categories developed during the course of the current analysis could be provided for referral source, organization, and occupation, with a box for a more specific entry. The provision of bounded lists for selection will enable a more accurate count for analysis and may also make the registration form easier to fill out. For the user, a return on investment option for us to investigate is posting the cumulative statistics of registered users online so individuals can see how they fit into the whole.

The survey itself has been shown to be a useful and generally well-written instrument. It is undoubtedly a bit too long. Given the difficulty several respondents had in getting the system to produce anything in response to query attempts, such a detailed survey seems even more inappropriate. Also, the reverse-sensed questions (i.e., approval or disapproval) may have been confusing to some respondents, as in disagreeing with statements of disapproval. An important conclusion with respect to the survey is that it will best be used for comparative analysis, such as tracking differences in reactions by different classes of ADL users or tracking the effects of changes to the interface. The absolute level of approval or disapproval is somewhat less informative, given uncertainty about respondents' baseline tendencies to approve or disapprove, to agree or disagree.

The Likert-scale questions provided quantitative data to complement the qualitative data generated by some of the other evaluation methodologies, such as several open-ended questions found on this survey. Quantitative data are often easier to interpret, more easily generalizable, and more useful for analytic comparisons. We believe that the quantitative items paint a more accurate impression of the overall or average reactions to the ADL. On the other hand, unstructured questions allowed respondents to make comments on the ADL system in their own words, providing a richer picture of how some respondents reacted to the system and also more directed suggestions for change

The results from the survey provide hints of characteristics that might be true of a portion of the ADL user population. The quantitative data from the Likert-scale survey questions provides a profile that can be used as a benchmark for future survey results; the qualitative data can be further mined for insights into the ways in which the interface or the system worked or didn’t work from the users’ perspective. However, it is clear the survey approach needs to be either administered so as to obtain data from a targeted set of users, or so that we have a statistically random sample of the user population (or subset) from which inferences can be made from the data collected. To have any hope of getting cooperation from users for this survey structure, the incentives for participation will have to be carefully considered. The survey also will have to be focused on a carefully selected small set of questions that will give both a snapshot of user reactions and longitudinal patterns through the repeated use of the questionnaire (for example, on a quarterly basis or tied to new releases).

From November 1995 to February 1996, when the first set of audio and video recordings were made of Map and Imagery Laboratory (MIL) and ADL use, user expectations and requests for functionality and services were obtained that parallel the findings from the users' session comments and surveys gathered eight months later. The audio and video transcripts were analyzed with the understanding that much can be learned from a telling case and what it makes visible (Stake, 19??, Santa Barbara Classroom Discourse Group, 199?). The different approach of domain analysis for the users' comments led to similar findings. The triangulation this makes possible across different sets of data, and the multiple perspectives one can use to analyze these data, allows one to make visible what may otherwise go unnoticed (Evertson and Green, 19??). The everyday practices may sometimes appear so commonplace as to be invisible. The multiple perspectives applied in these studies, and the different lenses allowing for different ways to view the data, provide opportunities to note areas of need during actual use. These data were collected within circumstances that were practical and real to the users, and were situated within contexts that affect the quality of the interaction. Understanding these practical situations and typical contexts informs future design decisions for the system. [ROI for ADL and users?]

The users who participated in the Target User Groups indicated an appreciation for being asked to participate. They seemed to feel that they were learning something useful to themselves and liked the sense that they were being incorporated into the design process. Despite this enthusiasm for participating, it was still difficult to get them all together for each session; we had a different group each time we gathered. The difficulties with keeping the same people coming to each session can be partly traced to the inevitable problems of scheduling. This was particularly difficult with teachers, The conclusion is that we need to make special arrangements to involve teachers, on their own turf and at times that they can meet outside of their classroom obligations.

The target user groups were also small and local and therefore the information we gathered from them is most appropriately interpreted as case studies and not as truly representative of the large user groups that they were selected to represent. This was compounded by the difficulty of the members of the TUGs really representing their groups at large. They could only be expected to represent themselves in the final analysis. Still, the TUG activity was very productive in giving us a basic set of user scenarios and user requirement statements which the ADL staff used to focus subsequent discussions within the team and with the ADL Design Review Panel which met in February 1997. These are user-grounded requirement statements and scenarios and thus valuable information for system designers.

The development of a model for target user group meetings was another outcome of this activity. This now needs to be tested in other locations to see if it is usable outside of the ADL environment and if complementary or contradictory patterns emerge.

The select group of users who returned beta tester surveys was very favorably impressed with web interface, finding it "wonderful" and "stimulating." They liked the tutorial and the ease of exploring the system. The terminology used in the interface was not a problem for them. They judged the screen layouts to be well done. The Map Browser, the Catalog, the Footprints, and the exit procedures all got high marks. They agreed that the needs of experienced users were taken into account. The free-text comments contained more negative comments than was indicated by the Likert scale responses.

In the TUG sessions, the interface ideas that surfaced (even though it was not a specific topic scheduled for discussion) covered ideas for a future interface as well as reactions to the current web interface. Reactions to the current interface led to requests for better tutorials, better ability to track where you are in the system, better quality control of the metadata, and simplified pages - especially the first page. Ideas for the future included user profiles, applet forms, context notes, and query by example.

Other good ideas for enhanced search capability and search help that came from the groups included more support in terms of customized screens, canned queries, intelligent assistance, and more explicit directions about what to do next. The idea that the system needs to be backed up by personal attention to help users work through their problems received general support. Simpler search methods (natural language searching and the use of icons for types of data) were requested. They also requested the ability to sort the result sets and to have support for iterative searching - that is, moving easily from a previous query to a new one.

It is apparent from the more experienced users' interactions and comments during videotaping sessions that the interface does provide functions and services that allow access to ADL's holdings. However, even for the more proficient users there are problems. For example, among the developers, there was confusion about the consequences for the ADL interface of hitting the "back" and "stop" buttons on the web browser. For the limited proficiency users, the web pages contained too much information, the options to customize were confusing, and it took too long to reload pages once a command was sent. For the less proficient users just getting to the data was problematic. It was difficult to discern the purpose of the ADL, what holdings were actually available, or what could be done with the data even if the user was able to search and retrieve a particular set. For all levels of use, there was the consistent request for the human connection, a way for the user to address another person. Users also suggested an 800 number helpline, an email address, FAQs, and improved context-sensitive help (the help pages in the currently available system sometimes do not match the pages to which they are supposed to be linked).

Many of the problems identified may be seen partly as resulting from ADL's failure to express its purpose and potential adequately in the interface. The intended audience of the prototype and the knowledge and skills needed to use ADL successfully are not obvious. It is clear that many of the users studied did not understand what ADL was trying to do and where it is in the development process.

For content, each target user group had a different selection of resources that they would like to see in ADL. Historical coverage was mentioned in some way in each group. Both the Earth Scientists and the Information Specialists mentioned aerial photographs. The Education Group, whose members didn't know as much about what is available, mentioned less technical sets of information. The content list of the Earth Scientist group confirmed the choice of sets to add to the library already made by the ADL staff.

Ideas for retrieval functions focused on: the delivery of an end product to users (printed copies and electronic copies); the ranking of retrieved sets by some parameter; being able to use the results of previous searches to build new queries or combine result sets; and on being able to visualize the result sets and browse items in the sets.

The TUG discussion on processing focused on: various customization capabilities, such as extracting portions of an image or combining information from various sources to build a map; the availability of GIS functionality and modeling software; format conversion; and on the architectural design problem of where this type of software should "live" - at the server, at the client, or at a third-party vendor.

To all but the most experienced users of ADL, the functionality is not readily apparent. Depending on the users' proficiency level, they will access more or less of the functionality, experiencing greater or lesser frustration. Those who are able to access a variety of the system's functions are sometimes even more frustrated than those who do not know the functions exist. This problem may be resolved as ADL is able to load more complete datasets and increase the overall contents of the system.

A major problem has been that ADL is a research project that must yet result in an actual digital library - and yet funds are not to be expended to build that actual digital library. A dearth of content (both actual data and metadata) has made ADL supremely uninteresting for any users other than those who are interested in a relatively small portion of southern California. Until the content has increased dramatically (metadata alone needs to be at about 1,000,000 records before it will be, from a spatial-data user’s viewpoint, very useful), users will continue to be frustrated trying to use ADL to find something useful for their purposes.

The immediate implications from these findings are that the ADL project needs to put a high priority on increasing the contents of the system and on providing a redesigned web interface to access those contents. For digital library design in general, the findings suggest that, although digital libraries provide a different environment and support new uses, they are still perceived as social environments and therefore should maintain a sense of person-to-person connection.

Possible points for conclusion of paper --

ADL Design Issues

* Usefulness of Evaluation Activities to Design

* Evaluation Cycle with Changing Prototypes

* Designing for Target User Groups

* How-to: Development Processes that Work for Interface Design

* Should DLIs Attempt a Federated UI?

Bibliography [not complete]

Deckelbaum, David A. 1989. A user survey conducted in the henry J. Bruman Map Library, university of California, Los Angeles, Fall 1988 [submitted in partial fulfillment of the requirements for the MLS degree, Graduate School of Library and Information Science, UCLA]. Western Association of Map Libraries. Information bulletin 20:170-96.

Larsgaard, M. 1987. User studies. Pp. 223-24 in: Map librarianship, an introduction. 2d ed. Littleton CO: Libraries Unlimited.

Leavey, M. D.; and Moore, E. E. 1988. I&R in an academic library [Helms-Cravens Library, Western Kentucky University]. Reference librarian 21:109-19.

Leeuwenburg, J. 1982. Map reference work in an academic library. Globe 18:9-18.

Mittra, D. K.; and Ghatak, A. K. 1993. Map collection of the National Library [of India] and its users’ pattern. INSPEL 27(2):65-72.

Perry, J. M. 1992. Use of map collections by genealogists, responses to a survey. Special Libraries Association. Geography and Map Division. Bulletin 170:22-31.

Seavey, C. A.; and Rex, H. F. 1992. Users and geographic areas of interest in an academic library map collection, 1983-1989, implications for policy development [at the University of New Mexico]. Meridian 7:15-25.

Treude, M. 1975. Reference services with maps. Special Libraries Association. Geography and Map Division. Bulletin 102:24-29.

Treude, M. 1981. Map library users in an academic setting. Library trends 29:453-71.

********************** Citations ***********************************

@INPROCEEDINGS{And95,

AUTHOR = "D. Andresen and L. Carver and R. Dolin and

C. Fischer and J. Frew and M. Goodchild and

O. Ibarra and R. B. Kemp and R. Kothuri and

M. Larsgaard and B. S. Manjunath and D. Nebert and

J. Simpson and and T. R. Smith and A. Wells and

T. Yang and Q. Zheng",

TITLE = "The {WWW} {P}rototype of the {A}lexandria {D}igital

{L}ibrary",

BOOKTITLE = "Proceedings of the International Symposium on Digital

Libraries",

ADDRESS = "Tsukuba, Japan",

YEAR = 1995

}

@ARTICLE{Cac95,

AUTHOR = "CACM",

TITLE = "Digital {L}ibraries (section on {D}igital {L}ibrary

{I}nitiatives",

JOURNAL = CACM,

VOLUME = 38,

NUMBER = 4,

PAGES = "57--64",

MONTH = apr,

YEAR = 1995

}

@INCOLLECTION{Ern72,

AUTHOR = "Ernst, R. and Yovits, M.",

TITLE = "Information Science as an Aid to Decision-Making",

BOOKTITLE = "Decision-Making: Creativity, Judgment [sic], and Systems",

EDITOR = "Brinkers, H.",

PUBLISHER = "Ohio State University Press",

YEAR = 1972

}

@ARTICLE{Fox95,

AUTHOR = "Fox, E. and others",

TITLE = "Digital Libraries",

JOURNAL = "Communications of the ACM",

VOLUME = 38,

NUMBER = 4,

MONTH = apr,

YEAR = 1995

}

@MANUAL{Nsf93,

ORGANIZATION = "National Science Foundation",

TITLE = "Research on Digital Libraries, Program Guideline NSF 93-141",

ADDRESS = "Washington, D.C.",

NOTE = "http://www.nsf.gov\-/ftp\-/CISE\-/program\-/nsf93141.txt",

MONTH = sep,

YEAR = 1993

}

@INCOLLECTION{Sug95,

AUTHOR = "Sugar, W.",

TITLE = "User-Centered Perspective of Information Retrieval

Research and Analysis Methods",

BOOKTITLE = "Annual Review of Information Science and Technology",

VOLUME = 30,

EDITOR = "Williams, M.",

PUBLISHER = "Information Today, Inc.",

YEAR = 1995

Alexandria Digital Library Prototype CD. (1995). [Redlands, CA: Environmental Systems Research Institute, Inc; [Alexandria Digital Library, UCSB].

Buttenfield, B.P. (babs@colorado.edu). Personal communication. Please contact her directly for more information about the ADL log analysis results.

Buttenfield, B.P. & Kumler, M.P. (1996). Tools for browsing environmental data: The Alexandria Digital Library Interface. Proceedings of the Third International Conference on Integrating Geographic Information Systems and Environmental Modeling. Santa Fe, New Mexico, January 21-26, 1996. Also available at http://www.ncgia.ucsb.edu/conf/SANTA_FE_CD-ROM/sf_papers/buttenfield_babs/babs_paper.html.

Marchionini, Gary. (1995). Information seeking in electronic environments. Cambridge University Press.

Mulaik, S.A. (1972). The foundations of factor analysis. New York: McGraw-Hill.

Norman, Donald A. (1988). The psychology of everyday things. New York, Basic Books.

Weedman, Judith. (August, 1996). The structure of incentive: Design and client roles in application-oriented research. Copy of paper submitted for publication. 33 pages.

Yankelovich Cybercitizen survey