Yale University Library

Report on Articles+ (Summon 2.0) Usability Testing

May 23, 2014

Product Tested: Summon 2.0, branded Articles+. Tested the preview version of Summon 2.0 at Yale http://yale.preview.summon.serialssolutions.com/#!/ now live as http://yale.summon.serialssolutions.com

Report written by Kathleen Bauer kathleen.bauer@yale.edu

Dates: Tests were run from April 28 through May 5, 2014

Locations:  Robert B. Haas Family Arts Library, Anne T. and Robert M. Bass Library, Thain Family Cafe in Bass Library, and the Center for Science and Social Science InformationI, all Yale University.

Students were approached and asked if they would participate in a 30 minute test of a library interface, and were given a small gift card for their time. Tests were run by three person teams, consisting of a recruiter, facilitator, and observer. Observers recorded actions/words of participants.  No software was used to record sessions. Tests were designed to test Articles+ for finding images, finding librar subject guides, relevancy of results, limitting to scholarly material and emailing a citation.

Report Contents

Summary

Test Participants

Prior Use of Articles+

Databases Used for Research

Test Question Results

Brief Summary

Question 1 Image Search

Question 2 Topic Search

Question 3 Find a Subject Guide (Libguide)

Debriefing

What Was Successful

What Was Less Successful

Most students didn’t know about Articles+.

What Library Staff Should Know

Recommendations

Summary

40% of undergraduate students and all graduate students in this test did not recognize Articles+ and had not used it (Articles+, Summon 1.0 has been live since January 2014).  After using it, 90% of undergraduates and 70% of graduate students reacted positively to Articles+ and stated they would use it again.Students liked that they knew they were searching library resources, which gave the results some authority beyond what Google offers. Students valued the facets as a way to narrow results, and were comfortable that Articles+ results were appropriate material collected by the library.   Negatives were that the interface was viewed as difficult by some, especially hovering over results and selecting from the Topic Explorer  (third pane on right in display).

All test participants  were able to search on a topic of their choice, and 75% thought results were relevant or what they expected to see.  85% of participants were able to successfully use Articles+ to find images, although some pointed out they preferred how Google Images functions. In both images and topic searching participants used Content (especially Articles and Books) and Discipline facets. Despite 70% of participants using facets at some point in testing, only 15% successfully used the Peer Review limit, even for a question suited to its use.  The subject librarian and associated guide recommendations displayed in the Topic Explorer were problematic because of display issues that can happen on mobile devices with small screens and when a facet has been selected  This is a problem Serials Solutions should examine and determine if they can improve how the Topic Explorer (third pane of results on the far right) is used. Guides embedded in result sets were more successfully located by participants than those in the Topic Explorer (67% versus 18% success rates).

The library should promote Articles+ to undergraduates who need a general search tool,  to overcome the lack of knowledge of this tool and to help them use it effectively. Exposing them early in their career, especially if Articles+ works well, may help them to develop a different relationship with the Library and Library resources as a good alternative to Google. To help librarians with promotion, we suggest that cheat sheet and instructor tips be gathered and made available to staff. More tips are discussed more under What Library Staff Should Know.

Serials Solutions should address issues around display of the Topic Explorer pane. It is sometimes unavailable, which is problematic when important material is displayed there. Other issues that Serials Solutions should explore are an easier way to deselect facets, and make images more obvious in search results, especially when words such as images or photographs are included in a search. See the complete list of Recommendations.

Test Participants

Participants were ten graduate students, post docs and staff, and ten undergraduates, with majors/departments Early Modern European History, English (2), Astronomy(2), Environmental Science(2), Management, Geology and Geophysics, Physics (2), Political Science (2), History of Science, Music, Biology (2), Spanish, Literature and undeclared.

Prior Use of Articles+

When asked if they had used Articles+ in the past, 30.0% (6), answered Yes. No graduate students reported using Articles+.

Databases Used for Research

Participants reported the research databases they typically use.

Test Question Results

Brief Summary

Question

Success Rate

Image Search

85% (17 of 20)

Topic search perceived relevancy

75% (15 of 20)

Limit to scholarly articles

15% (3 of 20)

Email a citation

60% (12 of 20)

Find a subject guide (in the Topic Explorer pane,recommended resource)

18.2% (2 of 11)

Find a subject guide within results

66.7% (6 of 9)

Question 1 Image Search

To start, I’d like you to imagine you are preparing a class presentation for which you need several very high-quality images of the Notre Dame Cathedral in Paris, and use Search Articles+ to find one or more images for that purpose. Do you see anything in the list of results you might use?

Success Rate 85% (17 participants )Success defined as a participant executing a search and finding the horizontal group of images in the list of results.

Participants used a variety of methods after searching. 20% (4) included the term images in the search, 25% (5) used Advanced Search, 35% (7) used the Content Type facet for Image and 20%(4) used the Discipline facet Visual Arts.

2014-05-13_1209.png

Before noticing the images in search results, more than half of the participants used facets to narrow the search results.

Question 2 Topic Search

Have you recently looked for articles on a topic for a class assignment or research? What was that topic? Where did you look for articles?

I would like you to go to Search Articles+ and look for some articles on your topic.

Suggestions if you don’t have a topic

  • CEO compensation
  • effects of climate change on developing nations
  • flash trading
  • graphene batteries
  • attention deficit disorder

Participants searched a range of topics of their choice.

2a. Please take a moment to look at the list of results. Do they look appropriate to your topic?  

Success: 75% (15) thought results looked appropriate or relevant.

 

2b. Let’s assume you need scholarly articles only for a class paper. Can you find a way to get rid of any popular material and news items?

Success: 15% (3) Success was defined as the selection of the Peer Review facet. 70% (14) chose instead to use a Content Type facet for Journal Articles and/or Dissertations.

2014-05-13_1222.png

Most participants ignored the Peer-Review limit on the upper left and instead chose facets for Content Types Journal Articles and/or Dissertations.

 

2c. In your work, have you ever needed to send a citation to yourself? We’d like to see if you can use this interface to do that. Show me how you would select a result and email it to yourself or a colleague.

Success Rate: 60% (12) Success was hovering over a record so that the email option displayed in the right pane and then selected Email.

Some participants could not complete this task because of display problems with the third, rightmost display pane. In some cases previous selections of a facet “more” option caused the third pane to not be available, and other times the pane did not display because of the screen resolution. In addition some participants found it difficult to hover over a record and view the options.

2014-05-13_1232.png

When a facet is expanded, the third pane is not available until the facet is closed.

Question 3 Find a Subject Guide (Libguide)

Please show me how you would find some basic, background information about the discipline of ethnomusicology. Take a minute to look at the display of results. Do you see anything on the page that you might consult if you needed to find resources at the Yale Library?

Success Rate 18.2% (2 of 11) Success was finding the subject guide recommended by Articles, displayed in the Topic Explorer, the third or right-side display panel.

A variation was asked substituting colonial Mexico for ethnomusicology.

Success Rate 66.7% (6 of 9) Success was discovering and selecting the guide for Colonial Mexico embedded in the list of results.

The ethnomusicology guide recommendation appeared in the third panel as a recommendation, whereas the Colonial Mexico guide was integrated in results. In this case display in the third pane caused the same issues as discovered in question 2, in that sometimes the third pane was not available.

2014-05-13_1522.png

In this image, the Yale University Library subject guide is recommended in the right display pane.

2014-05-13_1523.png

In this case a facet list has been expanded, removing the display pane where the guide had been recommended.

The guide for Colonial Mexico is displayed in the results, so the expanded facet causes no problem with display. 2014-05-13_1524.png

Debriefing

After using Articles+ in the test, participants were asked if they would use Articles+ again for their work, and if they considered it a possible alternative to using Google. Overall attitudes toward Articles+ were positive, although more so among the undergraduates.

80% indicated they would use Articles+ again, with 90% of undergraduates saying they would use it again.

75% of participants viewed Articles+ as a possible alternative to Google or Google Scholar, with 90% of undergraduates viewing it that way, but only 60% of graduate students.

What Was Successful

  • Students naturally and easily used facets, especially content type and disciplines.
  • Students liked that they could search for things they know the library owns.Results were perceived as “library approved” and that was a positive for Articles+ over Google.
  • It was easy to  get results, and most students were satisfied with the relevance of results.
  • Articles+ appeals to undergraduates, and they voiced enthusiasm for using it.
  • Students said they wished they had known about Articles+.

What Was Less Successful

  • Most students didn’t know about Articles+.

  • Students who were already familiar with search tools were less receptive to a new search tool such as Articles+.
  • Image thumbnails were not consistently placed or visible in search results.
  • The scholarly articles/peer-reviewed facet failed to work for students. It is unclear if wording played a role in this.
  • Students expected the Yale University Library logo in the header to return them to the main search screen, but it took them to the library’s home page.
  • The far-right display pane disappeared during testing for a variety of reasons, compromising usability of the interface.
  • There was some trouble getting to local experts because of the far-right display pane issue. It was better when the librarian/Libguide displayed within results.
  • Students were not aware what was in Articles+, and specifically did not know that Orbis material was not in Articles+. The Yale University Library logo on the search entry screen seemed to signify that everything in the library was searched, a la Google.
  • Top-ranked results were sometimes too old to be considered useful or relevant.
  • Icons were confusing (e.g. folder icon).
  • Some students wanted an Advanced Search and could not find it.

What Library Staff Should Know

  • Articles+ is not a well known resource, and needs to be promoted. We should explain it as being one part of the larger search environment, which includes everything from Google to specialized research databases.
  • Facets are sticky. Users should be warned that any facets/limits selected stay in effect throughout a search.
  • Articles+ offers citation and email functionality which can enhance use of this search tool.
  • Metadata can change how material is discovered. Summon discipline metadata can and should be added to Libguides.
  • Facets are available for Peer Review and Scholarly Material in Articles+. These limits are defined by Serials Solutions:

2014-05-20_1153.png

  • Material indexed in Articles+ includes images from Artstor, articles from journals, articles from popular magazines, news items (but not today’s news), books from HathiTrust and Ebrary. Orbis records are not searched in Articles+.
  • Screen resolution can be an issue. Check resolution on a monitor before you teach a class. You may need to adjust the resolution to see the third pane of information. Users of smart phones may not see the right pane either, depending in screen size.
  • When the right pane is not available, a small plus sign is displayed near a record, which will display email and citation options when selected.

  • When not on campus and not using VPN you can search SOME but not all of the Articles+ index. This material requires authentication to display:

2014-05-20_1215.png

Recommendations

  • Serials Solutions--Switch suggested librarian with encyclopedia article abstract (in the Topic Explorer right pane)
  • Serials Solutions--Address issues of the right pane disappearing (with loss of functionality) when the screen is too small or a facet is expanded.
  • Serials Solutions--Facets are sticky (remain in effect from search to search). It should be very obvious when limits are in effect. One suggestion would be to include a fairly prominent limit selection breadcrumb, making it simple to deselect one or more limit. Alternative would be one prominent Clear Limits option.
  • Serials Solutions--When a term such as images is in the search the row of images should be at the very top of search results.
  • Serials Solutions--Users wanted an easy option to search only images. Consider providing tabbed search results for News and Images, similar to Google use of horizontal options for search results.

In Google results including the term images causes the result display to change so that images are more prominent. In addition options at the top of results let users search search results by Images and News.2014-05-20_0908.png

  • Serials Solutions--make Advanced Search more prominent for power users.
  • Use instruction to increase knowledge of Articles+. Provide one-on-one instruction tailored to research needs.
  • In instruction discuss:
  • Limits for peer reviewed and/or scholarly material.
  • Images in Articles+ and the ArtStor database.
  • Facets and how they work.
  • How authentication or lack of authentication affects search results.
  • Material covered in Articles+--journal articles, newspapers, reference works, ebrary and HathiTrust (but not all ebooks), images.
  • How the right information pane in search results works.
  • Create a reference cheat sheet for librarians who will present or teach about Articles+.
  • Exclude outside Libguides currently displayed in results.
  • Make subject guides more findable with added metadata.
  • Make the link to Articles+ more prominent on the library’s home page and elsewhere. Articles+ search can be embedded in Libguides.
  • Add some information to the entry page to indicate what is searched and what is excluded.
  • Add text in search box “Search journals, dissertations, newspapers, and more…”
  • Provide a more prominent link to search Orbis in the Articles+ interface, especially since Orbis records are not searchable through Articles+.
  • Add an obvious link in the header back to the Articles+ entry page.
  • Do an additional round of testing in the fall 2014. Record sessions so that others in the library can view them.

Page