Product Tested: Summon 2.0, branded Articles+. Tested the preview version of Summon 2.0 at Yale http://yale.preview.summon.serialssolutions.com/#!/ now live as http://yale.summon.serialssolutions.com
Report written by Kathleen Bauer email@example.com
Dates: Tests were run from April 28 through May 5, 2014
Locations: Robert B. Haas Family Arts Library, Anne T. and Robert M. Bass Library, Thain Family Cafe in Bass Library, and the Center for Science and Social Science InformationI, all Yale University.
Students were approached and asked if they would participate in a 30 minute test of a library interface, and were given a small gift card for their time. Tests were run by three person teams, consisting of a recruiter, facilitator, and observer. Observers recorded actions/words of participants. No software was used to record sessions. Tests were designed to test Articles+ for finding images, finding librar subject guides, relevancy of results, limitting to scholarly material and emailing a citation.
40% of undergraduate students and all graduate students in this test did not recognize Articles+ and had not used it (Articles+, Summon 1.0 has been live since January 2014). After using it, 90% of undergraduates and 70% of graduate students reacted positively to Articles+ and stated they would use it again.Students liked that they knew they were searching library resources, which gave the results some authority beyond what Google offers. Students valued the facets as a way to narrow results, and were comfortable that Articles+ results were appropriate material collected by the library. Negatives were that the interface was viewed as difficult by some, especially hovering over results and selecting from the Topic Explorer (third pane on right in display).
All test participants were able to search on a topic of their choice, and 75% thought results were relevant or what they expected to see. 85% of participants were able to successfully use Articles+ to find images, although some pointed out they preferred how Google Images functions. In both images and topic searching participants used Content (especially Articles and Books) and Discipline facets. Despite 70% of participants using facets at some point in testing, only 15% successfully used the Peer Review limit, even for a question suited to its use. The subject librarian and associated guide recommendations displayed in the Topic Explorer were problematic because of display issues that can happen on mobile devices with small screens and when a facet has been selected This is a problem Serials Solutions should examine and determine if they can improve how the Topic Explorer (third pane of results on the far right) is used. Guides embedded in result sets were more successfully located by participants than those in the Topic Explorer (67% versus 18% success rates).
The library should promote Articles+ to undergraduates who need a general search tool, to overcome the lack of knowledge of this tool and to help them use it effectively. Exposing them early in their career, especially if Articles+ works well, may help them to develop a different relationship with the Library and Library resources as a good alternative to Google. To help librarians with promotion, we suggest that cheat sheet and instructor tips be gathered and made available to staff. More tips are discussed more under What Library Staff Should Know.
Serials Solutions should address issues around display of the Topic Explorer pane. It is sometimes unavailable, which is problematic when important material is displayed there. Other issues that Serials Solutions should explore are an easier way to deselect facets, and make images more obvious in search results, especially when words such as images or photographs are included in a search. See the complete list of Recommendations.
Participants were ten graduate students, post docs and staff, and ten undergraduates, with majors/departments Early Modern European History, English (2), Astronomy(2), Environmental Science(2), Management, Geology and Geophysics, Physics (2), Political Science (2), History of Science, Music, Biology (2), Spanish, Literature and undeclared.
When asked if they had used Articles+ in the past, 30.0% (6), answered Yes. No graduate students reported using Articles+.
Participants reported the research databases they typically use.
85% (17 of 20)
Topic search perceived relevancy
75% (15 of 20)
Limit to scholarly articles
15% (3 of 20)
Email a citation
60% (12 of 20)
Find a subject guide (in the Topic Explorer pane,recommended resource)
18.2% (2 of 11)
Find a subject guide within results
66.7% (6 of 9)
To start, I’d like you to imagine you are preparing a class presentation for which you need several very high-quality images of the Notre Dame Cathedral in Paris, and use Search Articles+ to find one or more images for that purpose. Do you see anything in the list of results you might use?
Success Rate 85% (17 participants )Success defined as a participant executing a search and finding the horizontal group of images in the list of results.
Participants used a variety of methods after searching. 20% (4) included the term images in the search, 25% (5) used Advanced Search, 35% (7) used the Content Type facet for Image and 20%(4) used the Discipline facet Visual Arts.
Before noticing the images in search results, more than half of the participants used facets to narrow the search results.
Have you recently looked for articles on a topic for a class assignment or research? What was that topic? Where did you look for articles?
I would like you to go to Search Articles+ and look for some articles on your topic.
Suggestions if you don’t have a topic
Participants searched a range of topics of their choice.
2a. Please take a moment to look at the list of results. Do they look appropriate to your topic?
Success: 75% (15) thought results looked appropriate or relevant.
2b. Let’s assume you need scholarly articles only for a class paper. Can you find a way to get rid of any popular material and news items?
Success: 15% (3) Success was defined as the selection of the Peer Review facet. 70% (14) chose instead to use a Content Type facet for Journal Articles and/or Dissertations.
Most participants ignored the Peer-Review limit on the upper left and instead chose facets for Content Types Journal Articles and/or Dissertations.
2c. In your work, have you ever needed to send a citation to yourself? We’d like to see if you can use this interface to do that. Show me how you would select a result and email it to yourself or a colleague.
Success Rate: 60% (12) Success was hovering over a record so that the email option displayed in the right pane and then selected Email.
Some participants could not complete this task because of display problems with the third, rightmost display pane. In some cases previous selections of a facet “more” option caused the third pane to not be available, and other times the pane did not display because of the screen resolution. In addition some participants found it difficult to hover over a record and view the options.
When a facet is expanded, the third pane is not available until the facet is closed.
Please show me how you would find some basic, background information about the discipline of ethnomusicology. Take a minute to look at the display of results. Do you see anything on the page that you might consult if you needed to find resources at the Yale Library?
Success Rate 18.2% (2 of 11) Success was finding the subject guide recommended by Articles, displayed in the Topic Explorer, the third or right-side display panel.
A variation was asked substituting colonial Mexico for ethnomusicology.
Success Rate 66.7% (6 of 9) Success was discovering and selecting the guide for Colonial Mexico embedded in the list of results.
The ethnomusicology guide recommendation appeared in the third panel as a recommendation, whereas the Colonial Mexico guide was integrated in results. In this case display in the third pane caused the same issues as discovered in question 2, in that sometimes the third pane was not available.
In this image, the Yale University Library subject guide is recommended in the right display pane.
In this case a facet list has been expanded, removing the display pane where the guide had been recommended.
The guide for Colonial Mexico is displayed in the results, so the expanded facet causes no problem with display.
After using Articles+ in the test, participants were asked if they would use Articles+ again for their work, and if they considered it a possible alternative to using Google. Overall attitudes toward Articles+ were positive, although more so among the undergraduates.
80% indicated they would use Articles+ again, with 90% of undergraduates saying they would use it again.
75% of participants viewed Articles+ as a possible alternative to Google or Google Scholar, with 90% of undergraduates viewing it that way, but only 60% of graduate students.
In Google results including the term images causes the result display to change so that images are more prominent. In addition options at the top of results let users search search results by Images and News.