Competence vs. Confidence: Assessment Knockdown!

ACRL Conference, Philadelphia PA, April 1, 2011

Amy R. Hofer, Portland State University | Margot Hanson, Golden Gate University

    ahofer@pdx.edu | mhanson@ggu.edu

Poster online at http://www.slideshare.net/muumuulibrarian/competence-vs-confidence-assessment-knockdown

Online handout at http://tinyurl.com/assessmentknockdown

Learning assessment tools and program evaluations tools aren’t interchangeable. What do satisfaction surveys or self-assessments tell us about student learning? Most people aren’t self-aware and students definitely fit that description. Competency theory suggests an inverse relationship between competence and confidence, but our assessment results show no relationship whatsoever!

Our poster will visually demonstrate the difference between conducting learning assessment and program evaluation. We will draw attention to utility of each type of assessment in producing meaningful evidence for the value of our instruction programs. This information is crucial for making strategic decisions about assessment: which studies are producing valid, usable results, and which ones can we afford to give up?

From the results of a learning assessment paired with self-evaluation at a business school, we show that there is not much relationship at all between what students think they know, and what they actually know when it comes to research skills. Applying the concepts of competency theory helps us argue that students are not good at judging whether they need research instruction.  Seeing this laid out in a graphic format also communicates in a high-impact way that you can’t measure student learning with a satisfaction survey.

For more information about our study, please see:

 Hofer, A.R., & Hanson, M. (2010). Upstairs-Downstairs: Working with a campus assessment coordinator and other allies for effective information literacy assessment. In 2010 CARL Conference Proceedings. Available from http://carl-acrl.org/Archives/ConferencesArchive/Conference10/2010proceedings/Amy-Hofer.pdf


Table of Contents:

I. Key Definitions

II. Identify Study Objectives

III. References

IV. Learning Assessment Quiz

V. Image credits from our poster


I. Key Definitions

 


II. Identify Study Objectives


III. References:

Gross, M., & Latham, D. (2009). Undergraduate Perceptions of Information Literacy: Defining, Attaining, and Self-Assessing Skills. College & Research Libraries, 70(4), 336-350. doi:VL - 70

Competency theory offers one potential explanation for why students who are not information literate report a high level of confidence in their ability to seek, evaluate, and use information. According to research performed in the domain of psychology, people who are incompetent, particularly in areas in which people commonly have some orientation, tend to believe that their skills are above average and to overestimate their performance on a skills test (p. 337).

Kruger, J., & Dunning, D. (1999). Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments. Journal of Personality and Social Psychology 77(6), 1121-34.

In sum, we present this article as an exploration into why people tend to hold overly optimistic and miscalibrated views about themselves. We propose that those with limited knowledge in a domain suffer a dual burden: Not only do they reach mistaken conclusions and make regrettable errors, but their incompetence robs them of the ability to realize it. Although we feel we have done a competent job in making a strong case for this analysis, studying it empirically, and drawing out relevant implications, our thesis leaves us with one haunting worry that we cannot vanquish. That worry is that this article may contain faulty logic, methodological errors, or poor communication. Let us assure our readers that to the extent this article is imperfect, it is not a sin we have committed knowingly (p. 1132).

Ragains, P. (1997). Evaluation of academic librarians' instructional performance: Report of a national survey. Research Strategies, 15(3), 159-175.

Gathering responses from students concerning their satisfaction with library instruction emerged as the most frequently-used type of evaluation. In contrast, students are actually tested on their knowledge and application of library and information search concepts only about half as often as they are asked their general impressions of instructional presentations and materials provided by librarians (p. 164).

When library instruction is to be evaluated in a formal manner, the purposes for doing so must be clearly defined and understood by librarians who provide the service, their supervisors, and administrators. Distinctions must be made between efforts to measure learning outcomes for students (normally determined by test results), the overall instructional program, or the performance of individual librarians (p. 169).

Sinkavich, F. J. (1995). Performance and metamemory: Do students know what they don't know?. Journal of Instructional Psychology, 22(1), 77-87.

Results indicate that students who demonstrated a high mean degree of confidence in their answers have higher examination scores when compared to those students who expressed a low mean degree of confidence in theft answers. Results also indicate that a significant difference exists between good and poor students in their ability to predict what they know and do not know. Good students were better able to predict their test item performance when compared to poor students. This result implies that good students have better metamemory accuracy than do poor students on multiple choice examinations. Finally, results indicate that through the use of replacement items students can significantly improve their scores on a test, or their standing in the class (p. 77).

back to top


IV. Learning Assessment Quiz (key):

 

Objectives:

A. Identify appropriate sources (for example, start research with reference sources and overviews to build context; use trustworthy websites; find substantive, objective articles)

B. Ask a great question that fits the scope of the assignment and requires research and analysis to answer

C. Demonstrate academic integrity by correctly citing sources

D. Use the library’s resources and services as part of the research/writing process

E. About/demographic

 

Questions (correct answer is bold)

1. If you have just found 5 sources for your research paper, what is the best next step to take? (A)

a. Scan (pre-read) a little bit of what you already have to find out whether you need more information

b. Brainstorm new topics in case you want to change your mind

c. Search for 5-10 more articles

d. Print everything out

e. I don’t know

2. When should you use a citation in your paper? (C)

a. When you directly quote a passage of less than 40 words

b. When you restate the author’s idea using your own words

c. When you use a long section in a block quote

d. All of the above

e. I don’t know

3. Which of the following topics is a great idea for a research paper? (B)

a. How has the Cash for Clunkers stimulus program affected the economic well-being of single mothers who lost their homes in Hurricane Katrina?

b. Cancer

c. Is communication the key to good management skills?

d. Is it ethical to target advertising to very young children in order to build long-term brand loyalty?

e. I don’t know

4. Your professor has assigned a research paper and part of your grade is to demonstrate critical thinking skills.  Which statement below describes the expectations for this assignment? (B)

a. You will find a way to criticize the information you found in your research

b. You will use your research as evidence for an analysis of a problem

c. You will hand in a report on the information you found in your research.

d. You will use only original ideas – nothing in the paper should come from an outside source.

e. I don’t know

5. Can you cite a web page that does not list an author or date of publication? (C)

a. Yes, move the title to the beginning of the citation and write “(n.d.)” for the date

b. Yes, you should cite it in your paper but not include a reference list entry for it

c. No, you have to use sources that have authors and dates

d. This kind of information is in the public domain and does not need to be cited

e. I don’t know

6. Which of the following is the most authoritative (reliable) source of information for finding key facts about a company? (A)

a. Advertisement paid for by the company’s competitor

b. Mission statement and annual report from the company’s website

c. Press release written by a public relations firm

d. Article on Wikipedia

e. I don’t know

7. At what point in a project are you most likely to ask for help? (D)

a. Beginning

b. Middle

c. Last minute

d. Never

e. I don’t know

8. Why should you cite your sources? (C)

a. If you don’t cite, you’re stealing somebody else’s words or ideas

b. So that your professor can find the source of the information you used

c. Because it is the expectation of the academic community that scholars build on the ideas of others

d. All of the above

e. I don’t know

9. Imagine that you have been assigned to research an industry.  Which option below is NOT a good first step in your research process? (A)

a. Use a library database to find an industry report

b. Check the library’s online reference collection

c. Look on the library shelves for recent journal articles

d. Talk to a librarian about how to get started

e. I don’t know

10. You were assigned a 5-7 page research paper and decided to write about the current economic crisis.  Which is the best strategy for narrowing your idea down to a manageable topic that fits the assignment? (B)

a. Use the topic of a Harvard Business Review case study

b. The current economic crisis is a manageable topic that does not need to be narrowed down

c. Google your topic

d. Using what you already know about the topic, brainstorm about the “5 W’s”: who, what, when, where, and why

e. I don’t know

11. What is the most important difference between searching for information on Google vs. in the library’s databases? (D)

a. Google always gives you the full text of the resource

b. Library databases give you access to in-depth information not available on Google

c. Google is better because you will always get more hits

d. You don’t need to evaluate the information you find in library databases because it is always scholarly

e. I don’t know

12. How many times during the semester do you get help from a librarian? (D)

a. Never

b. One time        

c. 2-5 times

d. More than 5 times

e. I don’t know

13. Please rate how strong your own research skills are right now. (E)

a. Excellent

b. Good

c. Fair

d. Poor

e. I don’t know

14. Do you think that the library and its services contribute to your academic success at GGU? (E)

a. Yes

b. Somewhat

c. Maybe

d. No

e. I don’t know

15. Have you had training in how to do research before?  Please check all that apply. (E)

a. I have taken a credit course on research skills

b. I have taken a class where a librarian made a visit of 1 hour or less

c. I have completed the PLUS program

d. I learned on my own

e. This is new to me

back to top


V. Images on our poster are from clipart.com 

back to top