1 of 15

Using a Rubric to Assess OER Quality: A Guided Example

2 of 15

AN INTRODUCTION

In this presentation, the Assessing OER Quality Rubric will be used to score an open educational resource as an example of how to use the rubric.

3 of 15

RUBRIC OVERVIEW

Items	1	2	3	4	5	Score
Maintenance Poster �	Almost non-functional. Broken links and outdated designs. Issues in modern browsers or machines. Minimal or non-existent multimedia. No dates or authors given.	Functional resources with most working links and downloads. Meets basic design and technical needs. Limited issues with modern browsers or machines. Some outdated standards for visuals or presentation.	Resources exceed basic functional standards. Majority of links and downloads work. Little to no issues with modern platforms. Updates or publication dates are clear. Satisfies presentation and visual display standards.	Resources show at least one sign of updating past publication. Links, visuals, and downloads all work on modern platforms.	Updated within the last two years. Working links and graphics. Works on modern platforms. Modern and mobile-friendly design. Exceeds presentation and visual display standards.	�� ___/5
Adaptability Poster �	The material/site does not include a statement of copyright. Users cannot engage in any of the following actions: Reuse, Retain, Revise, Remix, Redistribute (5R’s)	The material/site contains a statement of copyright, but copyright may be hard to locate, incomplete, or vague. Copyright does not clearly specify if users are allowed to engage in the 5R’s	The material/site contains a full, clear statement of copyright and specifies the ways in which the material can be adapted. Copyright allows users to engage in ONE or TWO of the 5R’s	The material/site contains a full, clear statement of copyright and specifies the ways in which the material can be adapted. Copyright allows users to engage in THREE or FOUR of the 5R’s	The material/site contains a full, clear statement of copyright and specifies the ways in which the material can be adapted. Copyright allows users to engage in all FIVE of the 5R’s	�� �� ___/5
Credibility Poster �	From an author(s) with no authority on the subject, or the author(s) cannot be found. Few to no references or sources listed. Design looks/feels unprofessional. Intended audience cannot be determined. Biased perspectives.	Authors can be found but have little to no authority on the subject. Organization has some authority. �Unclear intended audiences. Perspectives are somewhat biased.	From a distinct organization/ individual/group with authority on the subject. Background information of authors is highlighted. Design feels professional. Perspectives hold relatively little bias. Some references and sources listed.	From a larger/ established organization with authority on the subject. Credit and authors consistently highlighted and can be traced back to sources. Origin of information is frequently mentioned. Unbiased information.	Clearly established authors and reputable organizations. Evidence of how others use it. Excellent design and presentation. Comprehensive listing of references and sources.	�� ___/5
Accessibility Poster �	Content offers no accessibility options for your intended audience. No evidence of alternate text on images, closed captions on videos, video transcripts, difficult to use offline, etc. Accessibility is inconsistent.	Content offers very few accessibility options for your intended audience. These options are not thoroughly explained or highlighted. Accessibility is occasionally consistent.	Content offers some accessibility options for your intended audience. Content is either not easily accessible or is not fully implemented. Accessibility is inconsistent in course.	Content offers several accessibility options for your intended audience but not throughout the whole course or are not fully highlighted. Accessibility is mostly consistent.	Content consistently includes comprehensive accessibility options for your intended audience.	�� ___/5
Completeness Poster	Content/ideas are fragmented. No subtopics are represented. There are no links to supplemental resources.	Content is loosely organized. Some subtopics are represented. There are few to no links to supplemental resources.	Either content is not cohesive or subtopics are not represented. Smaller collection of supplemental resources.	Content cohesively covers main topics and most subtopics. Medium-sized collection of supplemental resources.	Content is fully fleshed out and covers all relevant subtopics. Links to many different supplemental resources.	�� ___/5
Design and Presentation Poster �	Course is hard to follow due to poor design and organization of content. Menu severely limits usability.	Course is hard to follow at times due to poor design choices and presentation styles. Menu somewhat limits usability. Visuals are sporadic.	Course is relatively simple to follow and the menu is satisfactory. Visuals help drive home material. Supplemental materials are present but need improving.	Course content and menu sections are easy to navigate. Graphics and supplemental materials are provided.	Course designed very well with organized content. The menu allows for easy navigation within content. Superior and cohesive graphics, video, and supplemental material.	�� ___/5
�	�	�	�	�	Total Score:	___/30

Let’s take a look at the rubric itself. This rubric utilizes 6 categories that contribute to the quality of an open educational resource: Maintenance, which refers to how up to date and well maintained an OER is; Adaptability, which refers to the OER’s licensing and how well the OER caters to the 5 Rs; Credibility, which refers to how credible or knowledgeable the creator of the OER is; Accessibility, which refers to how accessible the OER is to persons with disabilities; Completeness, which refers to how much of the topic is covered by the OER and how much supplemental material is provided; and finally, Design and Presentation, which refers to how organized and navigable the OER is. Each of these categories can be scored on a scale from 1 to 5, with 1 being the lowest score and 5 being the highest.

4 of 15

Scoring Outcome Ranges:

The rubric uses 6 categories scaled from 1-5. Scoring outcomes can be sorted into the following categories:

6–10 points = Low quality OER
11–19 points= Mid-quality OER
20–30 points = High quality OER

5 of 15

OER to be Assessed: O*NET Interest Profiler

6 of 15

O*NET Interest Profiler Description – Part 1

7 of 15

O*NET Interest Profiler Description – Part 2

8 of 15

Updated within the last two years. Working links and graphics. Works on modern platforms. Modern and mobile-friendly design. Exceeds presentation and visual display standards.

This fits with a high score of 5:

The footer of the webpage hosting the O*NET Interest Profiler shows the date the site was last updated.
No broken links
Webpage works on modern platforms.

How recently was the O*Net Interest Profiler updated?

Category 1: Maintenance

9 of 15

The material/site contains a full, clear statement of copyright and specifies the ways in which the material can be adapted

Copyright allows users to engage in all FIVE of the 5R’s

This fits with a high score of 5:

The website has a CC 4.0 license allowing full adaptability.

Category 2: Adaptability

10 of 15

Clearly established authors and reputable organizations. Evidence of how others use it. Excellent design and presentation. Comprehensive listing of references and sources.

This fits with a high score of 5:

Author information is readily available at the bottom of the webpage: it was created by the National center for O*NET Development (sponsored by the US Department of Labor and the Employment and Training Administration). Design looks/feels professional. Supplemental information is taken from US census data. It has a .gov URL.

How credible is the O*NET Interest Profiler?

Category 3: Credibility

11 of 15

Content consistently includes comprehensive accessibility options for your intended audience.

This fits with a high score of 5:

No videos are present on the O*NET Interest Profiler, but the website uses high contrast colors, a keyboard friendly layout, and easily legible text when enlarged. Appropriate alt text seems to be available on all images.

How accessible is the O*NET Interest Profiler?

Category 4: Accessibility

12 of 15

Content is fully fleshed out and covers all relevant subtopics. Links to many different supplemental resources.

This fits with a high score of 5:

Text

The O*NET Interest Profiler links to a large variety of supplemental information and resources

Category 5: Completeness

13 of 15

Course designed very well with organized content. The menu allows for easy navigation within content. Superior and cohesive graphics, video, and supplemental material.

This fits with a high score of 5:

The instructional content is of high quality, well organized and easy to navigate. The quiz itself has easily understandable terminology and the test it is derived from is widely used in the career counseling field. The instructions and explanations are clear and easy to understand. Although no video is present, all content is cohesive.

How well designed is the O*NET Interest Profiler?

Category 6: Design and Presentation

14 of 15

O*NET Interest Profiler

final score:

30/30

High Quality OER

15 of 15

Credits:

This presentation includes information from My Next Move by the U.S. Department of Labor, Employment and Training Administration (USDOL/ETA). Used under the CC BY 4.0 license. O*NET® is a trademark of USDOL/ETA.

Assessing OER Quality Rubric created by Rachel Wellman, Brett Feller, Trevor Williams, and Julie Gilson

Presentation by Julie Gilson

This presentation is licensed under CC BY-NC-SA 3.0. Attribution can be made to EME 5250 Group.