Qualitative Measures: What Rubrics Can’t Assess

Presentation for ITC | February 2023

PRESENTERS

ONLINE COURSE DEVELOPMENT RUBRICS

What Qualities Matter?

• Quality Matters sells rubrics, professional development opportunities, and certification courses all focused on their course design rubrics
• 8 Categories
• Examples: Course Overview and Introduction, Instructional Materials, Assessment and Measurement
• 42 Specific Standards
• Instructions make clear how to get started and where to find various course components.
• The instructional materials contribute to the achievement of the stated learning objectives or competencies
• The requirements for learner interaction are clearly stated.
• Courses can Pass / Fail and QM review

​

Online Course Development Rubric: OSCQR�

• Free online resource that can be used to help form new courses or review existing courses
• It is "not evaluative" and is a "professional development exercise"
• 6 Categories
• Examples: Overview and Information, Design and Layout, Assessment and Feedback
• 50 Standards
• Course provides an overall orientation or overview, as well as module-level overviews to make course content, activities, assignments, due dates, interactions, and assessments, predictable and easy to navigate/find.
• A logical, consistent, and uncluttered layout is established. The course is easy to navigate (consistent color scheme and icon layout, related content organized together, self-evident titles).
• Course includes frequent, appropriate, and authentic methods to assess the learners’ mastery of content.

Other Course Development Rubrics�

• California Community Colleges' California Virtual Campus (CVC) OEI Rubric
• University of Illinois: Quality Online Course Initiative (QOCI) Rubric
• And?

​

Similar Categories

Only Some Categories Have Results

WHAT WE DON’T LOVE ABOUT COURSE DESIGN RUBRICS

Rubrics Are Absolutely Prescriptive�

Course Development Rubrics Do Not "Humanize" Online Learning

• Nor do they help humanize the process of developing online classes, working with instructional designers, or becoming certified as an online instructor.

Rubrics Call for "Clear" - Whatever That Means

• The word "clear" appears more than 12 times on the QM Rubric
• If I explain everything clearly and in full detail, the course content becomes a maze, the syllabus becomes unreadably long, and information is not prioritized
• Often, a full and "clear" explanation makes assignment instructions overwhelming to students who don't know where to start or who get mired in all of explanation
• Also - what does "clear" mean? To whom?

​

QM-Type Rubrics Measure Things That Don’t Matter To Students

WHAT RUBRICS CANNOT MEASURE

Words Not On QM Rubric

Design for Neurodiversity

• Cognitive Load
• Simplified Instructions
• Options for submission style
• Recognition of student agency

​

You Get What You Measure – But What About:

• How well content is explained and emphasized
• How engaged students really are
• Options for students who fall behind or need more time
• How safe students feel asking questions or making mistakes
• Opportunities students have to be creative or make choices about their learning
• How students’ safety and privacy are being protected
• How well a course prepares students for future courses

​

If Course Design Rubrics Are Bad, What’s Good?

Quality Guides for Online Instruction Focus On:

• Accessibility, Inclusion, and Privacy
• Student Agency
• Flexibility Offered By Instructors
• Online Presence and COI
• Quality and depth of interactions and feedback
• Alignment to course objectives and purposeful assessment

Ensuring Quality Online Teaching and Learning:

• Qualitative checklists for instructors to use on their own classes
• Peer observations and mentoring (data-supported)
• Templates that meet requirements, faculty bring the personality
• Focus on instruction
• Trauma-informed teaching, culturally-responsive teaching, neurodivergent design

SOMETHING TO TRY:

Ask your students

What do you recommend?

www.allegrahanna.com