ePIC 2023
Vienna, Austria
6 December, 2023
What Do We Want to Cherish?
Re-examining Assessment in the Collective World
Nan Travers, SUNY Empire State University, USA
We measure what we cherish!
What do we cherish?
Issues of subjectivity and bias
Recognition of what someone knows and can do is subjective and biased
What is accepted as what is to be learned is subjective and biased
Validation of learning is subjective and biased
Mis-Recognition & Non-Recognition
Some thoughts about assessing learning
What is learned within a learning experience is
>, <, =, or ≠ what is taught
Not everyone learns the same within a learning experience
How and what one learns and the context of learning is important – but how learners connect learning is essential
Different levels of learning can be assessed regardless of the source
1
3
2
4
We Only Measure Evidence of Learning��Not Actual Learning
What Is IT?
When a learner gets it – what is IT?
When a learner doesn’t get it – what is IT?
Assessment of Learning
Classical Assessment Measures
Better at measuring what IS NOT known
Than what IS known
Assessments are a proxy!
Learning Oriented / Evidence-Based
Evidence
Learning
Descriptors
Student Prompts
Direct Assessment
Assessment of distinct behaviors or products that can be directly observed and/or measured
Global Learning Qualifications Framework
Framework to assess college-level learning:
Global Learning Qualifications Framework
A. Lower Level Learning - Learning experiences provide opportunities to: | No Evidence | Partial Evidence | Full Evidence |
Build a foundational knowledge-base using broad and specific knowledge, skills, and competencies as applied to relevant theories, methodologies, practices and quantitative applications. |
|
|
|
Gather, review, analyze, evaluate and respond to defined or routine problems drawing on relevant theoretical, practical, and prior knowledge and experiences. |
|
|
|
Apply relevant concepts, theories, and technical and professional knowledge in the analysis and resolution of practical issues within particular contexts. |
|
|
|
Research and collect information and relevant data from a wide range of resources to answer questions and/or solve problems; evaluate the quality, relevance, currency and accuracy of that information; and select and interpret appropriate information for the situation, problem or question. |
|
|
|
Engage in decision-making according to the standards of practice and ethics of the field. |
|
|
|
Communicate knowledge and demonstrate skills in content areas accurately, coherently and clearly that are informed by key concepts, techniques, developments and ethical standards in the field through the use of appropriate techniques, including: written, oral, visual and/or technology-facilitated methods. |
|
|
|
Engage in active listening, relevant discussions, and respectful discourse with individuals and/or groups with varying viewpoints and diverse backgrounds. |
|
|
|
Participate in and accept accountability and responsibility for own learning and work. |
|
|
|
Reflect upon one's own actions and implications in situations and take responsibility for these actions while working with others and/or solving problems. |
|
|
|
Self-identify and evaluate gaps in own knowledge, skills and abilities; use feedback to develop learning and professional goals; and engage strategies for improvement and to reach those goals. |
|
|
|
Demonstrate insight into one’s identity and biases and the influence they have on understanding, interacting and working effectively with others. |
|
|
|
College-Level Learning Worksheet
PLA Portfolio Competencies
Competency 1:
Identify Your Learning
1
Competency 2:
Use Your Learning
2
Competency 3:
Position Your Learning
3
Competency 4:
Communicate Your Learning
4
What is needed in the 21st century?
Engages self-assessment and inquiry to transfer and apply learning throughout life.
Engages interpersonal, cross-cultural and empathy competencies to communicate effectively.
Collaborates and works in teams with a customer focus.
Uses critical thinking, reflective learning and creativity to plan, organize, and solve problems.
Has initiative to be a catalyst and resilient, while also being adaptive and flexible.
Approaches work and learning with quality and integrity.
1
2
3
4
5
6
EMPLOYMENT & EDUCATIONAL COMPETENCY CLUSTERS
Knows the basic principles and concepts, mathematics, and tools and technologies of the field.
7
7
Competencies are a universal translator
Competencies
Course Learning Outcomes
Experiential Learning Outcomes
Lived experience language
Academic language
Knowledge Definition
Institutional Perspective
Program Perspective
Assessor Perspective
College Level Learning
Valuing
Lens
Personal Knowledge
Work Knowledge
Community Knowledge
Program Integration
Credential
What is knowledge?
Who defines knowledge?
Who gets to be recognized for their knowledge?
What knowledge is recognized?
What gets to be college knowledge?
Academic Knowledge
Learning is embedded within experiences
Community
Learning
Academic Learning
Workplace Learning
Personal
Learning
Prior
Learning
Learner �orchestrates own life, work & school
Learner responsibility
Work/ School/ Life/ Community
Segmented Learning Experiences
Individual Meaning Making
Collated Learning Experiences
Learners must make sense of their experiences on their own
All learning is experiential�Learning is lifelong and life-wide
Future
Learning
Prior
Learning
Emergent Learning
Convergent Learning
Relational
Learning
Experiences from work, education, and life.
Learning Processes
Self-Assessment
Inquiry
Transfer
Future
Learning
Convergent Learning
Relational
Learning
Assessment Processes
Self-Assessment
Inquiry
Transfer
Future
Learning
Convergent Learning
Relational
Learning
How does the learner …
What does the learner …
What do I know?
What do I not know?
What questions do I need to ask?
How do I answer those questions?
How do different pieces fit together?
How do different parts relate to other areas?
How does it fit with novel situations?
Challenge:
Develop scalable assessments that capture individualized learning.
Collective World
When we assess someone’s knowledge, we are also assessing collective knowledge.
We are determining how someone’s knowledge fits into and/or adds to the collective knowledge.
Nomological Network
Related Constructs
Further Related Constructs
VisualThesaurus.com
Chronbach & Meehl, 1955
NOMOLOGICAL NETWORK
Nomological Network
http://cmap.ihmc.us
A pre-conceived assessment only targets a small portion of the nomological network
What we cherish
Concept Map - Creativity
Business
Psychology
Nomological Network
http://cmap.ihmc.us
A pre-conceived assessment only targets a small portion of the nomological network
Collective Nomological Network
http://cmap.ihmc.us
Challenge:
Develop scalable assessments that capture individualized learning
Places learning within a collective knowledge scheme
Recognizes & credentials individual learning
Allows for the collective knowledge to expand