1 of 1

W

1. Dries, DR, Dean, DM, Listenberger, LL, Novak, WRP, Franzen, MA and Craig, PA (2017), An expanded framework for biomolecular visualization in the classroom: Learning goals and competencies. Biochem. Mol. Biol. Educ., 45: 69–75. doi:10.1002/bmb.20991

2. Bateman RC and Craig PA. (2010) Education Corner: A Proficiency Rubric for Biomacromolecular 3D Literacy. PDB Newsletter 45:5-7.

3. How to Write Program Objectives/Outcomes: http://assessment.uconn.edu/docs/HowToWriteObjectivesOutcomes.pdf (revised 2006).

4. Procko, K., Engelman, S., Jakubowski, H., Beckham, J.T., Dean, D.M., Franzen, M.A., Novak, W.R., Roberts, R., Roca, A.I., Shor, A.C. and Terrell, C.R., 2020. Meeting report: BioMolViz workshops for developing assessments of biomolecular visual literacy. Biochemistry and Molecular Biology Education.  49(2), 278-286.

References

Developing Assessments to Gauge Biomolecular Visual Literacy:

A 5-Step Approach to Ensuring Accuracy and Reliability  

Abstract

​

​

​

​

Assessment Development Process: Five-Step Approach to Ensuring Accuracy and Reliability  

Newsletter

sign up

3

1

5

6

7

Funding: This work has been supported by the National Science Foundation under award numbers DBI-1920270, DUE-1022793, -1323414, -1503811 and IUSE-1712268.

​

Any opinions, findings, and conclusions or recommendations expressed in this publication are

those of the authors and do not necessarily reflect the views of the National Science Foundation.

4

Program Number: 521.7 BioMolViz is a community of educators, assessment experts, and biomolecular visualization enthusiasts dedicated to improving biomolecular visual literacy. A central goal of the project is to create a repository of validated assessments that instructors can use in their classrooms to gauge students’ visual literacy. Through three-day workshops, we convene faculty to craft and peer review biomolecular visualization assessments in teams. These workshops are often the first step in our assessment development process. In this presentation, we will describe a five-step model for designing and validating assessments that consists of: 1. Item creation 2. Item revision 3. Expert panel review 4. Field testing and 5. Dissemination through the online BioMolViz repository. Results from the first three phases of the process will be reported, and our plan for field testing and disseminating assessments will be discussed. In 2020, our workshops and assessment development model moved online, which opened up new pathways for sustained engagement with workshop participants. In particular, we will share how a new working group emerged when we established our remote workflow. A subset of the group, The BioMolViz Fellows, continued working with our community for several semesters and has become integral to the iterative assessment revision process. Alongside our validation process, we will highlight insights for building online communities, establishing remote workflows, and sustaining long-term participant engagement.

1

Item Creation

2

Revise

3

Expert Panel

4

Field Testing

5

Publish & Disseminate

Is it valid?

Is it reliable?

Five Phases of Assessment Development

​

​

​

​

Kristen Procko1, Shelly Engelman2, Josh T. Beckham1, Daniel R. Dries3, Margaret A. Franzen4, Henry Jakubowski5, Walter R.P. Novak6, Rebecca Roberts7, Alberto I. Roca8, Audrey C. Shor9

​

The BioMolViz Framework

​

Broad topics in visual literacy

Several subtopics within the overarching theme

Define how the learner will demonstrate proficiency

Novice: first/second year undergraduates Amateur: junior/senior majors

STEP 1: Item Creation

  • BioMolViz Framework used for backward design of assessments
  • Workshops facilitate assessment creation
  • First feedback/revision performed in-workshop

& Online

Online at:

biomolviz.org/framework

STEP 2: Item Revision

  • An iterative revision process is applied to assessments
  • BioMolViz Fellows are a remote working group that update/revise items in partnership with steering committee members

Steering

Committee

BioMolViz

Fellows

Prompt: Which of the following images depicts an amphipathic alpha helix, viewed down the helix axis? Explain your answer.

​

Revised Prompt: Three α-helices are shown, viewed down the helix axis. Which image depicts an amphipathic α-helix? Explain your answer; include a definition of amphipathicity and how you used the images to identify the helix.

STEP 3: Expert Panel Review

  • Experts rate assessments on a four-point Likert scale:

​

​

​

  • Reported scores are the mean on the scale, where each number corresponds to an answer to each question above:

​

​

​

​

  • 4.0–3.6: accepted without modification

​

​

​

​

    • Interrater reliability examined, and training module

added for second round of expert review

  • To what extent is this item RELEVANT to the Primary Learning Objective?
  • To what extent is this item APPROPRIATE for the NOVICE/AMATEUR learner?
  • To what extent is this item CLEAR and/or unambiguous?
  • 1 = Not at all; major revisions required
  • 2 = Somewhat, but needs some revision
  • 3 = Good, but needs minor revision
  • 4 = Very much/Excellent

3.59–3.2: good, minor tweaks

3.19–2.8: some modifications

<2.8: major modifications

​

RELEVANT TO LEARNING OBJECTIVE 4.0

APPROPRIATE FOR LEARNER LEVEL 4.0

CLEAR & UNAMBIGUOUS 3.8

Expert Rating Example

STEP 4: Field Testing

  • Expert panel-reviewed assessments will be tested in classrooms in Fall 2022 & Spring 2023
  • We are seeking partners from a diversity of institutions to field test assessments at both the novice and amateur learner levels

STEP 5: Dissemination through the BioMolViz Repository

Searchable assessment database is currently under construction and will be available for users in Spring 2023. Please visit our website to stay updated: biomolviz.org

Partner with BioMolViz! Complete a brief interest form to participate in field testing assessments:

2

8

9