1 of 30

Montana Alternative Student Testing (MAST) Pilot Program: �Task Force Meeting #3 (Day 2)

Scott Marion, Juan D’Brot, & Chris Brandt

National Center for the Improvement of Educational Assessment

​

June 21, 2022

2 of 30

Overview of Day 1

9:00 Welcome & introduction to the meeting (10 minutes)

9:15 Reviewing the revised theory of action

10:15 Design considerations (break as needed)

Noon Lunch

12:45 Implementation considerations

1:45 Next steps

2:00 Adjourn

MT OPI Assessment Task Force June 20-21, 2022

2

3 of 30

Our Work Trajectory

MT OPI Assessment Task Force June 20-21, 2022

3

Project Framing

Specifying Goals and Intended Outcomes

Producing Detailed Theory of Action

Writing Final Report

May

June

July

4 of 30

OPI Google Site

4

1

https://sites.google.com/opiconnect.org/montanataskforceformathandelat/home

5 of 30

Meeting Logistics

5

1

Date

Time

Meeting Format

May 4, 2022

12:30 PM − 3:30 PM

Webinar

June 1, 2022

10:00 AM − 1:00 PM

Webinar

June 20, 2022

9:00 AM − 4:00 PM

In Person, DoubleTree Helena

June 21, 2022

9:00 AM − 2:00 PM

In Person, DoubleTree Helena

July 6, 2022

9:00 AM − 1:00 PM

Webinar

TBD

As needed.

Webinar

6 of 30

State Rate Reimbursements

6

1

Reimbursable Items

State Daily Rate

Receipts Needed

Breakfast

$7.50

Yes

Lunch

$8.50

Yes

Dinner

$14.50

Yes

Hotel

$94.00

Yes

Mileage

58.5 cents per mile

​

Non-Employee Reimbursement Form

7 of 30

Revised Theory of Action

9:15

MT OPI Assessment Task Force June 20-21, 2022

7

8 of 30

Revised Theory of Action

  • We took your great ideas and first created an overview theory of action as seen on the following slide

MT OPI Assessment Task Force June 20-21, 2022

8

9 of 30

MT OPI Assessment Task Force June 20-21, 2022

9

10 of 30

Detailed Theory of Action

  • The following slides depict the detailed theories of action focused on the following areas:
    • High-quality feedback
    • Use of multiple item types
    • Clear connections among standards, instruction, and assessment, and
    • Communicating the value of the assessment system using an understanding of current perceptions

​

​

MT OPI Assessment Task Force June 20-21, 2022

10

11 of 30

MT OPI Assessment Task Force June 20-21, 2022

11

12 of 30

MT OPI Assessment Task Force June 20-21, 2022

12

13 of 30

MT OPI Assessment Task Force June 20-21, 2022

13

14 of 30

MT OPI Assessment Task Force June 20-21, 2022

14

15 of 30

Theory of Action Review

  • Individual reflections and feedback (20 minutes)
  • Full group discussion and feedback (20 minutes)

​

  • Please note that we are seeking your advice on accuracy, completeness, and parsimony…

MT OPI Assessment Task Force June 20-21, 2022

15

16 of 30

Assessment Design Considerations

10:15

MT OPI Assessment Task Force June 20-21, 2022

16

17 of 30

MT OPI Assessment Task Force May 4, 2022

17

Assessment design is always a case of optimization under constraints.

18 of 30

Major Aspects of Assessment Design

  • “Construct” definition (required knowledge, skills, & dispositions)
  • Item specifications and item writing (includes item types)
  • Assessment specifications
    • Delivery approach (e.g., fixed form, adaptive assessment)
    • Modular, mini-summative
    • If modular, what grain size?
  • Administration rules and requirements
    • Timing (how often and/or how many times/year)
    • Sequencing (same order or flexible?)
  • Score reporting (for each through-year component and end-of-year)
    • Student and parent reporting
    • Classroom and school reporting
    • District and state reporting

MT OPI Assessment Task Force June 20-21, 2022

18

19 of 30

Assessment Design Considerations

  • In order to help guide the initial assessment design, we want you to work in groups, and use the theory of action to recommend design considerations for the assessment design components listed on the previous slide.
    • Note: You might want to refer to the design briefs we shared for the last meeting.
  • If you name something that is NOT in the theory of action, please note it so we can be sure it gets included.
  • Note: We will not address the construct definition today since the construct is really represented by the standards.
  • You will work in this Google doc with a table like this one…

MT OPI Assessment Task Force June 20-21, 2022

19

20 of 30

MT OPI Assessment Task Force June 20-21, 2022

20

Major Assessment System Component

“Minor” Component

Design Consideration

Rationale (tied to TOA)

Item Specification

Item types

 

 

Assessment Specifications

Delivery approach (computer/paper)

 

 

Item delivery (fixed-form, adaptive)

 

 

Modular or mini-summative for TY components

 

 

If modular, what grain size?

 

 

Administration Requirements

Timing: How often?

 

 

Sequencing: Fixed or flexible?

 

 

Score reporting

Student

 

 

Parent

 

 

Classroom

 

 

School & District

 

 

21 of 30

Assessment Design Considerations

  • Full group report out

MT OPI Assessment Task Force June 20-21, 2022

21

22 of 30

Lunch Break

12:00-12:45

MT OPI Assessment Task Force June 20-21, 2022

22

23 of 30

Implementation Considerations

12:45

MT OPI Assessment Task Force June 20-21, 2022

23

24 of 30

Implementation Considerations

  • What needs to be in place to maximize the likelihood of a successful pilot implementation? We started with these things:
    • Technology infrastructure
    • Administration guidance
    • Assessment literacy professional learning opportunities
    • Score report interpretation
    • Curriculum and instruction guidance
    • Leadership support
    • Decisions about curriculum and interim assessments
  • What are we missing?

MT OPI Assessment Task Force June 20-21, 2022

24

25 of 30

Implementation Considerations

  • Again, working individually for 15 minutes, please put as many ideas as possible into this Google doc regarding the various domains of implementation.

​

MT OPI Assessment Task Force June 20-21, 2022

25

26 of 30

MT OPI Assessment Task Force June 20-21, 2022

26

Implementation Issue

Implementation Considerations

Rationale/Link to TOA

Technology infrastructure

 

 

 

Administration guidance

 

 

Assessment literacy professional learning opportunities

 

 

Score report interpretation and use

 

 

Curriculum and instruction guidance and professional learning

 

 

Leadership support

 

 

Interim assessment guidance

 

 

27 of 30

Implementation Considerations

  • Full group report out

MT OPI Assessment Task Force June 20-21, 2022

27

28 of 30

Plans for July 6th (9:00-1:00)

MT OPI Assessment Task Force June 20-21, 2022

28

  • We will do our best to present a revised theory of action and associated text to represent your ideas regarding the things that need to be in place in order to make the pilot successful.

29 of 30

Questions/Comments?

What questions, comments, requests, suggestions do you have for any of us?

MT OPI Assessment Task Force June 20-21, 2022

29

30 of 30

Thank You!

MT OPI Assessment Task Force June 20-21, 2022

30