Illinois Democracy Schools Report and Data Walk-Through

2022 Report

Goals of this Resource

• This webinar will be;
• walking Democracy School champions through the report
• demonstrating users how to look at the topline
• showing people how to make pivot tables for deeper understanding of trends in your own data
• After watching the resource you should be able to understand how to use the Democracy School report, and what you can do with the rest of the data.

​

Introduction to the Report

IDS School Report

IDS Report

• What is in the report - walking through the Table of Contents
• What the report is for in the larger process:
• The report is not meant to be compared to other schools and there isn’t “statewide” average.
• The report is designed to help schools engage with their internal data to inform their deep assessment and continuous improvement efforts.
• Things report does and does not cover:
• Differences by different student groups (--> Use pivot tables)
• Specific response distribution of each questions (--> see toplines)

​

What is in the report

• The report covers;
• A big-picture description of the schools’ assets and areas for growth for instruction and culture components
• Components are measured on two metrics (access and quality) and by two stakeholders (Students and staff)
• Also includes some concrete data on activities and pedagogy that are commonly used, and notable data points.*
• Examples and how schools might use that information (e.g., relatively few are in social/issue-based extracurricular groups)
• *Only new schools get all 11 elements and renewal schools get selected elements.

What is not in the report

• The report does NOT cover;
• How students and staff of different subgroups responded (by race, gender, and roles etc.)
• What to do to make improvements in each area - because solutions should build on existing strengths, there are no good for all solutions.
• Evaluative conclusions about how each school is doing on different components. We provide data - you assess progress and challenges given all the context and conditions unique to your school.

​

Report Walk-through

Toplines

Topline

• What is a topline?
• How can topline be used?
• Unpacking a number which may be useful when:
1. You see large gaps in student and staff report on the same component - helpful to see what items were included
2. You see gaps between access and quality - may inform specific strategy as the questions are aligned with best practice which in turn informed IDS’s components.
3. You are interested in knowing student civic engagement
4. You are interested in additional data that might better inform a particular aspect of the school curriculum/life that surfaced in the schoolwide qualitative assessment.

Case 1: Deeper dive into a concept (student civic engagement) using topilnes

• In this video you will see how you can look up relevant question phrasing, and how people responded to specific items in their survey to achieve deeper understanding of the data about your school.
• In this case, we use an example of “civic engagement” domain, which is available through the student survey.
• Using toplines, stakeholders can review two different things within the Simulation Domain:
• What types of the civic engagement was more common among our students?
• Are some types of engagement particularly high or particularly low? Does our school want to set goals for student civic engagement?

Demonstration: Using Toplines

Analyzing Your Data with Pivot Table

Pivot Tables

• Who should be thinking about using a pivot table?
• You should be familiar with Excel spreadsheet
• Know what can and can’t be done in Pivot Tables
• Pivot table might be useful for;
• Identifying/refining targets for improvement
• Triangulating qualitative data/observations
• Internally evaluating impact on goals

​

When Pivot Tables are Useful - Examples Demo

• Unpacking a number which may be useful when:
• You see large gaps in student and staff report on the same component - helpful to see what items were included
• You see gaps between access and quality - may inform specific strategy as the questions are aligned with best practice which in turn informed IDS’s components.
• You are interested in knowing student civic engagement

​

Initial Steps to use the “raw data”

• Your data will look like this →
• Replace “#NULL!” and “-99” with blank using the “find and replace” function.
• Review the Topline document to identify (and write down) what you want to explore, by column labels.
• Student race →Q16.8
• Student class → Q16.1
• Civics content teaching → FCK (access and quality)
• Know what the numbers mean for each column of interest.
• Generally 1 = yes blank = no
• 1 to 4 scales - disagree to agree scale (4 being agree)

How to “clean” your data before Pivot Table

Case 2: Differences by Student Backgrounds

• In this case study, you will see how a pivot table can be used to look at the variations in percentages of students who met the criteria for Foundational Civic Knowledge Access and Foundational Civic Knowledge Quality metrics.
• We will demonstrate how this variations can be shown by students’ self-reported racial identity and class grade.

Demonstration - Variations in Index Scores

Case 3: Do Students’ Civic Readiness Vary by GPA?

• In this case, you will see a pivot table that is in a different format. Instead of 0 or 1 scoring like the IDS elements score, you will see that students gave a 1, 2, 3, or 4 as their response to questions related to their efficacy about implementing civic and democratic processes. 1 was “disagree” (they can’t do it” and 4 was “agree” (they can do it).
• You and other champions may be interested in understanding how the student responses varied by another factor, like self-reported academic performance.

Case 3: Student Civic Efficacy by Reported Academic Performance

Using Your Data - Dos and Don'ts

Appropropriate use:

• Planning for strategies that, together, work for all students and staff with considerations given to those who are not benefiting as much.
• Developing plans for qualitative assessment, for instance, by convening a focus group that is specifically for students who are not in extracurricular activities, or for students who identify as female.
• Creating a summary of data that can be shared with the school community for reflection and discussion

Inappropriate use:

• Naming (and shaming) groups and individuals who are “not pulling the weight” in providing democratic education
• Looking at the data to see where a particular response came from and reaching out to people specifically based on their responses
• Using the data itself for press or other types of PR efforts or publications as research study.

Resources and contacts

IDS resources:

• Brief description of 11 elements in the Illinois Democracy Schools model
• In-depth consideration of all elements of the IDS model, with indicators, descriptions, examples and resources
• CLAD tool: for compiling data during schoolwide assessment of civic learning
• Indicators for all 11 elements

To request your school’s data please write to:

• Kelly.Siegel_stechler@tufts.edu

Supplemental Materials

• Senior Researcher, CIRCLE at Tufts University
• Student Survey Instrument
• Staff Survey Instrument
• Questions aligned with each IDS element