1 of 16

DAY 3

INTERVENTIONS AND DATA COLLECTION

2 of 16

From Questions to Study Design

What intervention(s) will you need to do?

What kinds of data do you need to answer your question?

Who provides the data?

How will you collect it?

How often do you need to collect it?

Are there terms that need operationalizing/defining?

As you develop your research study, these are the things you need to think about. We will go over each step during this institute (not all today).

Intervention: may not be necessary, but if you do choose to try something new, you do need to make sure you think about both baseline and final data collection in order to make claims to change

Kinds of data: in order to triangulate, you want to think of multiple data sources that will help answer each question, ideally that can get at both 1^st and 2^nd order perspectives (especially if you’re focused on students trying something new)

Who provides data: if you are collecting data from students, you need to figure out how you will get it (esp. if you want to collect things they don’t normally turn in, such as notebooks)

How to collect it: SW can be collected in its original form (handwritten), scanned to pdf, photographed, transcribed—need to think about how you will collect other forms as well

How often: timing is important in many teaching-related things, such as when students are writing and revising. If you want to see how they revise, you will need to get a copy of what they write at multiple moments in time. One good way with typed papers is to have students writing in Google Docs, where you can look at the revision history. Or keep a camera to photograph what they have at the end of each class session.

Operationalizing terms: If you are interested in effects of praise on students’ motivation, you need to define praise and figure out how you will know when you see it. Same with motivation. The higher the level of inference, the more clear you need to be about your definitions of terms. This will also help you when it comes time to code data and write up your study!

3 of 16

INTERVENTIONS AND ACTION PLANS

WHAT IS AN INTERVENTION?

“A ‘NEW’ APPROACH TO SOME ASPECT OF INSTRUCTION, ADAPTATIONS OF CURRICULUM OR STRATEGIES DOCUMENTED AS EFFECTIVE IN AT LEAST SOME SETTINGS, INTEGRATION OF EFFECTIVE STRATEGIES IN NEW WAYS WITH DIFFERENT TYPES OF STUDENTS, OR TESTING OUT A NEW IDEA INFORMED BY THEORIES OF LEARNING AND TEACHING.”

FROM UC DAVIS SCHOOL OF EDUCATION, “WHAT IS TEACHER RESEARCH?” HTTP://EDUCATION.UCDAVIS.EDU/CREDENTIALMA-PROGRAM-INFO/WHAT-TEACHER-RESEARCH

4 of 16

PROJECTS AS INTERVENTIONS

REFLECT ON THE DEFINITION OF “INTERVENTION” FROM THE PREVIOUS SLIDE (WRITE-PAIR-SHARE)

WHICH ASPECTS OF THE DEFINITION APPLY TO THE PROJECT YOU ARE DEVELOPING?
HOW CAN DOING THIS PROJECT SHED LIGHT ON WHAT YOU ARE WONDERING ABOUT YOUR STUDENTS’ LEARNING?
WHAT ASPECTS OF THE PROJECT COULD YOU TWEAK TO BETTER FIT WITH YOUR WONDERS?

5 of 16

PROJECTS AS INTERVENTIONS

WHICH ASPECTS OF THE DEFINITION APPLY TO THE PROJECT YOU ARE DEVELOPING?
HOW CAN DOING THIS PROJECT SHED LIGHT ON WHAT YOU ARE WONDERING ABOUT YOUR STUDENTS’ LEARNING?
WHAT ASPECTS OF THE PROJECT COULD YOU TWEAK TO BETTER FIT WITH YOUR WONDERS?

“A ‘NEW’ APPROACH TO SOME ASPECT OF INSTRUCTION, ADAPTATIONS OF CURRICULUM OR STRATEGIES DOCUMENTED AS EFFECTIVE IN AT LEAST SOME SETTINGS, INTEGRATION OF EFFECTIVE STRATEGIES IN NEW WAYS WITH DIFFERENT TYPES OF STUDENTS, OR TESTING OUT A NEW IDEA INFORMED BY THEORIES OF LEARNING AND TEACHING.”

6 of 16

CONNECTING INTERVENTION TO REAL-WORLD CHALLENGES

Classroom

Improve current students’ learning experience

New approach for future teaching

Department, Campus, System

Share new approaches with colleagues

Showcase students' learning

Community

Improve local context

Highlight students' (and teachers') good work

Contribute to social justice

Action research ideally should have real-world impact at multiple levels

In your own classroom, you have current and future goals. Answering research questions can help you better understand your current students’ learning, so you can teach them better going forward. With that knowledge, you will be better able to teach your future students as well, since you can generalize from the current class to students you will have at this level (and possibly these students in higher levels)

At the institution level, you will have valuable pedagogical knowledge to share with colleagues in your department and beyond. You might be able to make recommendations about improvements to teaching and curriculum. You can also showcase your students’ learning, which may be something people in other departments and at the administration level are not aware of (think about how the art and theatre departments have concrete evidence of what their students can do, or the concrete canoes in civil engineering…). Language departments are so often fighting for the right to continue existing that having systematically collected evidence that projects do lead to better learning can be essential.

Finally, as with the community contribution goal of PBLL, AR should also contribute beyond the academic sphere. Many AR scholars also argue that the end results of AR should lead to social justice—for the students in participating classes and for the greater community. Given how many politicians these days doubt the need for language teaching or teachers at all, solid AR results along with a project can show why we need teachers and why language is important as well.

7 of 16

CONNECTING INTERVENTION TO REAL-WORLD CHALLENGES

REVIEW THE CORE PRINCIPLES OF PBLL FROM THIS MORNING

WHAT REAL-WORLD CHALLENGES WILL YOUR PROJECT (INTERVENTION) ADDRESS?

NOW REVIEW YOUR POTENTIAL RESEARCH QUESTIONS FROM YESTERDAY

HOW WILL ANSWERING THOSE QUESTIONS CONTRIBUTE TO IMPROVING

YOUR CLASSROOM?
YOUR INSTITUTION?
YOUR COMMUNITY?

JAMBOARD HTTPS://GO.HAWAII.EDU/DMX

8 of 16

DATA COLLECTION: NATURALLY OCCURRING FORMS

STUDENT WORK

PROJECT FINAL PRODUCTS
DRAFTS, PEER FEEDBACK, OTHER COMMENTARY
PRESENTATION SLIDES
ESSAYS
EXERCISES AND WORKSHEETS

OBSERVATION NOTES

QUICK NOTES WHILE TEACHING
FIELD NOTES OBSERVING STUDENTS IN ACTION

AUDIO AND VIDEO RECORDING OF CLASSROOM

SMALL GROUP DISCUSSIONS
PRESENTATIONS

9 of 16

DATA COLLECTION: RESEARCH SPECIFIC

SURVEYS
INTERVIEWS WITH INDIVIDUAL STUDENTS
FOCUS GROUPS WITH SEVERAL STUDENTS
TESTS OF SPECIFIC CONSTRUCTS (PRE/POST)

10 of 16

DATA COLLECTION: MITSUKO’S EXAMPLE

INITIAL RESEARCH QUESTION: HOW CAN I RAISE THE RELIABILITY (OR ACCURACY) OF STUDENTS’ SELF-ASSESSMENT?

Mostly saying

“My English talk is not good”

Students’ Self-Assessment Sheet on 1-minute English Conversation (pre-test)

11 of 16

DATA COLLECTION: MITSUKO’S EXAMPLE

VIDEO RECORDING OF 1-MINUTE CONVERSATION

Student work (information-gap activity)

12 of 16

DATA COLLECTION: MITSUKO’S EXAMPLE

SHIFT OF RQ FOCUS: RAISE MY STUDENTS’ AWARENESS OF THEIR OWN ORAL COMMUNICATION SKILLS

Students’ Self-Assessment Sheet on 3-minute English Conversation (post-test)

Mention about:

eye contact and body language
showing interest (“Oh, really?”)
asking questions
asking for time to think

...etc.

13 of 16

Research Question	Data Sources	When and how to collect	How data will help answer RQ
In what ways do students use vocabulary journals?	Student work: vocabulary journals, in-class essays, take-home essays	Journals: weekly, scan In-class essays: when finished; scan Take-home essays: on submission	Compare which words students wrote in vocab journal with words they used in writing samples
In what ways do students use vocabulary journals?	Observation: students doing in class writing	During in-class essays; take field notes and observe using seating chart	Note which students have vocab journal out during in-class writing and when open
What words do students choose for vocab journals, and why?	Student work: vocabulary journals	Weekly: chart word, source, reason in spreadsheet	Note where students found the words they included and what they wrote about reasons
What words do students choose for vocab journals, and why?	Interviews with students	At least once/term; audio record	Ask about reasons for choosing words

14 of 16

STUDY DESIGN

FOR EACH POSSIBLE RESEARCH QUESTION, DETERMINE WHAT DATA MIGHT HELP YOU ANSWER THE QUESTION

WHAT FORMS OF NATURALLY OCCURRING DATA WILL BE GENERATED DURING YOUR PROJECT?
WHAT ADDITIONAL RESEARCH-SPECIFIC INSTRUMENTS MIGHT YOU USE TO BETTER DOCUMENT ASPECTS OF YOUR STUDENTS’ LEARNING?

https://go.hawaii.edu/XM4

15 of 16

WORKSHOP TIME

POSSIBLE RESEARCH-FOCUSED THINGS TO DO:

DEVELOP AN ASSESSMENT YOU COULD USE AS A PRE-TEST/POST-TEST TO EVALUATE STUDENTS’ LEARNING OF DISCRETE LANGUAGE ITEMS
DEVELOP A SURVEY TO USE BEFORE DOING YOUR PROJECT TO IDENTIFY STUDENTS’ INTERESTS OR NEEDS
DEVELOP AN OBSERVATION TOOL FOR TAKING NOTES AS STUDENTS WORK ON THE PROJECT

16 of 16

Today I ….

Reflection

Take a minute to write a quick reflection or take away from today on a post-it and put it on your intro poster.
Take a minute to write a comment or question about something that you learned from someone else today (choose a different person than yesterday) and post it on their Intro Poster.