NCELP Phonics Assessments

​

Principles, design, creation

​

27.02.20, NCELP Residential 3

Introductory video – please click below to play.

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Phonics assessment team effort!

Robert Woore

Rachel Hawkes

Nick Avery

Giulia Bovolenta

Inge Alferink

Native speakers (checking & audio)

Natalie Finlayson

Emma Marsden

All the NCELP team!

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Outline

1. Principles of Phonics assessment

1.1 What are we trying to test?

1.2 How can we test it, validly and practically?

​

2. Question types

2.1 Task design

2.2 Item selection

​

3. Scoring

3.1 What do we mean by ‘accurate’ decoding?

3.2 How the scoring works

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

1. Principles of Phonics assessment

Objectives

• To test knowledge of Symbol-Sound Correspondences (SSC) covered to date in Y7 (i.e. a syllabus-based achievement test)

Why are we testing this?

• SSC knowledge is used
• when reading aloud (including ‘inner voice’) – linked to reading comprehension, vocabulary acquisition (inter alia)
• when listening (segmentation) and writing (spelling)
• Testing is bidirectional: both sound 🡪 print and print 🡪 sound

transcription reading aloud

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Skill acquisition theory

​

Ultimate goal!

Fluent, accurate decoding

Knowledge about a language feature

Knowledge about how to use a language feature

Fast and accurate use of the language feature

Extensive, repeated, meaningful practice

(DeKeyser, 2015)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Accuracy and fluency

Fluency and accuracy – both are important in L2 learning

• Accuracy: allows correct spoken and written forms of words to be learnt and used effectively in communication
• Fluency: reduces burden on working memory, thus allowing other ‘higher-level’ processes to be carried out more easily (e.g. inferencing when reading; conceptualizing ideas and formulating language when writing).

BUT fluency presupposes accuracy. There is no point automatizing incorrect knowledge!

🡪 Primacy of accuracy at this stage

< Wagen >

/ v ɑː ɡ ə n /

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Accuracy and fluency

< Wagen >

/ w a ɡ ə n /

<w> 🡪 / v /

<a> 🡪 / ɑ ː /

/ v ɑː ɡ ə n /

• More accurate L2 decoding associated with longer response times: Erler, 2003 (Year 7 learners of French in the UK); Li, 2019 (Chinese university students learning English)
• Speeded tests may make pupils more likely to fall back on L1 knowledge
• Main target of our phonics testing in Year 7 = accuracy
• Ideally, untimed (‘unspeeded’) tests. But practical constraints 🡪 time limit

• L1 ‘interference’ and its impact on fluency / accuracy:

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

a. Print to sound

• A number of different formats have been used to measure print-to-sound decoding
• Rhyme judgment task (Erler & Macaro, 2012); Sound-Alike Task (SALT) (Woore et al., 2018 – but French only)
• We believe the most valid method of assessing this is a ‘reading aloud task’ (RAT) (Woore 2006)

• Cf. Year 1 ‘Phonics Screening Check’ (DfE)

• But – trade-off between validity and ease of administration and scoring!

Scales image: by scott_kirkwood, openclipart.org.

Screenshot from Phonics Screening Check training video, DfE. www.youtube.com/watch?v=IPJ_ZEBh1Bk

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

a. Print to sound

• Year 1 ‘Phonics Screening Check’ (DfE):
• Real words and pseudowords
• What’s the difference? Why include pseudowords?

Pseudowords:

• By definition, unknown to the learners
• Learners cannot draw on existing knowledge of the words’ pronunciations

🡪 A ‘pure’ measure of their knowledge of symbol-sound correspondences

However, we considered pseudowords problematic:

• Ethically / pedagogically
• In terms of validity (Papagno, Valentine & Baddeley, 1991)

• Unfamiliar real words
• Low frequency
• Not in SoW to date

Standards and Testing Agency, 2018: 2018 key stage 1 phonics screening check: pupils’ materials

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

a. Print to sound

Familiar versus unfamiliar words

• Woore (2018) – compared Y10 MFL pupils’ pronunciation of familiar / unfamiliar French words
• For each participant, a set of orthographically matched word pairs was created. On average, 14 words pairs per participant.
• Each pair consisted of 1 word they knew and 1 word they didn’t know (as shown by a vocab test), sharing the same spelling body
• Unit of analysis = spelling body (phonological rime)
• …thus controlling for word-level variables

pain

trois

grain

chois

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

a. Print to sound

p < .001, d = 0.33

<pain> - /pæ̃/

<chat> - /ʃa/

<grain> - /ɡɹɛɪn/

<fat> - /fat/

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

a. Print to sound

Summary so far

• Reading Aloud Test
• Administered individually
• Unfamiliar words to test SSC knowledge (to stop them using existing knowledge of whole word pronunciations)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

b. Sound to print

• Transcription task – can they write what they hear?
• However… languages vary in “orthographic depth”:

Image of train tracks: public domain licence, https://libreshot.com/train-tracks-top-view/

Image of tree: Lairich Rig / Thousand-year-old yew tree / CC BY-SA 2.0

<a>

<u>

/a/

/u/

vair

vert

verts

ver

vers

verre

verres

vaires

vaire

verd

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

a. Print to sound

• So, in French particularly 🡪 multiple possible spellings for a given sound

ver, vers, verre, verres, vert, verts, vair

• For unfamiliar words: proliferation of ‘correct’ answers

* vaire, * vaires, * verd, * verds, * vère, * vères, * vaîre, * vaîres …

• So, we tweaked the response format to constrain possible responses
• Blanks for individual letters.
• E.g. “ v _ _ ”

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2. Item selection

For each language:

• We listed all the target SSC covered in the SoW
• For each SSC, we identified 3 low-frequency words (>5000 frequency ranking for French and Spanish, >4037 for German) and ensured not covered in SoW
• Therefore, pupils very unlikely to know them!
• Where possible, we controlled for orthographic and phonological length – i.e. short monosyllabic words (French and German) or mostly 2 syllables (Spanish)
• We tried to avoid words that look like English words (interlingual homographs), taboo words, etc.!

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2. Item selection

• How many items in the test?

• Completeness of coverage for individual pupils – diagnostic feedback
• Testing burden and time demands – aim to fit tests within a single lesson!
• We don’t want assessment to soak up too much teaching and learning time!

Scales image: by scott_kirkwood, openclipart.org.

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2. Item selection

• 15 items per pupil
• On-line test: random selection of SSC for both the reading aloud and dictation components. Random selection (1 of three items) for each SSC
• Class as a whole should cover all the SSC, but individual pupils do not

🗶

• Incomplete diagnostic feedback for individuals

✔

• Takes less time for individuals to complete
• Prevents copying – all individuals have a different selection of items
• Diagnostic feedback (to inform teaching) at whole class level

• But – these are summative tests.
• Diagnostic tests are already available – teacher- / self- / peer-assessment

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Sample items – reading aloud

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Sample items – reading aloud

🡪 Time limit = compromise!

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Sample items – reading aloud

15 items to read aloud

t

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Sample items – dictation

🡪 Time limit = compromise!

Number of items?

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Sample items – dictation

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Pen-and-paper tests:

• Pre-selection of 15 items

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Scoring – reading aloud

< v e a u >

A note on ‘foreign accent’. Two dimensions to accuracy in reading aloud:

1. Accuracy of SSC knowledge: how graphemes / symbols are mapped onto L2 phonemes / sounds
2. Accuracy of pronunciation of L2 phonemes / sounds.

This is our focus for the phonics assessment!

/ vjʉ / / viː əʊ /

/ vo /

/ vəʊ /

✔

✔

🗶

• A foreign accent is hard to shift even for the most dedicated learner (even if they want to)
• You can be intelligible (and comprehensible) with a foreign accent.
• We are testing SSC knowledge (phonics) rather than native-like pronunciation (phonetics).

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Scoring – reading aloud

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Scoring

3. Scoring

Dictation task – automatically scored by the software platform ☺

Reading aloud task – marked individually by teachers (☹?)

• Workload implications – need to make individual decisions about this.
• One option would be to sit with individuals as they complete the speaking part of the test, and mark it in ‘real time’ as they go along.

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Outline

1. Principles of Phonics assessment

1.1 What are we trying to test?

1.2 How can we test it, validly and practically?

​

2. Question types

2.1 Task design

2.2 Item selection

​

3. Scoring

3.1 What do we mean by ‘accurate’ decoding?

3.2 How the scoring works

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes