1 of 11

Assessment design: achievement tests - phonics

Principles of phonics assessment

Item selection

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2 of 11

Principles of Phonics assessment

Objectives

To test knowledge of Symbol-Sound Correspondences (SSC) covered to date in Y7 (i.e. a syllabus-based achievement test)

Why are we testing this?

SSC knowledge is used

when reading aloud (including ‘inner voice’) – linked to reading comprehension, vocabulary acquisition (inter alia)
when listening (segmentation) and writing (spelling)

Testing is bidirectional: both sound 🡪 print and print 🡪 sound

transcription reading aloud

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

The phonics assessment aims to test pupil’s knowledge of the Symbol-Sound Correspondences (or SSC hereafter) which they covered to date in Year 7. In other words, it is a syllabus-based achievement test. It is trying to find out to what extent they have actually learnt what we are trying to teach them in phonics up to the point of testing.

There are various other presentations and handouts dealing with the rationale for teaching phonics so we won’t go into that in great depth here but we believe SSC knowledge is important, both when going from print to sound so for example when we are reading aloud, it might not actually be out loud but with one’s inner voice. This has been linked to various aspects of language learning such as reading comprehension and vocabulary acquisition. It is also important when going in the other direction, so for example from sound to print, so for example when spelling words when you are writing, also when listening to help you segment the stream of speech that is flowing past you as you listen. To reflect these two directions, print to sound and sound to print, our test includes both a transcription tasks—so to what extent can people hear the language and write it down?—and also a reading aloud test looking at the extent to which people can convert the written language into appropriate phonological forms.

3 of 11

a. Print to sound

Summary

Reading Aloud Test
Administered individually
Unfamiliar words to test SSC knowledge (to stop them using existing knowledge of whole word pronunciations)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

4 of 11

b. Sound to print

Transcription task – can they write what they hear?
However… languages vary in “orthographic depth”:

Image of train tracks: public domain licence, https://libreshot.com/train-tracks-top-view/

Image of tree: Lairich Rig / Thousand-year-old yew tree / CC BY-SA 2.0

<a>

<u>

/a/

/u/

vair

vert

verts

ver

vers

verre

verres

vaires

vaire

verd

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

We then had to decide how to test SSC when moving in the other direction so from sound to print and for this we decided upon a transcription task, so in other words can learners write down what they here accurately. One of the challenges we then came up against is that languages vary in their orthographic depth so orthographic depth is the extent to which a writing system has transparent and consistent mappings between written symbols on the one hand and spoken sounds on the other. There is much more information about this in some of the other phonics presentations so for example if you looked on the Resource Portal for the phonics presentation from Residential 1 there is more information on this but just to recap we said that Spanish we use these visual metaphors which we thought might be nice to use with pupils to raise awareness of the nature of the learning challenge that they are up against.

In Spanish we had this image of parallel train tracks because for a given written symbol there tends to be usually just one way of pronouncing it so the written letter ‘a’ tends to be pronounced /a/ but conversely for a given sounds /a/ there tends to be just one way to write it, so the written vowel a. English we said is much more complicated and we had a visual metaphor of a cats’s cradle because, and we suggested this might be a nice exercise to do with pupils actually, if you take the sounds /u/ it can be written in multiple different ways as you can see on the left and if you took one of these written representations like the ‘ough’ obviously that can be pronounced in a number of different ways so English is much less consistent in its mappings both from sound to print and from print to sound so for that reason English is considered an extremely deep alphabetic orthography. French, finally, we said was an interesting case because, and we used the image of a tree at least above the ground, because we said for a given spoken form like the word /ver/, the sound /ver/, it could be written in a number of different ways, as you can see here ‘verre’ ‘vers’ ‘ver’ and in some ways, they are not in fact real words, so pseudo-words, like these three here. So, there are actually multiple ways of writing down this same spoken sound but for a given written representation, so for each of these like the leaves on the tree, there is only one way of pronouncing them which is “ver”. So, French is interesting because it is asymmetrical in its mappings.

5 of 11

b. Sound to print

So, in French particularly 🡪 multiple possible spellings for a given sound

For unfamiliar words: proliferation of ‘correct’ answers

ver, vers, verre, verres, vert, verts, vair

* vaire, * vaires, * verd, * verds, * vère, * vères, * vaîre, * vaîres …

So, we tweaked the response format to constrain possible responses

Blanks for individual letters.
E.g. “ v _ _ ”

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

So, the upshot of this was, particularly in French of our three target languages, there are multiple possible spellings for a given sound so if we are thinking about a transcription task the problem is that we end up with a proliferation of possible correct answers. As an example, if our spoken word had been ‘ver’ we would have had to allowed all of these real words and all of these pseudo words as correct answers and because the words are unfamiliar to the learners you can’t say what is the correct way to spell it. All of these would have been permissible spellings of this sound. Luckily, we managed to find a simple solution to this challenge, what we did was tweak the response format so we blanked out individual letters so in this example here if we leave two blanks like this _ _, then in fact there is only one correct answer in how to write the spoken form “ver” and none of the others would fit the two gaps. We used this blanking out letters mechanism just to constrain the possible responses.

6 of 11

Item selection

For each language:

We listed all the target SSC covered in the SoW
For each SSC, we identified 3 low-frequency words (>5000 frequency ranking for French and Spanish, >4037 for German) and ensured not covered in SoW
Therefore, pupils very unlikely to know them!
Where possible, we controlled for orthographic and phonological length – i.e. short monosyllabic words (French and German) or mostly 2 syllables (Spanish)
We tried to avoid words that look like English words (interlingual homographs), taboo words, etc.!

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

In terms of item selection, what we did for each language was we listed all of the target SSCs covered in the Scheme of Work and for each of those we then identified three low-frequency words which exemplify that particular SSC. So, in French and Spanish they are all less-frequent than the 5000 most frequent words in the language using the corpus that we used, the reference source that we used. In German, they are all less frequent than the 4,037 most frequent words and the reason for the different figures there is it is just what the frequency dictionaries that we used went up to. We also ensured that the words had not been covered previously in the Scheme of Work and so we concluded that pupils are very unlikely to know any of these words. Hopefully, we are getting a pure measure of their SSC knowledge. Where possible we also controlled for things like orthographic length, so how many letters the words contain in their written form and also their phonological length so how many syllables. In French and German, they are short monosyllabic words and in Spanish they are mostly bisyllabic. We also tried to avoid words that look like English words and we tried to avoid rude words, or words that look like rude English words and so forth, and thank you to those people who picked up on a couple of items that slipped through the net in the first pass. Hopefully, we have ended up with a set of word which meet all of those criteria.

7 of 11

SSC in French

SSC	French 1	French 2	French 3
SFC –x	toux	poux	houx
SFC -d	jard	lard	dard
SFC –t	prit	vit	mit
SFC –s	bris	gis	spis
a	nase	prase	jase
i	brime	cime	trime
eu	yeuse	beuse	Meuse
e	reloge	remous	recel
o	lot	plot	trot
eau	veau	seau	sceau
au	chaule	gaule	taule
u	cossu	cornu	crépu
ou	zou	clou	flou
é	pré	dé	ré
en	ment	sent	gent

French SSC	French 1	French 2	French 3
an	flan	cran	ban
on	pond	tond	gond
SFe	rame	crame	lame
in	crin	clin	pin
ain	tain	nain	zain
è	sème	pèse	sève
ê	vête	blême	bêche
ai	daine	raine	gaine
oi	aloi	coi	aboi
ch	croche	hoche	loche
ç	glaçon	maçon	pinçon
qu	quinte	quine	quille
j	jauge	jonc	joug
-tion	dation	rection	brution
-ien	salien	jovien	danien

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

8 of 11

SSC	Spanish 1	Spanish 2	Spanish 3
a	aneblar	anejar	anegar
e	ente	ende	enqué
i	impar	inflar	inapto
o	opo	oteo	oso
u	umbral	ubre	urna
ll	llagar	llorón	llacsa
ca	cantal	caló	casal
co	copón	coser	cota
cu	cubil	cuna	cutre
ce	cerner	cera	cedro
ci	cisne	cisma	cima
z	zapar	zanja	zarpa

SSC	Spanish 1	Spanish 2	Spanish 3
que	quemón	quejón	quedón
qui	quico	quiá	quilco
ga	gamba	gamo	ganga
go	gozque	godo	goma
gu	guma	gula	gusa
j	jirón	jirafa	jinete
ge	gemir	germen	gema
gi	gibón	gilar	gibar
ñ	ñoño	ñapa	ñaque
v	vaina	vasco	valla
rr	arralar	arrasar	arroba
r	aparar	amparar	aparear
h	hendir	henchir	hender

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

9 of 11

German SSC	German 1	German 2	German 3
a long	Gral	Mahl	Pfahl
a short	prall	satt	pat
e long	nebst	bebt	hebt
e short	bleckt	Depp	Speck
ei	Gleis	reiß	rein
i long	Pik	ihrzt	Nil
i short	Rind	litt	stritt
ie	Viech	Vieh	lieh
ch soft	glich	strich	wich
ch hard	Joch	roch	Bach
o long	Thron	roh	Floh
o short	flott	moll	Zoll
u long	Stur	Schnur	Schur
u short	Sund	Pult	Spund
er stressed	Werft	nervt	perlt

German SSC	German 1	German 2	German 3
er unstressed	Gießer	Becher	Dreher
ss, final s, ß	Steiß	beißt	Strauß
ü long	schürt	kühlt	schnürt
ü short	zückt	knüllt	hüllt
ö long	Föhn	krönt	tönt
ö short	Mönch	gönnt	Löss
ä long	brät	rät	sät
ä short	ätzt	dämmt	schwänzt
äu	häuft	säuft	säumt
w	Walm	warf	webt
z	zart	zahm	zupft
sch	Schoß	Schaum	schnöd
v	viert	vors	vorm
s (like z)	Sau	Sumpf	sinkt
sp-	Spieß	spuckt	spann

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

10 of 11

Item selection

How many items in the test?

Completeness of coverage for individual pupils – diagnostic feedback
Testing burden and time demands – aim to fit tests within a single lesson!
We don’t want assessment to soak up too much teaching and learning time!

Scales image: by scott_kirkwood, openclipart.org.

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

In designing the test, we came upon all kinds of decisions that we had to take such as how many items? Again, this was a balancing act. On the one hand, it would have been nice to have complete coverage of all the SSCs for each individual pupil then we would have been able to provide diagnostic feedback at the individual pupil level, in other words we would have been able to tell individual pupils which SSC they got right and wrong and which they needed to do further work on. But we also had to think about the so-called testing burden, how much time and effort it would take to do the test and we had to bear in mind that they are not only doing the phonics test but also the vocabulary and grammar and we wanted, as far as possible, for it to be able to fit within a single lesson and we didn’t want the assessment to soak up too much teaching and learning time because after all that is the most important thing.

11 of 11

Item selection

15 items per pupil
On-line test: random selection of SSC for both the reading aloud and dictation components. Random selection (1 of three items) for each SSC
Class as a whole should cover all the SSC, but individual pupils do not

✔

Takes less time for individuals to complete
Prevents copying – all individuals have a different selection of items
Diagnostic feedback (to inform teaching) at whole class level

🗶

Incomplete diagnostic feedback for individuals

But – these are summative tests.
Diagnostic tests are already available – teacher- / self- / peer-assessment

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

We came up with the figure of 15 items per pupil both for the reading aloud task and for the transcription task, that’s in the first iteration of these tests and we weren’t exactly sure whether that is the right number and whether it would need to change, perhaps reduce, if pupils seem to need a bit longer for each item, to think about each item. So, it maybe that if you are listening to this presentation at some later point it may be that there is now a different number of items but our initial go at this was to have 15 items per pupil so in the online test, the one that is done via the platform called Qualtrics there is a random selection from all the SSC which are covered in the Scheme of Work, there is a random selection of 15 of those. 15 for the reading aloud task and then a random selection of 15 for the dictation task and then for each of those SSC, you will remember on the previous slide, there were three words per SSC so one of these is randomly selected for each SSC for each pupil. What this mean is: first of all the class as a whole should cover all of the SSCs between them, but an individual pupil will not; and it also means that because for a given SSC there is one word out of a given random three selected it should mean that two pupils sitting next to each other are doing this test online at the same time they should have a different set of items both to read aloud and to transcribe in the dictation test so we think this has some advantages, the main one in our minds when deciding to do it this way was that it would take less time for individuals to complete and that it would be impossible for pupils to copy off each other because they are all doing different items. We think that there should still be scope for diagnostic feedback to inform planning and teaching at the whole class level so as a whole class it should still be possible to see which SSCs students, on the whole, are doing well with.