Session 4: Assessment��… or: how do we know whether the pupils are getting it, and how can it fit into our existing reporting structures?

Stuart Williams

​

Dartford Grammar School

​

20/11/20

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Principles

1. If we are focusing on phonics, vocabulary and grammar in our teaching, we also need to focus on these in our assessment.
2. We also need to understand how students are able to apply their knowledge more holistically and respect existing KS3 reporting systems (which are often set up to report on listening, speaking, reading and writing).
3. We should avoid excessive demands on teacher workload.

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

‘Achievement’ �Vs. �‘Applying your knowledge’ tests

The ‘Achievement’ tests (February & June each year)

• Tests to what extent students have learnt everything up to test
• Assesses a principled sample of P,V, and G

​

The ‘Applying your knowledge’ test (June each year)

• Brings together PVG in more holistic assessments
• As ‘knowledge’ turns to ‘skill’: more recognisable listening, reading, speaking and writing tasks.
• Assesses ability to apply knowledge in context through:
• listening comprehension,
• oral picture description,
• short translation tasks.

​

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Availability of NCELP assessments

• Paper assessments are currently available, in all three languages, for:
• Year 7 Achievement tests for Feb and June testing cycle (undergoing minor revisions for new release 25^th Jan 2021)
• Year 7 Applying your knowledge tests for June testing cycle

​

• Paper assessments will be available 25^th Jan 2021 in all three languages for:
• Year 8 Achievement tests for February testing cycle

​

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Online tests

• NCELP is also developing online assessments, currently available to schools in the NCELP network
• Advantages of online testing include:
1. ease of delivery
2. control of the speed/timing
3. automated scoring
4. add items and randomise
5. ability to collate and analyse data on a large scale
• Online tests can be made available to schools outside the NCELP network on request via enquiries@ncelp.org upon confirmation that the relevant NCELP SOW has been followed, specifying the language & year, since Sept 2020 (for y7) Sept 2019 (for y8).
• Currently we estimate we can accommodate up to 50 such requests for the February tests, as we need to extract data and return scores to individual teachers
• This session draws on examples from both paper and online assessments, but the question types and principles are identical

​

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Principles

1. If we are focusing on phonics, vocabulary and grammar in our teaching, we also need to focus on these in our assessment.
2. We also need to understand how students are able to apply their knowledge more holistically and respect existing KS3 reporting systems (which are often set up to report on listening, speaking, reading and writing).
3. We should avoid excessive demands on teacher workload.

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Assessment of phonics

1. Principles of Phonics assessment

​

2. Question types and examples

​

3. Scoring and marking

PHONICS

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

1. Principles of Phonics assessment

Objectives

• To test knowledge of Symbol-Sound Correspondences (SSC) covered to date

Why are we testing this?

• SSC knowledge is used
• when reading aloud (including ‘inner voice’) – linked to reading comprehension, vocabulary acquisition (inter alia)
• when listening (segmentation) and writing (spelling)
• Testing is bidirectional: both sound 🡪 print and print 🡪 sound

transcription reading aloud

PHONICS

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

a. Sound to print

• Transcription task – can they write what they hear?
• At least we not assessing their English…

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

b. Print to sound

• Reading Aloud Test
• Administered individually
• Unfamiliar (but real!) words to test SSC knowledge �(to stop them using existing knowledge of whole word pronunciations)

PHONICS

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2. Question types

For each language:

​

• We listed all the target SSC covered in the SoW;
• For each SSC, we identified 3 low-frequency words and ensured they were not covered in SoW, so that pupils are very unlikely to know them;
• We tried to avoid words that look like English words.

PHONICS

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Example 1 - reading aloud, reading 🡪 speaking

15 items to read aloud

PHONICS

ACHIEVEMENT

Read the following list of German words aloud.

You won't know these words. Just say them as you think they should sound in German.

You will get marks for pronouncing the bold, underlined parts of each word.

If you’re not sure, don’t worry – just have a go and do your best.

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Example 2 – dictation, listening 🡪 writing

PHONICS

ACHIEVEMENT

You will hear 15 Spanish words. You will hear each word twice.

​

Complete the spelling of each word by filling in the missing letters.

​

There may be more than one way of spelling them.  Just type in any one possible spelling for each word.

You won’t know these words. Don’t worry – just do your best! 

​

1. _ mbral
2. _ _ ma
3. _ _ mba
4. _ _ rafa
5. _ _ na
6. a _ _ oba

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Scoring – reading aloud

A note on ‘foreign accent’. Two dimensions to accuracy in reading aloud:

1. Accuracy of SSC knowledge: how graphemes / symbols are mapped onto L2 phonemes / sounds
2. Accuracy of pronunciation of L2 phonemes / sounds.

This is our focus for the phonics assessment!

< v e a u >

/ vjʉ / / viː əʊ /

🗶

/ vo /

✔

/ vəʊ /

✔

• A foreign accent is hard to shift even for the most dedicated learner (even if they want to)
• You can be intelligible (and comprehensible) with a foreign accent.
• We are testing SSC knowledge (phonics) rather than native-like pronunciation (phonetics).

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Marking

PHONICS

ACHIEVEMENT

Dictation task (can be automatically scored by the software platform ☺

Reading aloud task (has to be marked individually by teachers as machines don’t yet recognise learner speech in these three languages! ☹

Workload implications – need to make individual decisions about this.

Only takes 1-2 minutes per pupil, and, with a little practice, can be marked ‘live’.

Could use this formatively, and so just sample a few pupils in your class to garner a range of achievement to date?

​

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Assessment of vocabulary

1. Principles of vocabulary testing

​

• Question types and examples

​

• Scoring and marking

VOCAB

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

1. Principles of vocab testing

Objectives

​

To test a) breadth and b) depth of knowledge of vocabulary studied

​

​

Considerations

​

• Syllabus-based achievement test

​

• Productive vs receptive knowledge

​

• Recognition vs recall

​

• Oral and written modalities

VOCAB

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

a. Breadth of knowledge

Objective 1: To find out how many words students know

​

​

• Tests knowledge of a sample of words from SOW

​

• All words in the SOW appear in the test once only

​

• Good coverage of different parts of speech (noun : verb : other)

(French: 2:1:2; Spanish: 3:1:3; German: 2:1:3)

VOCAB

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

b. Depth of knowledge

Objective 2: To find out how well students know target words

​

Recognition tests

• multiple choice activities whereby learners select or guess the correct response from the alternatives given

​

Recall tests

• demands the production of responses from memory
• more difficult than recognition because learners must search for the correct response from the stores in their mind

VOCAB

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2. Question types

80 test items

Listening

​

20 x spoken meaning recognition

Reading

​

20 x written

meaning recall

Writing

​

20 x written

form recall

Speaking

​

20 x spoken

form recall

40 receptive items

40 productive items

Bias towards recall:

​

• encourages active vocabulary building from the beginning

​

• minimises use of multi-choice format - average score increase of 16.7% due to guessing in a 6-choice format (Stewart & White, 2011) and up to 25% in 4-choice (Stewart 2014)

VOCAB

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Example 1

Modality

Listening

​

​

Type of activity

Spoken meaning recognition

​

​

Knowledge tested

Meaning (definition)

VOCAB

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Example 2

Modality

Listening

​

​

Type of activity

Spoken meaning recognition

​

​

Knowledge tested

Meaning (definition) Meaning (association)

VOCAB

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Example 3

Modality

Reading

​

​

Type of activity

Written meaning recall

​

​

Knowledge tested

Meaning (definition)

VOCAB

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Example 4

Modality

Writing

​

​

Type of activity

Written form recall

​

​

Knowledge tested

Form (written)

Meaning (definition)

VOCAB

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Example 5

Modality

Speaking

​

​

Type of activity

Spoken form recall

​

​

Knowledge tested

Form (spoken)

Meaning (definition)

VOCAB

ACHIEVEMENT

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Scoring (tolerance)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Assessment of grammar

1. Principles of grammar testing

​

2. Example items

​

3. Scoring and marking

GRAMMAR

ACHIEVEMENT

1. Principles of grammar testing

Objective

To test receptive and productive knowledge of grammar studied

​

Considerations

​

• Isolating grammatical knowledge; disassociating from vocab knowledge

​

• Ensuring coverage of the range of grammar features taught

​

• Receptive knowledge (recognising meaning as well as form)
• In reading and listening
• Productive knowledge (accuracy of production)
• In writing and speaking

GRAMMAR

ACHIEVEMENT

2. Example 1

Testing written and aural receptive knowledge

• Testing ability to recognise meaning as well as form

​

• Multiple choice; here matching to English equivalent

GRAMMAR

ACHIEVEMENT

2. Test creation: example question

Testing written and aural receptive knowledge

• Testing ability to recognise meaning as well as form

​

• Multiple choice; here matching to TL alternative

Isolating recognition of gender and number

Example 2

GRAMMAR

ACHIEVEMENT

Isolating productive knowledge of syntax (word order)

• Drag-and-drop into correct order

Example 3

GRAMMAR

ACHIEVEMENT

2. Test creation: example items

Testing written productive knowledge

• Testing ability to accurately produce the structure
• Open text box

Testing understanding of question formation

(Check pupils understand that ‘do’ auxiliary is not needed)

Isolating grammatical knowledge by providing verb infinitive

(Avoid missing answers due to lack of vocabulary)

Example 4

GRAMMAR

ACHIEVEMENT

Testing oral productive knowledge

• Testing ability to accurately produce the structure(s)
• Combining a number of structures within each item

Testing subject-verb agreement.

Isolating grammatical knowledge (from vocab knowledge) by providing verb infinitive

Example 5

GRAMMAR

ACHIEVEMENT

3. Scoring

Our mark scheme recommends:

0.5 mark for incorrect indefinite article (but gender correct)

0.5 mark for incorrect case

(but gender correct)

0.5 mark for incorrect indefinite article (but gender correct)

0.5 mark for incorrect case and definite article (but gender correct)

GRAMMAR

ACHIEVEMENT

PVG in terms of “LRSW” in achievement tests

Principles

1. If we are focusing on phonics, vocabulary and grammar in our teaching, we also need to focus on these in our assessment.
2. We also need to understand how students are able to apply their knowledge more holistically and respect existing KS3 reporting systems (which are often set up to report on listening, speaking, reading and writing).
3. We should avoid excessive demands on teacher workload.

Applying your Knowledge test - Listening

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Applying your Knowledge test - Reading

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Applying your Knowledge test - Writing

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Applying your Knowledge test: Speaking

Look at the picture.

​

Describe it to a friend by saying these sentences in French:

​

1. Amir is wearing a green shirt.

2. It is nice weather in England today.

3. Three dogs are playing in the park.

4. There is an island and there are two bridges, but there is no café.

5. The church is in front of the big buildings.

​

​

To work out what to say, try to break each sentence into groups of words.

​

Before you start the test, please go to: vocaroo.com. It will open in a new tab.

Click the red record button, then come back to this test.

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Principles

1. If we are focusing on phonics, vocabulary and grammar in our teaching, we also need to focus on these in our assessment.
2. We also need to understand how students are able to apply their knowledge more holistically and respect existing KS3 reporting systems (which are often set up to report on listening, speaking, reading and writing).
3. We should avoid excessive demands on teacher workload.

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

• All tests are available as paper versions
• Elements that are available online are self-marking
• Each test comes with accompanying guidance, transcripts, mark schemes and short/long scoring spreadsheets
• Assessment materials are organised as collections on the Resource Portal e.g., Fr, Germ, Sp: year 7 term 2

Marking and scoring

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Assessment Collections

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Want to give a flavour of the tests to pupils?

There are PowerPoint presentations and videos of teachers delivering a lesson to prepare pupils for the test at the Oak Academy website.

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Links to tests, audio files, mark schemes

Tests, including audio, transcripts and scorings spreadsheets are available in the

French, German & Spanish “test collections” on the NCELP Resource Portal. Quick links to the Y7 assessments are linked in the table below

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Oak Academy website

www.thenational.academy

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Summary

1. The NCELP assessments ascertain the progress the pupils are making through the SOW content (achievement) and their ability to apply their knowledge (applying your knowledge).
2. They are designed to give a thorough judgment, using sound research as to what good progress looks like.
3. They assess progress on PVG and LRSW.
4. All tests are available in paper versions in collections, with all accompanying materials required.
5. Elements of the tests (listening and reading) can be completed on computer (self-marking) for NCELP schools.

Also, up to 50 others, who confirm they have followed the NCELP SOW, provide language, year group, and date pupils started NCELP SOW by emailing enquiries@ncelp.org (first come, first served)

• Videoed preparation lessons are available on www.thenational.academy .

​

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Session 4: Assessment

​

Q&A

Stuart Williams

​

Dartford Grammar School

​

20/11/20

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes