SOW-based vocabulary testing for NCELP Y7�

Principles, design, creation

​

NCELP Residential 3

​

Natalie Finlayson / Emma Marsden

​

Date updated: 02/03/20

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Outline

1. Vocab testing team

​

• Principles of vocabulary testing

​

2.1 Breadth of knowledge testing

​

2.2 Depth of knowledge testing

​

3. Question types

​

• Scoring

​

• Summary

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

1. Vocab testing team

Test leads: Natalie Finlayson & Emma Marsden

French

​

Test developer

Kirsten Somerville

​

Proofreader

Ivan Avaca

​

​

Spanish

​

Test developer

Ivan Avaca

​

Proofreaders

Nick Avery

Pep Mateos Gonzalez

​

German

​

Test developers

Inge Alferink

Natalie Finlayson

​

Proofreaders

Inge Alferink

Natalie Finlayson

With huge thanks to: Giulia Bovolenta, Victoria Hobson, Stephen Owen, Helen Thomas, Catherine Morris, Ciaran Morris, Jack Peacock, Laurence Anthony

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2. Vocab testing principles

Objectives

​

To test breadth and depth of knowledge of vocabulary studied in Y7 Terms 1.1.1 - 2.1.1

​

​

Considerations

​

• Syllabus-based achievement test

​

• Productive vs receptive knowledge

​

• Recall vs recognition

​

• Oral and written modalities

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2.1 Breadth of knowledge

Objective 1: To find out how many words students know

​

Sample size

​

​

• Target time: 12 minutes

​

• Pilot testing indicates students can comfortably answer one question in 9 seconds

​

• 720 / 9 = 80 total test items (split equally between L, R, W & S)

​

SOW coverage:

​

French: 43%* Spanish: 35% German: 35%

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2.1 Breadth of knowledge

Objective 1: To find out how many words students know

​

Item pool

​

• To test overall knowledge of words in SOW – words distributed amongst students

​

• Words are equally likely to be selected from randomised pools

​

• All words in the SOW appear in the test once only

​

• Part of speech ratios (noun : verb : other) broadly upheld within different question types

(French: 2:1:2; Spanish: 3:1:3; German: 2:1:3)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2.2 Depth of knowledge

Objective 2: To find out how well students know target words

​

Nation 2013, p. 538

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2.2 Depth of knowledge

How well do you know the words we have learned so far?

​

​

1. I have seen this word before.

2. I know what the word means.

3. I can read the word aloud.

4. I can spell the word correctly.

5. I can use the word in a sentence.

6. I know the gender of nouns.

​

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2.2 Depth of knowledge

Recognition tests

​

• multiple choice activities whereby learners select or guess the correct response from the alternatives given
• such tests may strengthen any existing memory traces (McDaniel & Mason, 1985)

​

Recall

​

• demands the production of responses from memory
• more difficult than recognition because learners must search for the correct response within their mental representation of the newly experienced information (Cariana & Lee, 2001; Glover, 1989; McDaniel & Mason, 1985).  

​

Definitions adapted from Jones (2004)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Question type 1

Modality

Listening

​

​

Type of activity

Spoken meaning recognition

​

​

Knowledge tested

Meaning (definition)

Read 2000, Chapter 3

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Question type 2

Modality

Listening

​

​

Type of activity

Spoken meaning recognition

​

​

Knowledge tested

Meaning (definition) Meaning (association)

Read 2000, Chapter 3

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Question type 3

Modality

Reading

​

​

Type of activity

Written meaning recall

​

​

Knowledge tested

Meaning (definition)

Read 2000, p. 163

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Question type 4

Modality

Reading

​

​

Type of activity

Written meaning recall

​

​

Knowledge tested

Meaning (definition)

Use (collocation)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Question type 5

Modality

Writing

​

​

Type of activity

Written form recall

​

​

Knowledge tested

Form (written)

Meaning (definition)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Question type 6

Modality

Writing

​

​

Type of activity

Written form recall

​

​

Knowledge tested

Form (written)

Meaning (definition)

Use (collocation)

Laufer & Nation 1995

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Question type 7

Modality

Speaking

​

​

Type of activity

Spoken form recall

​

​

Knowledge tested

Form (spoken)

Meaning (definition)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Question types

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

3. Question types

80 x test items

Listening

​

20 x spoken meaning recognition

Reading

​

20 x written

meaning recall

Writing

​

20 x written

form recall

Speaking

​

20 x spoken

form recall

40 x receptive

40 x productive

Bias towards recall:

​

• encourages active vocabulary building from the beginning

​

• minimises use of multi-choice format - average score increase of 16.7% due to guessing in a 6-choice format (Stewart & White, 2011) and up to 25% in 4-choice (Stewart 2014)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

4. Scoring (binary)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

4. Scoring (tolerance)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

4. Scoring (tolerance)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

4. Scoring (tolerance)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

5. Summary

The NCELP Y7 vocabulary test …

​

​

• is a syllabus-based vocabulary test designed to measure vocabulary breadth and depth

​

• provides a highly reliable snapshot of student achievement in a manageable timeframe

​

• tests all words featured in the SOW tested by distributing randomly amongst students

​

• tests recognition and recall skills across four modalities

​

• thoroughly tests different elements of word knowledge tested, in line with aspects taught

​

• provides automated scoring of 6/7 question types

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

References

Cariana, R. B., & Lee, D. (2001). The effects of recognition and recall study tasks with feedback in a computer-based vocabulary lesson. Educational Technology Research & Development 49 (3), pp. 23-36.

​

Glover, J. A. (1989). The "testing" phenomenon: Not gone but nearly forgotten. Journal of Educational Psychology, 81(3), 392-399.

​

Jones, L. (2004). Testing L2 vocabulary recognition and recall using pictorial and written test items. Language Learning & Technology 8 (3), pp. 122-143.

​

Laufer, B., & Nation, P. (1995). Vocabulary Size and Use Lexical Richness in L2 Written Production. Applied Linguistics, 16, pp. 307-322.

​

McDaniel, M. A., & Mason, M. E. J. (1985). Altering memory representations through retrieval. Journal of experimental psychology. Learning, Memory and Cognition 11, pp. 371-385.

​

Nation, I. S. P. (2013). Learning vocabulary in another language. Cambridge: Cambridge University Press

​

Read, J. (2000). Assessing Vocabulary. Cambridge: Cambridge University Press.

​

Stewart, J. (2014). Do Multiple-Choice Options Inflate Estimates of Vocabulary Size on the VST? Language Assessment Quarterly, 11(3), pp. 271–282. doi:10.1080/15434303.2014.922977

​

Stewart, J., & White, D. A. (2011). Estimating guessing effects on the vocabulary levels test for differing degrees of word knowledge. TESOL Quarterly, 45 (2), pp. 370–380. doi:10.5054/tq.2011.254523

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Questions?

Thanks for listening!

​

​

Any questions?

natalie_eloise

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes