1 of 15

Assessment design: achievement tests -vocabulary

Vocab testing principles

Question Types�Question type distribution�Question allocation

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2 of 15

Vocab testing principles

Objectives

To test breadth and depth of knowledge of vocabulary studied in Y7 Terms 1.1.1 - 2.1.1

Considerations

Syllabus-based achievement test

Productive vs receptive knowledge

Recall vs recognition

Oral and written modalities

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

The overall aim of the vocabulary test is twofold. We want to find out not only how many of the words in the Year 7 schemes of work students have learned, which I’ll refer to as breadth of knowledge testing, but also how well students know these words, which I’ll call depth of knowledge testing.

To do this, we need to create a syllabus-based achievement test, which means we are only testing students’ knowledge of words encountered in the schemes of work so far. We are not testing their ability to work with cognates, or to infer meaning from context at this stage.

We bore three main considerations in mind when choosing question types. First, we wanted to make sure that students could not only recognise forms and meanings of L2 words when presented with them, but that they could recall these from their memories without the need for prompts. This consideration informed our decisions regarding question formats, such as the number of multi-choice and free production questions to include.

We also wanted to test students’ receptive and productive knowledge in equal measure, which meant that 50% of the questions needed to test listening and reading skills, and 50% written and spoken production. This, in turn, assured that students’ knowledge of both oral and written language was adequately assessed.

3 of 15

Breadth of knowledge

Objective 1: To find out how many words students know

Sample size

Target time: 12 minutes

Pilot testing indicates students can comfortably answer one question in 9 seconds

720 / 9 = 80 total test items (split equally between L, R, W & S)

SOW coverage:

French: 43%* Spanish: 35% German: 35%

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

To get an overall estimate of students’ breadth of knowledge, we would ideally like to test them on each word encountered in the schemes of work so far. Given that students will have learned over 200 words by this point in the year, however, this was deemed unfeasible in terms of both timing and concentration levels.

As the target length of the full Year 7 spring test is 40 minutes, divided between the strands of phonics, vocabulary and grammar, we estimated that a total test time of 12 minutes, or 720 seconds, was an appropriate aim for the vocabulary component. Pilot testing indicated that students could comfortably answer one question in 9 seconds, and so 80 questions in 12 minutes, giving us a nice round target figure of 20 questions for each of the modalities (listening, reading, writing and speaking).

As you can see, these figures results in high scheme of work coverage – 35% of words in the German and Spanish schemes of work will be tested, and 43% of those in French. This suggests the test results will give a highly reliable indication of students’ individual and collective performance – especially when you consider that some major online tests, such as the Vocabulary Size Test for learners of English, consider a coverage of just 1-4% sufficient for this purpose.

4 of 15

Breadth of knowledge

Objective 1: To find out how many words students know

Item pool

To test overall knowledge of words in SOW – words distributed amongst students

Words are equally likely to be selected from randomised pools

All words in the SOW appear in the test once only

Part of speech ratios (noun : verb : other) broadly upheld within different question types

(French: 2:1:2; Spanish: 3:1:3; German: 2:1:3)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Although we’ve established that 80 words is a very reliable sample size, it is still useful for us to test knowledge of all words in the schemes of work, so that we can identify any words which may pose more challenges than others, for example.

One way to do this is to distribute the words to be tested amongst the students. Qualtrics, which is the program we are using to host the tests, allows us to code questions for each vocabulary item in the scheme of work, and then present students with random samples of this. This has the added practical advantage that students cannot copy one another, as no two sets of questions will be exactly the same.

To ensure that students are presented with 80 different words (and not the same word tested in multiple ways) each word must be included in the test only once, meaning that 25% of words in the schemes of work are assigned to reading tests, 25% to listening tests, and so on.

The word class ratios of individual languages, shown here at the bottom of the slide, should also be broadly upheld within different question types, ensuring that students aren’t presented with a random sample containing only nouns, for example.

5 of 15

Depth of knowledge

Objective 2: To find out how well students know target words

Year 8+

tested separately

Year 8+

Nation 2013, p. 538

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Now on to depth of knowledge testing. As previously mentioned, we want our tests to give us an idea of not only how many words students have learned, but how much they know about those words. So, we need to incorporate depth of knowledge testing into our design by choosing question types which assess different elements of word knowledge.

The parts of word knowledge on which NCELP vocabulary teaching is based draw on the aspects identified here by Paul Nation, who divides word knowledge into three main areas – form, meaning and use. If you’ve been adopting the NCELP schemes of work, you’ll notice that students’ first encounters with vocabulary items are in isolation, allowing students to focus on developing their knowledge of the spoken and written form. In future encounters with these words, which are incorporated systematically as part of our regular revisiting cycle, words are presented in context, to allow students to deepen their knowledge of meaning and use.

Not all elements of word knowledge will be tested, as not all have yet been covered in the schemes of work – students’ word pools simply aren’t big enough to incorporate teaching of word family structure or appropriate register use.

Knowledge of the grammatical patterns into which words enter is, of course, assessed in a separate part of the test.

The elements of word knowledge that remain – spoken and written form, all elements of meaning and collocation – are the elements on which our Year 7 teaching, and, therefore testing, is based.

6 of 15

Depth of knowledge

Recognition tests

multiple choice activities whereby learners select or guess the correct response from the alternatives given
such tests may strengthen any existing memory traces (McDaniel & Mason, 1985)

Recall

demands the production of responses from memory
more difficult than recognition because learners must search for the correct response within their mental representation of the newly experienced information (Cariana & Lee, 2001; Glover, 1989; McDaniel & Mason, 1985).

Definitions adapted from Jones (2004)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

7 of 15

Question type 1

Read 2000, Chapter 3

Modality

Listening

Type of activity

Spoken meaning recognition

Knowledge tested

Meaning (definition)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Transcript

So, moving on now to present the seven question types we’ve identified as suitable for the Year 7 spring test.

On this and the slides that follow, you will see a screenshot of how questions are presented to students on the left, and a summary of the question type and knowledge tested on the right. Students who take the test will be presented with 10 examples of each question type 1-6.

Question Type 1, shown here, is a listening test which requires students to hear L2 words in isolation, and recognise their meanings from a set of pictures or English translations presented to them. Students hear words twice on clicking play, with a 1 second gap in between.

I’ve included a reference to Read (2000) in the top right here if you’re interested in reading more about the usefulness of out-of-context presentation of vocabulary for memorising meaning in the early stages of language learning.

�

8 of 15

Question type 2

Read 2000, Chapter 3

Modality

Listening

Type of activity

Spoken meaning recall

Knowledge tested

Meaning (definition) Meaning (association)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

9 of 15

Question type 3

Read 2000, p. 163

Modality

Reading

Type of activity

Written meaning recall

Knowledge tested

Meaning (definition)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

10 of 15

Question type 4

Modality

Reading

Type of activity

Written meaning recall

Knowledge tested

Meaning (definition)

Use (collocation)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

11 of 15

Question type 5

Modality

Writing

Type of activity

Written form recall

Knowledge tested

Form (written)

Meaning (definition)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Transcript

Question 5 is the first of two written exercises which ask students to recall the L2 form of an English word, and provide a translation. Words are presented with an informative context in both languages so that students understand the intended sense of the word, which they may have learned with multiple meanings.

It is important to note that students are not expected to draw on any grammatical knowledge when producing the words. The questions are designed so that expected answers will always take the forms that students have practised on Quizlet. Nouns beginning with consonants will appear alongside their definite articles, nouns beginning with vowels alongside their indefinite articles, verbs in their infinitive forms, and adjectives in their masculine or feminine forms as indicated in question.

12 of 15

Question type 6

Laufer & Nation 1995

Modality

Writing

Type of activity

Written form recall

Knowledge tested

Form (written)

Meaning (definition)

Use (collocation)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

13 of 15

Question type 7

Modality

Speaking

Type of activity

Spoken form recall

Knowledge tested

Form (spoken)

Meaning (definition)

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

14 of 15

Question type distribution

Listening

10 x spoken meaning recognition

10 x spoken meaning recall

Reading

20 x written

meaning recall

Writing

20 x written

form recall

Speaking

20 x spoken

form recall

10 x recognition, 70 x recall

Bias towards recall encourages active vocabulary building from the beginning and minimises use of multi-choice - a score of 16.7% can be achieved by guessing in a 6-choice format, and of 25% in 4-choice

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Transcript

In terms of question distribution, you’ll see on this summary slide that there is a strong bias toward recall in our test. There are two main reasons for this.

First, in keeping with NCELP principles, we want to encourage students to build a strong active vocabulary from the very beginning, and so we are testing them mainly on their ability to retrieve meaning and form from memory.

Second, a focus on recall gives us the freedom to include varied question types, and limit the number of multiple choice exercises. Overreliance on multiple choice allows students to gain a significant number of marks from guessing. In a 6-choice format, students can achieve a score of 16.7% without any knowledge at all. In a 4-choice format, this figure increases to 25%.

We do acknowledge that this results in a more challenging test, and we may revise the ration in future years following analysis of students’ performance in this pilot study.

15 of 15

Question allocation

Factors influencing question type allocation: concreteness, polysemy (multiple meanings)

Factors influencing distractor choice: word class (noun, verb, adj, …), spelling

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

Transcript

The planning process for assigning words to question types involves the use of a very sizeable spreadsheet! Here, you see a snapshot of the planning phase for German.

One factor which must be considered in when allocating question types to specific words is the degree to which a word represents a concrete concept and so can be expressed using an image. More concrete items can be assigned to picture-matching tasks, while more abstract words lend themselves better to context-based exercises – where the sentence helps to give meaning to the target word. The degree to which words are polysemous also plays a role, as this also affects the amount of context needed to ensure that the target meaning we wish to elicit is clear to students.

Thought must also go into choosing appropriate distractors for multiple choice questions. Tallies must be kept to record the number of times each word in the scheme of work appears as a distractor, so as not to over- or underrepresent any particular word.

Two main factors influence the choice of distractors appearing alongside a target item. First, all distractors must belong to the same word class as the target item, so that students cannot use their grammatical knowledge to select answers by process of elimination. Second, the spelling of words used as distractors plays a role in question differentiation – distractors similar in form to the target item, containing many of the same letters, add an extra challenge.