1 of 14

Assessment design: achievement tests -grammar

Coverage of grammar

Test creation process

Question Types��

Material licensed as CC BY-NC-SA 4.0�

Rachel Hawkes

2 of 14

Coverage of grammar - German

Grammar feature	Reading	Listening	Writing	Speaking
Present continuous formation Two forms in English vs. one in TL	-	-	4 items	6 items
Question formation Subject-verb inversion; do-aux in English vs. TL	4 items	4 items
Subject-verb agreement (weak) 1^st / 2^nd / 3^rd singular	3 items	3 items
Subject-verb agreement (irregular) haben / sein (1^st, 2^nd, 3^rd sing, 1^st pl); mögen (1^st, 2^nd, 3^rd sing)	3 items	3 items	4 items	1 item
Article agreement Def / indef; gender; number; case (nom/acc)	4 items	-	4 items	3 items
Plural noun formation -en; umlaut + -e; -e	-	-	4 items	-
Negation nicht + verb; nicht + adjective	4 items	-	-	-
Subject and object pronoun agreement Gender; number; case (nom/acc)	4 items	-	3 items	3 items

The table here shows the coverage of grammar features in the German version of the test.

By coverage here, this refers to what the pupils are actually being tested on. In the columns labelled Reading, Listening, Writing, and Speaking, I have indicated the number of items each pupil responds to for that grammar feature so you can see how the different grammatical forms were spread across the different modes and modalities.

The first step in building the grammar test was to identify which grammar features and rules had been taught so far and identify the specific structures that had been introduced for each. For example, in terms of subject-verb agreement for weak verbs in German, the 1^st, 2^nd, and 3^rd, person singular forms have been introduced so far and so the items targeting that particular grammatical structure included instances of all of those particular forms that had been introduced so far.

Ideally, we would have tested everything in all modes and modalities, but that was simply not possible within the time constraint that we had.

Therefore, wherever possible, and as you’ll see from the table, we have tried to ensure that each structure has been tested receptively and productively either in written or spoken form.

Within the productive items, particularly in the Speaking part of the test, we were able to combine testing of multiple structures within one question, for example by asking students to produce a short phrase or sentence which would contain a number of different grammatical features. For example, in the Speaking part of the test, we have a set of items in which students had to create statements and questions with weak verbs, therefore were testing both word order and verb agreement in that set of items.

In the German test specifically we made the decision not to test negation (i.e. kein + noun) or the possessive adjective (mein +noun) in this round of testing and that was because the 2^nd and 3^rd person possessive adjectives will be introduced later in the spring / summer terms. Therefore, we felt it was more efficient to test all of these structures together in one go at the end of Year 7.

(Updated to reflect Feb 21 issue of Gm7.1.2.6 achievement test)

3 of 14

Coverage of grammar - French

Grammar feature	Reading	Listening	Writing	Speaking
Preposition “to” + article	-	4 items	-	-
Subject-verb agreement (regular -ER) 1^st / 2^nd / 3^rd singular; 1^st / 2^nd / 3^rd plural	8 items	-	7 items	9 items
Subject-verb agreement (irregular) être; avoir; faire 1^st / 2^nd / 3^rd singular; 1^st / 2^nd / 3^rd plural	6 items	-
Present continuous formation Two forms in English vs. one in TL		-
Question formation Intonation; do-aux in English vs. TL	-	-	-
Article & adjective agreement Def / indef; gender; number	3 items	-	2 items
Uses of “avoir” and “être”	3 items	-	-	-
Adjectival word order Post-nominal	3 items	-	-	-
Noun gender agreement	2 items	-	-	-
Two-verb structures	2 items	-	2 items	-
Possessive adjectives mon/ma/mes; ton/ta/tes	2 items	-	-	-
Adjective agreement Gender; number	-	-	2 items	-
Subject pronouns il / elle meaning “it”	-	-	2 items	-
Noun phrases	-	-	-	2 items

The table here shows the coverage of grammar features in the French version of the grammar test. Again you can see the number of items included for each grammar structure in the Reading, Listening, Writing and Speaking columns. The first two structures in the table were only tested productively, because here we wanted to know that students recognise that the two present tense forms in English (the simple and present continuous) map onto only one form in French. And similarly, we wanted to check that students recognise that French does not have an equivalent of the English ‘do-auxiliary’ for questions. Therefore, for these two particular structures it seemed to make most sense to provide students with an appropriate English sentence and a test that they knew how to accurately form it in French.

In addition, French only includes one set of items in the Listening part of the test. That is because many of the structures (for example most of the verb endings) are not audible in French, therefore it made most sense to focus on testing these in Reading, rather than Listening parts of the test.

We’ve tried to make best use of the time available for the grammar section of the test, in this particular case focussing more on the written rather than the oral modality when we are testing receptive knowledge in French.

(Updated to reflect Feb 21 issue of Fr7.1.2.6 achievement test)

4 of 14

Coverage of grammar - Spanish

Grammar feature	Reading	Listening	Writing	Speaking
Question formation Intonation; do-aux in English vs. TL	-	-	6 items	6 items 2 items (tener, querer)
Subject-verb agreement (regular -AR) 1^st / 2^nd / 3^rd singular	4 items	4 items	6 items
Subject-verb agreement (irregular) estar (1st, 2nd, 3rd sing); ser (1st, 2nd, 3rd sing / 3rd pl); tener (1st, 2nd, 3rd sing / 1st, 3rd pl); querer (1st, 2nd, 3rd sing); hacer (1^st, 2^nd, 3^rd sing); dar (1^st, 2^nd, 3^rd sing)	6 items	6 items	4 items (all)
Article & adjective agreement Def / indef; gender; number	4 items	-	4 items	2 items
Adjectival word order Post-nominal	4 items	-	-	2 items
Negation no + verb	4 items	-	-	3 items (-AR)
“hay” vs. “tiene”	4 items	-	-	-

5 of 14

Test creation process

Each question tests a specific grammatical feature (or combination of features)

Size of the test

Target time: 15 minutes (R/L/W); 4 minutes (S)

50 test items (R/L/W); 13 items (S)

Question item pool

Items created using vocabulary from SoW (reviewed to ensure no clash with vocabulary test)

Each pool contains an equal number of instances of each structure taught

(e.g. equally likely to be tested on 1^st person singular as 2^nd person singular in subject-verb

agreement questions, etc.)

Variation between languages in weighting of different modes / modalities, due to variation in nature of grammar features being tested in each language.

In terms of creating the test, we have designed a specific type of question for each grammatical feature that we were including in the test for each language and in the later parts of the presentation we will have a look at some examples of the questions that were included.

In terms of time limit, we had 15 minutes allocated for completing the Reading, Listening, and Writing items, and 4 minutes for the Speaking part.

Pupils will respond to 50 items across the Reading, Listening and Writing sections, and 13 items in total in the Speaking part of the test.

Inevitably there is some variation between the three languages in terms of the weighting of the different modes and modalities, and this is due to the specific grammar features that were tested in each of the languages. As we saw earlier in French, for example, the Listening part of the test is much smaller than it is in German and in Spanish.

In terms of creating the question item pool, for each question type we had a pool of possible items that the pupils could respond to. In the online version of the test, we asked the survey software to select a subset of these items to show to each student (based on the number of items specified for that particular grammar feature, as listed in the coverage tables for each language, which we looked at on the previous slides).

The question items were created using vocabulary from the SoW. We also reviewed all of the items against the vocabulary test to ensure that there was no clashes.

Each pool of items for each grammar structure contains an equal number of instances of each structure that had been taught. For example, the pool of items for the questions targeting subject-verb agreement contained an equal number of items for each of 1^st, 2^nd, and 3^rd person singular forms in German. This means that each student is equally likely to be tested on 1^st person singular as they are on 2^nd or 3^rd person singular, etc.

Over the next few slides, I will show you some examples of the different question types. I’ve picked a selection of examples from across the languages. Where I have provided an example for a grammar feature in one language, and that feature has also been tested in another language, then the same principles were applied.

6 of 14

Test creation: examples items

Testing written and aural receptive knowledge

Testing ability to recognise meaning as well as form

Multiple choice; here matching to English equivalent

Multiple choice options appear in a random order for each item

(Avoid position indicating correct answer)

First, we will have a look at some examples of items that were found in the Reading and Listening parts of the test, so these are testing the students’ receptive knowledge. Pretty much all of the questions in these sections are multiple choice.

With these items we really wanted to ensure that we were testing students understanding of the meaning of the grammar feature as well as their recognition of the form. So, in the example that you can see on the slide here, which is testing knowledge of subject-verb agreement, rather than giving the answer options in the target language, we have deliberately given them in English. This is therefore really testing that the pupils understand, in this case for example that the –ons verb ending refers to the first-person plural ‘we’. If we had given the answer options in French, then the student may recognise the form, for example recognise that nous always appears with the –ons ending, but we can’t be 100% sure that the student definitely knows and understands what the –ons ending means.

It’s useful to note here that students’ knowledge of pronouns is tested in the writing and speaking parts of the grammar test for French and German, were the students have to recall a phrase or a short sentence. Pronoun knowledge are also tested as part of the vocabulary test.

For the instances in French and German, where the verb ending is the same for two ‘persons’ (for example here 1^st and 3^rd person singular regular verb endings are the same in French), these options were combined as one option (e.g. I OR he/she above). So, it is important to acknowledge here that with these questions we are not explicitly testing that pupils know that in this case that the 1^st / 3^rd person take the same ending.

7 of 14

Test creation: examples items

Testing written and aural receptive knowledge

Testing ability to recognise meaning as well as form

Multiple choice; here matching to TL alternative

Isolating recognition of gender and number

8 of 14

Test creation: examples items

Isolating receptive knowledge of syntax

Recognising function (statement / question) indicated by word order

Remove punctuation

(Avoid . or ? indicating answer)

Including variety of ‘subjects’

(Avoid reliance on ‘du’ to indicate question)

9 of 14

Test creation: examples items

Isolating productive knowledge of syntax

Drag-and-drop into correct order

Article / noun / adjective appear in a random (vertical) order for each item.

(Ensures learner is paying attention to each element and its correct position)

10 of 14

Test creation: examples items

Testing written productive knowledge

Testing ability to accurately produce the structure
Open text box

Testing understanding of question formation

(Check pupils understand that ‘do’ auxiliary is not needed)

Isolating grammatical knowledge by providing verb infinitive

(Avoid missing answers due to lack of lexical knowledge)

Here we have examples from Spanish of written productive items.

In general, written productive items required pupils to supply just one element, rather than producing a whole sentence – this was really to make the scoring of the written items much more manageable.

The first example shows an item which is targeting students’ awareness of the fact that Spanish does not require an alternative to the English ‘do-auxiliary’.

You’ll also notice that we have provided a gloss of the verb infinitive for students. This was done in order to ensure that we were isolating grammatical knowledge of these items. By providing the verb infinitive, we are not relying on pupils being able to recall the appropriate lexical item in order to complete the question successfully. They have the correct word in the target language, so the question is really testing whether they can produce the appropriate verb inflection.

11 of 14

Test creation: examples items

Testing written productive knowledge

Testing ability to accurately produce the structure
Open text box

Isolating grammatical knowledge by indicating gender (directly or via article).

(Ensure that pupils are not reliant on recalling gender of a specific lexical item)

12 of 14

Test creation: examples items

Testing written productive knowledge

Testing ability to accurately produce the structure
Open text box

Testing syntax alongside subject-verb agreement.

Half of the pool included verbs in present continuous form

(Check pupils’ understanding that there is one present tense structure for simple and continuous meanings)

13 of 14

Test creation: examples items

Testing oral productive knowledge

Testing ability to accurately produce the structure(s)
Combining a number of structures within each item

Testing subject-verb agreement and question formation.

Isolating grammatical knowledge by providing verb infinitive

(Avoid missing answers due to lack of lexical knowledge)

Note: glosses not provided for other elements of the sentence which are not being tested (e.g. object)

14 of 14

Test creation: examples items

Testing oral productive knowledge

Testing ability to accurately produce the structure(s)
Combining a number of structures within each item

No gloss provided for irregular verbs

(Irregular verb forms taught as individual lexical items, rather than transforming from the infinitive)

Isolating knowledge of gender / number (& case) agreement for articles (& adjectives)

(Noun form provided and gender indicated)

Here we have two more examples from the speaking section which are testing irregular verb agreement, & article / noun agreement.

For irregular verbs, we have not provided a gloss of the verb infinitive. This is because in the SoW, irregular verb forms have been taught as individual lexical items, rather than as a transformation from the infinitive. So, we felt that providing the irregular verb infinitive in these items could actually cause confusion rather than being helpful for the students.

For the items testing article agreement, we provide a gloss of the target noun and the target adjective where appropriate, and we have also indicated the gender, so that we are testing whether pupils can accurately produce the correct article and adjective ending, rather than relying on them being able to recall that particular gender for that noun.