Input, output, & feedback: a triad approach to L2 pronunciation training in a CALL environment

​

Lillian Jones

liljones@ucdavis.edu

​

AATSP Annual Conference 2022

July 9th, 2022

Valorando nuestras raíces y construyendo nuestro futuro

Valorizando nossas raízes e construindo nosso futuro

Agenda

1. Area of Focus
2. Theoretical Framework & Motivation
3. Previous Scholarship

Present Study

1. Research Questions
2. Methodology
3. Findings
4. Implications for teaching and research
5. Limitations & future directions
6. Conclusions
7. References

2

Computer-assisted pronunciation training

automatic speech recognition

Input, output & feedback

3

Interactionist Framework for SLA

Receiving input, engaging in negotiation for meaning, and producing output, which includes (corrective) feedback, during meaning-focused interaction facilitates second language (L2) development ( Ziegler, Parlak & Phung, 2022)

4

Receive Input

Produce Output

Receive and negotiate feedback

Sociocultural Framework for SLA

5

• Language as a social phenomenon and L2 learning is a goal situated within social interaction (Mitchell, Myles, & Marsden, 2013)
• “Contextual factors such as time, place, and mediating technologies that are relevant for communication” (Chapelle, 2009)

Humans (learners) interacting

with computers

Humans interacting with themselves

• output as input to produce scaffolding, (Lantolf, 1994; Lantolf & Thorne, 2007)

Three language skill components of this study

👂 Listening Comprehension

Listening comprehension is a collaborative activity, involving receiving what the speaker has said, understanding what is relevant to the listener, empathizing with the speaker’s motivation (Brandl, 2006)

​

​

🗣 Pronunciation Training

Segmental and suprasegmental features and how they affect the intelligibility, comprehensibility and accentedness of one’s speech (Derwing & Munro, 2014)

​

🎯 Explicit/”Targeted” Feedback

“Intervention where the learner is provided with information about their utterances; it is specific, evidence-based,and actionable in respect to the L2 targeted production to further pronunciation development” (Bajorek, 2017, p. 28)

6

Target Language Input

• Language learners need comprehensible Target Language (TL) input to develop communicative competence in the TL (Barcroft & Wong, 2013; Ellis, 2005)
• Need an abundance of aural input to develop listening comprehension skills (Brandl, 2008)
• Learners need input that is abundant, varied and authentic (Bueno-Alastuey & Gómez-Lacabex, 2022) to acting as target models in order to develop L2 skills such as pronunciation (Neri et al. 2001; Saito & Hanzawa, 2017)
• However,
• There is often a lack of target language input (exposure) in early language classrooms (Ellis, 2005)
• Sufficient comprehensible input is not enough for pronunciation skills to develop, explicit practice is also necessary (Bajorek, 2017; Neri et al. 2001)

​

7

Pronunciation Practice & Targeted Feedback

• L2 classroom often neglects explicit pronunciation training and feedback (Olson, 2014) and may fail to facilitate an ideal setting for speaking skills to develop (Brandl, 2008), specifically fluency and pronunciation, due to a paucity of exposure to quantity and quality of target language input (Ellis, 2005)�
• Receiving timely, explicit, and individual feedback is essential in developing L2 skills (Ellis, 2006), especially regarding pronunciation (Saito & Lyster 2012)�
• Corrective feedback for adult L2 learners allows them to notice the discrepancies between their output and the L2, which mere exposure doesn’t necessarily accomplish (Neri et al. 2001)�

​

8

Computer Assisted Pronunciation Training (CAPT) & Automatic Speech Recognition (ASR)

• Can provide large amounts of meaningful and varied (often unlimited) input, as both sound files of individual sounds and words in online dictionaries; increased time on speaking practice (output) and individualized attention and/or feedback; (Bueno-Alastuey & Gómez-Lacabez, 2022)
• targeted, timely & explicit feedback (Neri et al., 2001)
• offer learners opportunities to notice gaps and correct errors (Chapelle, 2009)
• visual feedback (Golonka, 2014)
• Speech recognition tools can promote L2 pronunciation skills by giving “learners individualized, granular feedback in real time on the comprehensibility of their spoken language” (Nielson, 2022, p. 235)

�

​

9

Research Questions

• RQ1.
• How does native speaker input affect the accuracy of L2 Spanish pronunciation in a short-term immediate context?�
• RQ2.
• What is the learner’s perception of the explicit NS input and its role in the learner’s oral production?
• What is the learner’s perception on the low-stakes CALL task to practice pronunciation with explicit input and targeted feedback?

10

Hypothesis

Using audio recordings of native speakers to provide authentic native-speaker (NS) input prior to a pronunciation training activity will support second language (L2) pronunciation gains more than those students who do not have access to the NS audio recordings.

11

Methodology - participants

• 203 Beginning University Spanish language learners
• Hybrid course, 2 days in person and 3 days activities on Canvas (interactive quizzing, exploratory tasks, writing, listening comprehension, etc.)
• Divided into two groups Experimental & Control Group
• The control groups can be subdivided into two subcategories, in-person (n=45) and online (n=42). The experimental group is in-person only (n=116).

12

Methodology

Treatment

• Completed 9 weekly pronunciation practice activities
• (E group only) Listened to recordings of native speakers (NS) saying target vocabulary words and phrases �
• Read and listened to a longer contextualized text
• Read and spoke into ASR platform (iSpraak)
• Received visual feedback �

Feedback

• Received an accuracy score, isolated target words and a transcription of what they said

13

Both groups

Pronunciation training task

14

1

2

3

Results (Data collection & methodology)

Accuracy

Numerical score (0-100%)

Targeted Feedback Words (for improvement) Any mispronounced words in isolation

​

Testimonials

Opinions and feedback about the exercise from the learners

15

Average score (by group across the quarter (over modules))

​

​

​

16

Any participants with 5+ missing assignments removed

Score distribution

17

Experiment versus Control (all)

Experiment versus Control (both sub groups)

Score distribution

18

What does a low mean mean?

The average feedback score % waivers between 99% and 95%. However, when the mean is calculated it includes a 0%, which indicates a missing assignment, not an extremely low score on the activity. So, when we view outliers of means:

1. a high mean indicates that most (or all) the weeks’ assignments were completed and with a 95%+
2. a low mean is indicative not necessarily of low feedback scores, but rather not turning in the assignment

19

• Participant #117
• Control in person

20

PARTICIPANT PROFILE

Mean score: 99.125 (M1:M9)�M1: 100, M9: 100

M3

“The iSpraak activities help me practice my pronunciation; they're helpful because I can read it as many times as I want.”

M4

“The ISpraak activities are helpful with pronunciation.”

M5

“The ISpraak activities are not very helpful because I don't get any feedback.”

99%

M7

“The iSpraak activity because I can't get feedback on how to properly pronounce the words.”

100%

98%

98%

• Participant #151
• Control online

21

PARTICIPANT PROFILE

Mean score: 60.87 (M1:M5; øM7-9)�M1: 99, M9: 0

M1

“The pronunciation online activity gave me a lot of trouble and I spent a lot of my time trying to correct little things my microphone had difficulties picking up. As a result, I had to shout each syllable which was not very coherent if I was just talking.”

M3

“The pronunciation activity got me again. Felt like I was screaming into my mic trying to over exaggerate my pronunciation because the computer wasn't processing what I was saying or it would show up correct then change before showing me the final sentence.”

96%

98%

Control in person P140 • Mean: 98.75 (M1-M9)

• M3: “The pronunciation activities would be more helpful to do in person in class, because they technology does not grasp all the words we speak due to mic issues.” (98%)

• M7: “pronunciation excersizes least useful, because the mic can't depict all words said. So even if prounounced correctly, you could lose points because the mic can't hear all things said clearly.” (99%)

• M8: “pronunciation, because the technology behind it doesn't work” (99%)

​

22

​

Control online P178 • Mean: 53 (M1-M5)

​

• M3: “I like the iSpraak assignments. They're kind of fun and good at making me enunciate my words better. It's good speaking practice.” (76%)�

• M4: “I like the iSpraak activities, it helps me practice pronunciation and actually speaking Spanish out loud when I'm doing assignments. I probably wouldn't otherwise.” (84%)

​

​

Experimental P7 • Mean: 87.7 (M1-M9)

​

• M1: “The activity that I found the most useful this week was the activity from Wednesday were we went over the pronunciations of the vocabulary. The ¡Spraak! activity gave me time to practice actual reading and speaking at the same time. I figure that the repetition is what I find the most useful in these activities, as it embedded the vocabulary in my mind.”.” (100%)�

• M7: “The activity that I found the least useful was the pronunciation activity because it was cut off at the end. I find it disorienting when words are cut off because it makes me question whether or not the word is real. However, the rest of the activity was fine and helpful.”” (97%)

​

​

Experimental P77 • Mean: 87.87 (M1-M4, M9)

​

• M4: “the ispraak pronunciation because the website was not working and blocked my sound off halfway through speaking.”” (61%)

​

Missing assignments?

23

No filtering!

Missing assignments?

24

No filtering!

M7 M8 M9

No data collected

Why are there so many missing assignments as the quarter progresses?

25

1. Cross referenced all of the participants from previous slide (missing any assignments between M7, M8 & M9) with submitted testimonials

2. Looked for testimonials with potential reasons as to why they did not submit assignments

​

Still going through the data, but two main themes emerged:

1. issues or perceived issues with technology
2. Assignment not challenging / perceived pre-existing high skill/familiarity with pronunciation

• “I think the ispraak is a little bit slow in recognizing my voice. I spend many hours reading the passage but I still did not get high score. Also, sometimes the isprrak website cannot record my voice completely. For example, i read the whole passage, isprrak can only record part of it
• “iSpraak Speaking activities because I don't see the learning in just reading off the screen.”
• “iSpraak activity because I'm just blindly reading off the screen.
• “The pronunciation online activity gave me a lot of trouble and I spent a lot of my time trying to correct little things my microphone had difficulties picking up. As a result, I had to shout each syllable which was not very coherent if I was just talking.”

• “[least useful] Potentially the pronunciation module because I am very comfortable with it.”
• “I found the pronunciation least useful because I am comfortable with my pronunciation already.”
• “Pronunciation comes easily to me to the iSpraak assignments are always simple!”
• “ISpraak because I feel comfortable with pronunciation.
• “Ispraak helped a lot with my pronunciation even though it was a bit difficult to use sometimes”
• “The iSpraak is useful for pronunciation but difficult and frustrating to use because it will stop recording me in the middle of my speech.”

​

“Problematic words”

26

cebolla, modernas, zazu, recuerdos, allí, silvia, culinaria, tren, cinco, once

Top 20 Most frequent problematic words by group

*zazu - foreign word; not commonly registered for Spanish in ASR tool

Targeted phonemes

1. Soft /c/ - cinco, cebolla, once
2. Hard /c/ - recuerdos, culinaria,
1. ‘cu’ -> recuerdos
3. /ll/ - allí

Gives further evidence-supported insight into what we integrate into early Spanish language learner’s pronunciation practice

Testimonial analysis

29

What did the learner’s think? (RQ2)

1. Downloaded all of the weekly “self assessments” of all the six class sections in the study �
2. Modified version of Grounded Theory to observe emergent themes, which made themselves visible from pre existing survey questions (Glaser & Strauss, 2009) �
3. Searched for any comments about “pronunciation, “*audio”, “ispraak” (typically located in open-ended questions like, “Most useful activity?”, “Least useful activity?” and last question)
1. Is there anything you could do to do better in this course?
2. Is there anything your instructor could do to help you do better in this course?
3. Do you have any additional comments or questions for your instructor? (about the course, class content, etc). �
4. Organized these topics into categories of “most useful”, “least useful’, “technology”, and “general”

* E treatment (audio dictionaries)

Testimonial analysis

30

• 203 participants
• 45 self-assessments (9 weeks, 6 class sections/groups)
• Many of these numbers come from the same participant, just different module
• Only including search criteria for “audio”, “ispraak”, “pronunciation”

most useful

31

“I found the audio recordings the most useful because I was able to hear Spanish from people with different accents. I found it especially helpful with the alphabet because some of the letters were confusing but hearing them said helped. I also liked being able to listen to it as many times as I needed.”

“I found this [audio recordings] to be the most useful because I was initially pronouncing a majority of the words wrong and it was only once I listened to the audio did I realize the correct pronunciation. It was also helpful since I could replay it multiple times and compare my pronunciation with the speakers in order to say it correctly.”

“I liked them all but I really like iSpraak because it lets you practice speaking in the comfort of your own home. I think it's important when learning to have a space you feel comfortable to practice because sometimes doing it in class brings a bit of anxiety.”

“Listening to those audios are most useful because it helps me to know what I need to practice more and at least get some familiars with the speaking speed.”

least useful

32

“Listening to pronunciation of the vocabulary because it just feels like busy work.”

“Listening to the audio clips of the vocabulary because I know how to pronounce most of those already.”

“I found iSpraak to be the least useful because it didn't have much to do with the actual leaning [learning] course we had planned this week.”

“iSpraak activity because I'm just blindly reading off the screen.”

technology

33

“having to upload audio files is very hard for me”

“The audio listening was not useful to me this week because the files had a hard time loading compared to previous weeks. / It would be helpful to get the audio files working. “

“...but I think the AI was a little wonky and kept misunderstanding what I was saying”

“I couldn’t figure out the pronunciation activity but that was because my iPad was not updated”

“The pronunciation activity because it did not give me accurate feedback.”

“ispraak because it is inaccurate”

general

34

“I really enjoyed the speaking activity that we turned in online. It helped me slow down my spanish speech and made me less nervous to speak in spanish in class.”

“The online pronunciation is great as it helped me practice my speaking.”

More practice with pronunciation; pronunciation; need to practice

“I still find the audio assignments really hard, but I don't know how to deal with it. Hope I can improve my listening skill in the future.”

“Pronunciation is one of my top priorities and I would appreciate feedback to help fine-tune it.”

• Participant, E group FQ 21

35

Emergent themes

1. Early Spanish language learners (LL) are aware of the need for pronunciation for both listening comprehension and oral production

4. Learners with previous exposure to target language (TL) input perceive pronunciation activities as unnecessary

2. Early Spanish LL ask for explicit pronunciation practice

​

3. Learners need to be trained and guided on what to expect from technology and how to most effectively use it

36

*note - few comments about the audio dictionaries (Exp.)

Implications for research & teaching

• Provides insight into problematic pronunciation issues (i.e. specific phonemes, isolated words or phrases)
• Inform foreign language classroom practices, targeted pronunciation practice, especially for early language learners, individuals & groups of learners�
• (Learners) notice linguistic errors (gaps), which can help support their skill development �
• Insight into learner-machine interaction
• Chatbot & AI development
• Learners testing intelligibility of their L2 output with a “native speaker” (NS) machine (ASR is based on NS input)

37

Implications for research & teaching

• User perceptions of computer-assisted pronunciation training (CAPT) technology tool�
• Reminder to instructors and curriculum designers to leverage already existing tools, such as the LMS first (i.e. Canvas ) �
• inform assignment development: missing assignments and comments�
• Some students like the pronunciation practice/CAPT tool consistently, and other consistently do not�

​

​

38

Limitations

• No pre & post assessment (although in a sense if you measured just M1 & M9 you could glean this information) �
• Lack of access to learner’s demographic information
• For example, no controlling for language background and did not remove native and/or heritage speakers �
• Technological limitations of ASR technology with texts that are too long (i.e. M7 cuts off “Por último, tiene que enfriar y servir con fruta de temporada.) (this has since been updated, and will go into effect the next time this course is held)�
• Scores are all pretty similar (high), small sample size, makes it difficult to see major trends; good that they didn’t perform worse

39

Future directions

• Few comments about audio dictionaries; modify study and assignments so students notice the difference (individually)
• Survey participants to gain insight into Learner Variables (Task Difficulty, Robinson, 2001): aptitude, motivation, anxiety, intelligence, etc. to make comparisons across results
• Making a training guide/video for students to have a better understanding of how iSpraak works, the pedagogical theory behind it, as well as understanding and accounting for the limitations of technology
• For example to answer why they don’t get feedback, address technology “not working” concerns, best practices, what to expect, etc.
• Analyze students’ comments for mentions about “feedback”

40

Conclusions

• Response to hypothesis: there does not appear to be any statistically significant trends between the control and the experimental group (audio dictionaries)�
• Through positioning listening comprehension, pronunciation practice, and explicit feedback in an immediate setting in a computer-assisted language learning (CALL), low-stakes environment, this can provide early language learners an opportunity to develop L2 pronunciation skills, earlier than that might normally�
• Language learners are aware of the necessity of listening and speaking skills from their initial learning stage
• Noticing and metalinguistic awareness is key for L2 development (Ellis, 2009; Hulstijn, 2011)

41

Thank you!

• The course coordinator: Claudia Sánchez-Gutiérrez
• The course team teachers: Paloma Fernandez-Mira, Leigh Houck, Ana Ortega Perez
• Academic Technology Services team at the university: Margaret Merrill & Joey Van Buskirk
• Joel Rodríguez-Medina (data analysis and visualization)

42

Thanks!

Any questions?

You can find me at:

• liljones@ucdavis.edu
• lilliancjones.com
• 🐦 @Lily_Jones20

43

Lillian Jones

PhD Candidate | Associate Instructor

Department of Spanish and Portuguese

References • 1 of 2

44

Bajorek, J.P. L2 Pronunciation in CALL: The Unrealized Potential of Rosetta Stone, Duolingo, Babbel, and Mango Languages. Issues and Trends in � Educational Technology, 5 (1). 24-51.

Barcroft, J. & Wong, W. (2013). Input, input processing and focus on form. In J. Herschensohn & M. Young-Scholten (Eds.), The Cambridge � Handbook of Second Language Acquisition (pp. 627-647).

Blake, R. (2000). Computer mediated communication: a window on L2 Spanish interlanguage. Language Learning & Technology, 4(1), 111-125.

Brandl, K. (2008). Making Communicative Language Teaching Happen. Pearson-Prentice Hall.

Bueno-Alastuey, Ma. & Gómez-Lacabex, E.. (2022). Technology and Pronunciation. In SLA. In N. Ziegler & M. González-Lloret (Eds.), The Routledge � Handbook of Second Language Acquisition and Technology. Routledge.

Derwing, T. M., & Munro, M. J. (2005). Second language accent and pronunciation teaching: A research-based approach. TESOL Quarterly, 39(3), � 379–397.

Ecke, P. (2004). Language attrition and theories of forgetting: A cross-disciplinary review. International Journal of Bilingualism. 8(3), 321-354.

Ellis, R., Loewen, S. & Erlam, R. (2006). Implicit and explicit corrective feedback and the acquisition of L2 grammar. Studies in Second Language � Acquisition, 28(2), 339-368.

Ellis, R. (2009). Task-based language teaching: sorting out the misunderstandings. International Journal of Applied Linguistics, 19(3), 221- 246.

Ellis, R. (2009). Implicit and explicit learning, knowledge and instruction. In R. Ellis (Ed.), Implicit and explicit knowledge in second language learning, � testing and teaching (pp. 1-25). Bristol, UK: Multilingual Matters.

Foote, J., Trofimovicha, P., Collinsa, L. & Soler Urzúab, F. (2016). Pronunciation teaching practices in communicative second language classes. The � Language Learning Journal, 44(2), 181-196.

García, C., Nickolai, D., & Jones, L. (2020). Traditional Versus ASR-Based Pronunciation Instruction: An Empirical Study. CALICO Journal, 37(3), � 213-232. DOI: https://doi.org/10.1558/cj.40379

Glaser, B. G., & Strauss, A. L. (2009). The discovery of grounded theory: Strategies for qualitative research (9th ed.). Piscataway, NJ: Transaction.

Golonka, E. M., Bowles, A. R., Frank, V. M., Richardson, D. L., & Freynik, S. (2014). Technologies for foreign language learning: A review of � technology types and their effectiveness. Computer Assisted Language Learning, 27(1), 70–105.

Hulstijn, J. H. (2011). Language proficiency in native and nonnative speakers: an agenda for research and suggestions for second-language � assessment. Language Assessment Quarterly, 8(3), 229-249.

Lantolf, J. (1994). Sociocultural theory and second language learning [Special issue]. Modern Language Journal, 78(4).

IRB Reference #: 1851269-1

References • 2 of 2

45

Lantolf, J., & Pavlenko, A. (1995). Sociocultural theory and second language acquisition. Annual Review of Applied Linguistics, 15, 38–53.

Lantolf, J. P., & Thorne, S.L. (2007). Sociocultural theory and second language learning. In Bill B. Van Patten & W. Jessica (Eds.), Theories in second � language acquisition: An introduction (pp. 693-701). New York, NY: Routledge.

Larsen-Freeman, D. (2006). The emergence of complexity, fluency, and accuracy in the oral and written production of five Chinese learners of English. Applied � Linguistics 27.4, 590–619.

Mitchell, R., Myles, F. & Marsden, E. (2013). Second Language Learning Theories. Routledge.

Morris & Blake. (2020). Technology and Oral Communication

Robinson, P. (2001). Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics, 22(1), � 27-57.

Skehan, P. (2003). Focus on Form, Tasks, and Technology. Computer Assisted Language Learning, 16(5), 391-411.

Levis, J. (2007). Computer technology in teaching and researching pronunciation. Annual Review of Applied Linguistics, 27, 184-202.

Lord, G. (2019). Incorporating technology into the teaching of Spanish pronunciation. In R. Rao (Ed.), Key issues in the teaching of Spanish pronunciation: From � description to pedagogy (218–236). New York, NY: Routledge.

Neri, A., Cucchiarini, C. & Strik, H. (2001). Effective feedback on L2 pronunciation in ASR-based CALL. Proceedings of the workshop on Computer Assisted � Language Learning, Artificial Intelligence in Education Conference, pp. 40-48. https://hdl.handle.net/2066/76203

Payne, S. & Whitney, P. (2002). Developing L2 Oral Proficiency through Synchronous CMC: Output, Working Memory, and Interlanguage Development. CALICO � Journal, 20 (1), 7-32.

Ruan, S., Jiang, L., Xu, Q., Liu, Z., Davis, G. M., Brunskill, E., & Landay, J. A. (2021, April). EnglishBot: An AI-Powered Conversational System for Second � Language Learning. In 26th International Conference on Intelligent User Interfaces (pp. 434-444).

Saito, K. & Hanzawa, K. (2017). The Role of Input in Second Language Oral Ability Development in Foreign Language Classrooms: A Longitudinal Study. � Language Teaching Research, 22(4), 398-417.

Saito, K., & Lyster, R. (2012). Effects of form-focused instruction and corrective feedback on L2 pronunciation development of /ɹ/ by Japanese learners of � English. Language Learning, 62(2), 595–633. https://doi.org/10.1111/j.1467-9922.2011.00639.x

Skehan, P. (2009). Modelling Second Language Performance: Integrating Complexity, Accuracy, Fluency, and Lexis. Applied Linguistics, 30(4), 510-532.

Skehan, P. (2003). Task-Based Instruction. Language Teaching, 36, 1-14.

Thorne, S. & Smith, B. (2011). Second Language Development Theories and Technology-mediated Language Learning. CALICO Journal, 28(2), 268-277.

Wright, C. & Tavakoli, P. (2016) New directions and developments in defining, analyzing and measuring L2 speech fluency. International Review of Applied � Linguistics in Language Teaching, 54 (2), 73-77.

Ziegler, N., Parlak, O., & Phung, H. (2022). Interactionist Perspectives and the Role of Computer-Mediated Communication in SLA. In N. Zieger & M. � González-Lloret (Eds.), The Routledge Handbook of Second Language Acquisition and Technology. Routledge.