Generative AI�pair or pAIr programming: Friend or Threat ?

Ozan Evkaya

ozan.evkaya@ed.ac.uk / oevkaya@ed.ac.uk

​

University Teacher in Statistics

Financial Times, 28 Nov 2023

The Conversation, 19 Apr 2023

No More Marking Blog,�5 Feb 2023

arXiv, 20 Jul 2023

Overview

• Human-Human (pair) vs Human-AI (pAIr): https://arxiv.org/abs/2306.05153
• Is the AI programming partner comparable to a human pair programmer?
• Are they applicable to the same contexts, can they achieve similar or better performance, and should people interact with them in the same way?

​

• Related Tools to use
• GitHub Copilot example
• You can find some ongoing effort on creating alternatives to GitHub Copilot (such as FauxPilot, OpenCopilot or Tabby)

​

• Pros and Cons
• Thinking loudly

​

​

​

​

Gen-AI as pair programmer?

• Comparison of human-human and human-AI pair programming, exploring their similarities and differences in interaction, measures, benefits, and challenges.

​

• Summary on moderating factors on the success of human-human pair programming, which provides opportunities for 'pAIr' programming research

​

• For human-AI pair programming’s moderators, much was unexplored – we do not know what could make human-AI pair programming more or less effective

Qianou Ma, Tongshuang Wu, Kenneth Koedinger (2023) Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programming, available at https://arxiv.org/abs/2306.05153 

​

Key notes

• They noted that ''No current study sets up a three-way comparison for human-AI, human-human, and human-solo''

​

• ''Additionally, there is not enough research for a comprehensive review, so we cannot reach any conclusion on the effectiveness of human-AI pair programming yet''

​

• ''There also seems to be evidence that women benefit from pair programming more than men do [1, 2]''

​

• ''Researchers also found that less-skilled students learn and enjoy more than more-skilled students in pair programming [3, 4]''

Key notes

• ''How well partners collaborate have been important factors that affect pair programming effectiveness [5, 6], and cooperative behavior and positive interdependence are key to pair programming success [2].''

​

• ''In human-AI pair programming, educators are worried that easily available code-generation tools may lead to cheating, and over-reliance on AI may hinder students learning [7]. However, no study has formally evaluated it''

''In human-AI interaction, given Copilot’s amazing capability to write code in different languages, some have argued that Copilot can take on the role of the “driver” in pair programming, allowing a solo programmer to take on the role of the “navigator” and focus on understanding the code at a higher level [8].''

''In human-AI pair programming, some may argue that the human is solely responsible in the human-AI pair [9], but the accountability of these LLM-based generative AI is still under debate [7].''

Key messages to think about

• ''Existing pAIr studies lack realistic deployment in the workspace or classroom, and a larger sample size would also be desirable. Researchers of both pair programming paradigms use various study designs to examine what affects the effectiveness of pair programming. The literature on human-human pair programming has shown mixed results in many outcome variables, including quality, productivity, satisfaction, learning, and cost. For human-AI pair programming, or mostly human-Copilot in this paper, there are still only few works with incomprehensive measures, but a mixed outcome is also observed.''

​

• ''Collaboration: In terms of collaboration, it is frequently reported that creating smooth collaboration is challenging in both industry [10] and educational context [11, 12]. Given that the free-rider problem reduce pair programming’s effectiveness [11] and regular role-switching potentially alleviates the driver’s cognitive load and ensures balanced learning outcomes [13, 14], it would be interesting to explore if LLM-based AI can be configured to avoid over-help, support role-switching, and how to best support the human-AI pair to collaborate.''
• ''Re-prioritize programming skills: Co-working with AI requires a special skill set, and future work could explore how to support students to better develop these crucial skills. Bird et al. [15] argued that the popularity of LLM-based programming assistants will result in the growing importance of reviewing code as a skill for developers. Nonetheless, in Perscheid et al. [16]’s interview, none of the professional developers remembered training on debugging at school''
• ''Boost students’ self-confidence: Pair programming has been shown to benefit students with lower self-efficacy and self-confidence levels [17] and women [18] more, which could make it a pedagogical tool to engage more vulnerable or underrepresented populations in CS.'' 

​

  ''Nowadays, as an LLM-based agent can support more natural interaction and provide good quality explanations in  the introductory programming context [18], it would be interesting to explore if LLM-based AI could resolve some limitations mentioned in pedagogical and conversational agent works before''

What can we do? Thinking loudly

• Trying to answer available questions or create new ones collaboratively
• Potential new settings [maybe new roles like evaluator, or human-driver vs AI-driver cases]
• Prepare experimental designs for our courses with certain group of students [collaborating with TEMSE]
• Maybe, it is possible to benefit from Gen-AI for some students having certain issues and not able to attend workshop sessions [students with special issues, temporary health barriers who are not able to attend]
• For our courses, which outcome variables are important and how to decide?

​

Ozan Evkaya

ozan.evkaya@ed.ac.uk

​

University Teacher in Statistics

[1] Brian Hanks, Sue Fitzgerald, Renée McCauley, Laurie Murphy, and Carol Zander. 2011. Pair programming in education: a literature review. Comput. Sci. Educ. 21, 2 (June 2011), 135–173. https://doi.org/10.1080/ 08993408.2011.579808

[2] David Preston. 2006. Using collaborative learning research to enhance pair programming pedagogy. SIGITE Newsl. 3, 1 (Jan. 2006), 16–21. https://doi.org/10.1145/1113378.1113381

[3] E A Chaparro, Aybala Yuksel, Pablo Romero, and Sallyann Bryant. 2005. Factors Affecting the Perceived Effectiveness of Pair Programming in Higher Education. Annual Workshop of the Psychology of Programming Interest Group (2005). https://www.semanticscholar. org/paper/c095f0d9b17cd9c2851000534740e7cc087253fa

[4] Phil Maguire, Rebecca Maguire, Philip Hyland, and Patrick Marshall. 2014. Enhancing collaborative learning using pair programming: Who benefits? AISHE-J 6, 2 (June 2014). https://ojs.aishe.org/index.php/ aishe-j/article/view/141

[5] Mustafa Ally, Fiona Darroch, and Mark Toleman. 2005. A framework for understanding the factors influencing pair programming success. In Extreme Programming and Agile Processes in Software Engineering. Springer Berlin Heidelberg, Berlin, Heidelberg, 82–91. https://doi. org/10.1007/11499053_10 

[6] W Sun and G Marakas. 2009. The True Cost of Pair Programming: Development of a Comprehensive Model and Test. Americas Conference on Information Systems (2009). https://www.semanticscholar.org/ paper/647fc48650e4f19962c8a6feb87f3bdedde9dd04

[7] Brett A Becker, Paul Denny, James Finnie-Ansley, Andrew LuxtonReilly, James Prather, and Eddie Antonio Santos. 2023. Programming Is Hard - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 500–506. https://doi.org/10.1145/3545945.3569759

[8] Saki Imai. 2022. Is GitHub Copilot a Substitute for Human Pair programming? An Empirical Study. In 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). ieeexplore.ieee.org, 319–321. https://doi.org/10. 1145/3510454.3522684

​

​

​

​

​

​

​

[9] Advait Sarkar, Andrew D Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with artificial intelligence? (Aug. 2022). arXiv:2208.06213 [cs.HC] http://arxiv.org/abs/2208.06213

[10] Andrew Begel and Nachiappan Nagappan. 2008. Pair programming: what’s in it for me?. In Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement (Kaiserslautern Germany). ACM, New York, NY, USA. https: //doi.org/10.1145/1414004.1414026

[11] Nachiappan Nagappan, Laurie Williams, Miriam Ferzli, Eric Wiebe, Kai Yang, Carol Miller, and Suzanne Balik. 2003. Improving the CS1 experience with pair programming. In Proceedings of the 34th SIGCSE technical symposium on Computer science education (Reno Navada USA). ACM, New York, NY, USA. https://doi.org/10.1145/611892. 612006

[12] Laurie Williams, Eric Wiebe, Kai Yang, Miriam Ferzli, and Carol Miller. 2002. In support of pair programming in the introductory computer science course. Comput. Sci. Educ. 12, 3 (Sept. 2002), 197–212. https: //doi.org/10.1076/csed.12.3.197.8618

[13] Carolina Alves De Lima Salge and Nicholas Berente. 2016. Pair Programming vs. Solo Programming: What Do We Know After 15 Years of Research?. In 2016 49th Hawaii International Conference on System Sciences (HICSS). 5398–5406. https://doi.org/10.1109/HICSS.2016.667

[14] Karthikeyan Umapathy and Albert D Ritzhaupt. 2017. A Meta-Analysis of Pair-Programming in Computer Programming Courses: Implications for Educational Practice. ACM Trans. Comput. Educ. 17, 4 (Aug. 2017), 1–13. https://doi.org/10.1145/2996201

[15] Christian Bird, Denae Ford, Thomas Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowdermilk, and Idan Gazit. 2023. Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools. Queueing Syst. 20, 6 (Jan. 2023), 35–57. https: //doi.org/10.1145/3582083

[16] Michael Perscheid, Benjamin Siegmund, Marcel Taeumel, and Robert Hirschfeld. 2017. Studying the advancement in debugging practice of professional software developers. Software Quality Journal 25, 1 (March 2017), 83–110. https://doi.org/10.1007/s11219-015-9294-2

[17] Lynda Thomas, Mark Ratcliffe, and Ann Robertson. 2003. Code warriors and code-a-phobes: a study in attitude and pair programming. SIGCSE Bull. 35, 1 (Jan. 2003), 363–367. https://doi.org/10.1145/792548. 612007

[18] Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. (April 2023). arXiv:2304.03938 [cs.CY] http://arxiv.org/abs/2304.03938

​

​

​