LAK17 Writing Analytics Workshop • Participant Working Notes

Public workshop website • These notes: http://bit.ly/lak17wanotes 

AM: Mapping intervention potentials

What are the primary obstacles to good writing? (bullet as many as you think of)

Thinking Like a Student - problems

Supporting students overcome writing problems


Mapping intervention potentials

Thinking like a student:

What is the primary obstacle to good writing?

Thinking like a researcher/educator/ academic:

What pedagogic strategies support learning to write/tackle this obstacle?

Thinking like a technologist:

How can analytics augment this strategy?

A lack of knowledge on development of ideas and the strategies/tools for taking rough, fuzzy, ill-formed ideas and developing them over multiple iterations towards a polished product.

Collaboration - peer review, editor/ non-editor. Competencies for participating in peer review process.

Making the process visible - emerges over time towards a polished product (which may be always different). Organic growth rather than step by step.

Different types of support for different periods in an iterative process.

Allowing time for the process.

Brainstorming tools, ideas mapping - Capturing the messy, fuzzy ideas. Free write - take the topics and prompt for development. Remove the threat of writing being read - just between the student and the computer for initial free write - analytics helps with formulation of initial ideas for first sharable draft.

Scroll through an exemplar over time.

Director’s commentary over the exemplar edits.

Model mental models - make the thinking of the (good) writer visible.

??

In-process formative feedback and metacognitive scaffolding

Automated feedback; intelligent agents

Unfamiliarity with the domain

Timely feedback; find ways to ensure that students are focusing on conceptual cohesion

Different priorities among faculty as to the quality of a piece of writing -- two people may grade differently / may evaluate arguments differently

Can create rubrics to create a common ground between the different point of views on argumentation

Benchmarking / calibrated peer review tasks

Detect change between drafts

Create graphs to identify “binding terms” / terms that appear to be central to a concept or group of concepts

Identifying the right tools for measurement - making sure we’re measuring what we want to measure

Analytics that consider both the “what” and the “when”

Interpreting data is a challenge - use data that makes sense, not what is most convenient to collect

Design analytics that have theoretical backing in learning science.

Analytics should provide guidance for the next step without overwhelming the learner.

Writer’s block. Not knowing what is expected or the style for a specific genre. Weak at building the argumentative structure of the text.

Prepare checklists/How-Tos/scaffolds for students. Hands-on training with existing articles. Peer-review (on already published articles) as class/course exercise. Reflection on the feedback comments.

Writing tools plugins, fuzzy analytics of correspondence between different rhetorical elements of a document (claim matching results, contributions appearing in abstract, main body, conclusion).

Make use of peer review

Analytics can facilitate peer review

Know all of the best words (stylistic vocabulary)

  • Active reading focusing on stylistic features (‘identify sentences that fulfil this function’)
  • Asking students to explicitly diversify/domain-ify their language use/to engage in read and replace strategies
  • Phrase bank - possibly automated replacement/suggestion of phrases against known overused phrases (BUT does it transfer?
  • N-gram based wordclouds (,or similar)

Know all of the best words (domain vocabulary)

  • Active reading strategies focusing on domain features        
  • Asking students to explicitly diversify/domain-ify their language use/to engage in read and replace strategies
  • Thesaurus tools

Appropriate depth and style (short v long sentences, over v under elaboration)

The structure of a particular genre of text and cultural differences therein

  • Ask students to take a text and to restructure it
  • Provide a scaffold/template (possibly alongside an exemplar)
  • Ask students to identify which section a paragraph is on
  • Ask students to select the ‘best’ exemplar conclusion/intro from a set of examples
  • Copy the structure and reimplement it using your own information

Questions:

What is an “analytic”?  -- Anything you can count

Tool overviews and sites of use

Tool

How can these tools be deployed in practical pedagogic contexts?

What knowledge is needed to do that successfully?

A3R Project using AWA - Gibson

http://utscic.edu.au/tools/awa 

Provides near instant reflective writing feedback on submission (i.e. not during writing)

Best used in contexts where reflection explicitly taught, and really embedded in teaching.

Baseline reflective writing knowledge

Ability to take action given lack of highlighting (i.e., no reflective moves identified)

Bolo interactive writing analytics

Content knowledge/background knowledge gap. Large number of articles i.e. ‘searching for and filtering information sources’ 

Provides real-time keyterm suggestions to recommend new resources to learners, based on instructor list of terms + associated resources.

Has interface to show paragraphs that relate to particular keyterms identified.

Instructor must have key-terms and associated articles

How to integrate the new info into paper

Domain knowledge

?

Elena Cotos (Iowa State University) – Research Writing Tutor (RWT) – A corpus-based platform for data-driven writing (webinar discussing RWT)

  • Instructional module - describes moves/steps with video-lectures about research articles
  • Explore published writing module - exposes students to published articles providing exemplars of each communicative goal/move within published literature
  • Concordancer - shows examples of any particular move
  • Feedback module - Shows how similar to a published paper (in the discipline/genre) a submitted paper is - giving ‘over’ ‘under’ or ‘goal’ ratings re: move presence in sections, and particular steps within those sections.  Also highlights the moves on the text itself

Data-driven learning approach in writing instruction

Students lack genre awareness

Understanding of genre conventions (communicative moves and rhetorical strategies) in disciplinary writing

?

M-write David Harlan, Tim Kay, et al.,

http://ai.umich.edu/portfolio/m-write/

Anne Gere, Ginger Shultz, Chris Teplovs, Dave Harlan (University of Michigan) – M-Write: A Large-Scale Laboratory for Writing Analytics Research (pdf)

Actionable feedback based on large corpora of student texts.

  1. Faculty identify key concepts
  2. Develop prompts/rubrics
  3. Students draft responses
  4. Automated peer-review, feedback, and revision of texts

How do we use the corpus to provide diagnostics to instructors and students?

?

Christian Rapp, Otto Kruse, Madalina Chitez, Jakob Ott. (Zurich University of Applied Sciences) – Thesis Writer (TW) – an Intelligent Tutoring System for Writing  Instruction and its Study (Thesis Writer abstract (pdf); and webinar discussion of the tool)

www.thesiswriter.eu available in German and English.

I would like to draw your attention to that conference http://eataw.eu/conferences.html where we will have a 3 (!) hour symposium on tools supporting writing. Most of the European tools will be represented.
European Literacy Network. Working Group three looks at tools:

http://www.is1401eln.eu/en/working-groups/working-group-3/ ELN.

“Proposal wizard”: Structured writing template for writing a proposal with brief guidance at each stage - designed to get it written fast first.

“Thesis editor”: Then more feedback on submission - Phrase book, examples, and tutorial, and a corpus tool/concordancer for keywords.

Gives holistic overview, and allows collaboration

User guide, overview intro to text genre

Bahar Sateli and Rene Witte (Concordia University) – Personal Research Assistants for Young Researchers (pdf).

http://www.semanticsoftware.info/from-papers-to-triples 

How do students and academics make sense of vast amounts of literature?

System to deal with corpus of (computer science) articles.

Underlying techniques:

  • Semantic representation of scientific articles (using semantic web technologies, RDF, etc.)
  • Rhetorical entity extraction with text mining.
  • Scholarly user profiling (what a researcher knows already? what is needed for the task at hand?)

The presented research is more an “automated workflow” than a readily available tool.
Required knowledge to integrate the workflow in a tool:

  • Semantic web techniques (RDF, SPARQL)
  • Rhetorical elements fundamentals
  • GATE framework

Noureddine Elouazizi, et al. (University of British Columbia) – A Formal Semantics-informed Learning Analytics Technology for Analyzing Written Argumentation

How do students learn to argue in the genre?

Give instructors feedback on a corpus of student essays - instructors get a dashboard showing how students are doing on each

Give students feedback on their own data

Given gauges/visual traffic lights on quantity of particular things within a text

How do we encourage students to draft?  (limit numbers of drafts to submit, but give feedback per-graph)

What does good argument look like?

What do argumentative moves look like?

What’s the right argument (e.g. some issues there’s a correct(er) side to be on!)

What’s the ‘correct amount’ of a particular move?

Danielle McNamara and Laura Allen (Arizona State University) – The Writing Pal: A Writing Strategy Intelligent Tutoring System (ITS) (SoLET Lab)

Persuasive writing (not source based) designed to support basic argumentative writing (structure of a text, notion of a ‘thesis’, etc.).

Instructors can direct students to particular tasks

Feedback doesn’t focus on errors or line-by-line feedback but the strategy they need to focus on

Blended activities workbook to support modeling

Lower engagement if tools are seen as ‘separate’ online only

Preferably, some understanding of the strategies

Gaps in how these existing tools address the obstacles earlier?

Group focus - Designing resources for practitioner-oriented tutorials for the writing problem your tools address

Group 1 - Thesis Writer & Bolo

Group 2 - Personal Research Assistants & RWT

http://www.d3ai.iastate.edu/conf_wkshop/ddsi/04_o_connor_ddsi_20160128.pdf 

Group 3 - Mwrite & WPal

Materials to support the skill of writing vs quality of the content

Group 4 - AWA (Gibson) and ARGANA (Noureddine)

Is a tool trying to improve academic thinking, or the student’s understanding of a particular genre of academic writing? (Or both?)

Both tools focus on the latter only (the argument or reflection might be weak/nonsensical) in order to help the student reflect on whether they have communicated their ideas in an appropriate way. Help them reflect on “Did you really mean to say this?” (because that’s how it will be read in this genre...)

In coming to understand the academic moves that define a genre, this may in turn help them improve their thinking (in the way that any good visualisation/symbol system can ‘talk back’ to the user). But the student will have to learn how to “read” the analytics feedback (as with any visualisation/symbol system)

Formative feedback tools like these cannot (or aren’t designed to) grade the overall quality. The tools don’t have a model of “good” per se, but rather, are built based on detecting the hallmarks of a genre, and based on confidence that “giving this kind of feedback is generally helpful” — beyond the NLP, feedback design is a critical step.

High performing writers will invariably break any set of genre rules — but hopefully they understand that they’ve done so.

The importance of framing the tool for students:

Briefings are important so the students don’t come to the tool cold. But we can’t depend on students attending/remembering such briefings: the tools themselves do also need to convey how they should be used.

Writing tasks will fundamentally shape student use of the tool

What is “good”?

There are NLP models of writing (eg rhetorical parser), and models of how the learner will engage with these models will shape the UI

Sculpture metaphor - you work the stone until the inner figure is revealed “clearly enough” — requires a lot of revision

PM: Thinking like an educator

Educator perspective - from writing tasks to analytics augmented writing support

Reflection

Argumentation

Abstract

What learning problem/task would produce these?

What interventions would support writing in context of that learning?

What tools or data would give insight for those interventions?

What pre-requisite knowledge is required to engage with that tool/data

What would a user (student, teacher, other?) actually do with the tool/data?

Notes on keynote talk

Feel free to share thoughts here

Graham, S., & Perin, D. (2007). Writing next: Effective strategies to improve writing of adolescents in middle and high schools –A report to Carnegie Corporation of NewYork. Washington,DC:Alliance for Excellent Education.

https://www.carnegie.org/media/filer_public/3c/f5/3cf58727-34f4-4140-a014-723a00ac56f7/ccny_report_2007_writing.pdf 

Evidence Centred Design https://www.ets.org/Media/Research/pdf/RR-03-16.pdf 

Notes on next steps (closing session)

Where do we go next?...

Miscellaneous notes

Want to make a note of something? Nowhere else to put it? Here’s the place to put it…

Why don’t these tools have greater uptake?

Trust

Assessing impact

Ethics of deploying experimental tools

Participant Biographies

Please put any relevant information below, we’ll convert this to a ‘view only’ doc and link from the workshop website after the day.

Simon Knight - University of Technology Sydney

I’m one of the workshop organisers, and a lecturer at the University of Technology Sydney. I am primarily interested in student epistemic cognition (their understanding of where knowledge comes from, how it is justified, and how claims are inter-related). One means through which to investigate epistemic cognition is via writing practices and the written products that students produce in developing evidence-based texts.  I’m interested in open education, and development of resources to build capacity in using evidence and high quality assessment in education.

Simon Buckingham Shum - University of Technology Sydney

Workshop organiser, Professor of Learning Informatics at the University of Technology Sydney, and Director of the Connected Intelligence Centre. I’m interested in analytics for building the critical competencies that students will need for the future of work, and citizenship. In this context, analytical and reflective writing are framed as windows onto the mind of a  learner’s capacity for critical thinking and deep reflection. My HCI background and work on visualizing argumentation have shaped how I think about the user experience, and the role that visual feedback can play in shaping cognition.

Andrew Gibson - University of Technology Sydney

Workshop organiser, Research Fellow in Writing Analytics with the Connected Intelligence Centre. I’m particularly interested in Reflective Writing Analytics, and in student focused Learning Analytics.  More generally I have an interest in the ways we bring together computational techniques and human problems.

Danielle McNamara - Arizona State University - Professor in Psychology and the Institute for the Science of Teaching and Learning. Conduct research in writing, comprehension, learning, and language. Developed the Writing Pal, Coh-Metrix, iSTART, and other natural language processing tools.  Examine language in multiple contexts.  

Laura Allen - Arizona State University

Workshop organizer, PhD student in Cognitive Science at Arizona State University. I am interesting in examining the cognitive and affective processes involved in language comprehension, writing, and knowledge acquisition, and to apply that understanding to educational practice by developing and testing educational technologies.

Eric Cooper - Intel

My background is 30+ years in Learning Science, AI, Intelligent Tutoring Systems, Computer Science - research and product. My role at Intel is largely “thought leadership” and “ecosystem development” - so I spend a lot of time learning about emerging technology trends and analyzing how they might impact education K-20 globally. LinkedIn profile.

Bahar Sateli - Semantic Software Lab, Concordia University, Montréal, Canada

I am a PhD Candidate (ABT) at the Department of Computer Science and Software Engineering, Concordia University (Montréal). My research focuses on applications of semantic technologies, in particular text mining and semantic web, in knowledge-intensive domains. My PhD topic is Semantic Publishing, which examines automatic knowledge extraction and formal modeling of scientific literature, with the ultimate goal of creating a semantically-rich knowledge base of queryable scholarly data. I have several years of academic and industrial work experience, including startups in Montréal, collaborations with the University of Jena, Germany, as well as Concordia's Centre for Structural and Functional Genomics. Most recently, my work won the Semantic Publishing Challenge at ESWC 2015 (Task 2 - Most Innovative Approach), the Semantic Publishing Challenge at ESWC 2016 (Task 2 - Runner up) and a Best Paper Award at the WWW 2015 SAVE-SD Workshop on enhancing scholarly data.

Selected Readings:

Tim McKay - University of Michigan, Ann Arbor

I’m a data scientist, drawing inference from large data sets. My research over the last 25 years has been in two main areas: observational cosmology and higher education, and I am a Professor in the Departments of Physics and Astronomy as well as in our School of Education. I’ve also been an academic administrator, leading an 1800 student Honors Program for the College of Literature, Science, and the Arts at Michigan. These days, I spend my time acting as Director of the Digital Innovation Greenhouse (DIG) within the Office of Academic Innovation at Michigan. Within DIG, I lead development of the ECoach system for computer tailored communication. DIG works on a number of other projects, including MWrite, which is creating technologies which support using writing-to-learn pedagogies at scale, especially in large introductory science courses.

Dave Harlan - University of Michigan, Ann Arbor

I’m a software developer with the  Digital Innovation Greenhouse (DIG) within the Office of Academic Innovation at the University of Michigan.  I work full-time on the MWrite project, whose goal is to enable writing-to-learn in large format gateway courses.  I’ve been at DIG for about a year, prior to which I worked for University of Michigan’s Information and Technology Services.  Before my time at the University I worked in a number of roles for IBM.

Peter Foltz - Pearson and University of Colorado, Boulder”

I am a cognitive scientists and like to keep my feet in both academia and industry.   I am most interested in doing basic research and then translating it into practice and widespread use in education.   I currently lead an organization working on data analytics for writing for higher education within Pearson. I’m also a adjoint professor at the University of Colorado Boulder’s Institute of Cognitive Science.   I have developed a number of technologies that are widely used (millions of students) and also conduct research ranging from reading comprehension, writing analytics, clinical assessment, and 21st century skills, particularly collaborative problem solving.    

Heeryung Choi -  University of Michigan, Ann Arbor

Learning Science, data science, peer matching, PhD student (working with Christopher Brooks)

Jeanette Samuelsen - University of Bergen

I am a PhD candidate at University of Bergen (started in January 2017), where my project is about developing a Learning Analytics (software) architecture for Higher Education. My focus as of now is primarily on collecting and merging data from different educational sources, where the data can be of varying levels of structure (structured, semi-structured, unstructured text). In my project I will use semantic technologies. I also have an interest for Natural Language Processing.

Maren Scheffel - Open University of the Netherlands

Currently I am a researcher at the Welten Institute (Research Center for Learning, Teaching and Technology) of the OUNL. I originally studied computational linguistics at the University of Edinburgh and the University of Bonn which is why I am interested in this workshop although I have not done any work or research related to this field of study for quite a while. I previously worked at the Fraunhofer Institute for Applied Information Technology (FIT) focussing on aspects related to technology-enhanced learning. Since 2014 I have been working at the Welten Institute where I was involved in the management as well as the research for the LACE project and now contribute to the SHEILA project, the SafePAT project and the CompetenSEA project. My PhD work focuses on creating an evaluation framework for learning analytics.

Margaret Bearman - Deakin University, Melbourne Australia

I’m from the Centre for Research in Assessment and Digital Learning. I have a background in health professions and higher education as well as computer science. I’m interested in thinking about theoretical and conceptual basis of learning, practices of teaching and how technology intersects with these.

Grete Netteland - Sogn and Fjordane university college

I supervise master students .  I am also an institute leader.  It represents a challenge to write academic text. I am very interesting in how we can make it easier to write this kind of texts.

Elena Cotos - I am faculty at Iowa State University, Applied Linguistics Program. I am also the director of the Center for Communication Excellence, Graduate College. I do corpus-based ‘move’ analysis for technology-supported genre-based writing pedagogy, also research EAP/ESP, AWE, CALL. I developed the Research Writing Tutor. (see https://works.bepress.com/elena_cotos)

“Christian Rapp - Zurich University of Applied Sciences”

Sonya Woloshen - Simon Fraser University

I am a 3rd year PhD student at SFU in the Educational Technology and Learning Design programme. I am also a Secondary School French Immersion teacher.  I am new to the area of writing analytics but am eager to learn more and apply this to my profession and research endeavors surrounding second language acquisition with a particular focus on writing.

Jin myeong Chung - KERIS(Korea Education & Research Information Service)

I am working on publishing paper about Educational policy. Sometimes working with researchers from other institution, universities.

And I am also 2nd year PhD student at KNU in Computer Science, so interested in how to deal with good writing.

Jovita Vytasek- Simon Fraser University

PhD Graduate student in Educational Technology and Learning Design - Faculty of Ed. I also work for SFU Student Learning Commons to support students and would like to learn more about learning and writing tools.

Yi Cui - Simon Fraser University

PhD student in Educational Technology and Learning Design. Also working as researcher at New York University.

Thomas Ullmann - The Open University UK

http://qone.eu/ullmann 

Dr. Thomas Ullmann is Lecturer at the Institute of Educational Technology at the Open University, UK. He researches especially in the area of Technology Enhanced Learning, with his background in empirical educational science, and computer science. His current focus of research is text analytics for learning. He is especially interested in the automated detection of reflection in writings.

Check out ReflectR: http://qone.eu/reflectr

Lauren Barrows- Paragon Testing Enterprises (Vancouver, Canada) www.paragontesting.ca

Paragon Testing creates English language tests for high-stakes purposes, Canadian immigration and Canadian citizenship (CELPIP), and post-secondary entry (CAEL). The tests assess reading, writing, listening, and speaking proficiency.  The CAEL test is an integrated skills test. The speaking and writing performances are rated by trained human markers. The reading and listening sections are machine scored.

I am interested in assessing writing, training writing and speaking raters and item writers, automatic scoring, and creating writing prompts to elicit writing that targets a particular genre.

https://www.linkedin.com/in/laurenkennedybarrows/