LAK17 Writing Analytics Workshop • Participant Working Notes
Public workshop website • These notes: http://bit.ly/lak17wanotes
Thinking Like a Student - problems
Supporting students overcome writing problems
Thinking like a student:
What is the primary obstacle to good writing?
Thinking like a researcher/educator/ academic:
What pedagogic strategies support learning to write/tackle this obstacle?
Thinking like a technologist:
How can analytics augment this strategy?
A lack of knowledge on development of ideas and the strategies/tools for taking rough, fuzzy, ill-formed ideas and developing them over multiple iterations towards a polished product.
Collaboration - peer review, editor/ non-editor. Competencies for participating in peer review process.
Making the process visible - emerges over time towards a polished product (which may be always different). Organic growth rather than step by step.
Different types of support for different periods in an iterative process.
Allowing time for the process.
Brainstorming tools, ideas mapping - Capturing the messy, fuzzy ideas. Free write - take the topics and prompt for development. Remove the threat of writing being read - just between the student and the computer for initial free write - analytics helps with formulation of initial ideas for first sharable draft.
Scroll through an exemplar over time.
Director’s commentary over the exemplar edits.
Model mental models - make the thinking of the (good) writer visible.
In-process formative feedback and metacognitive scaffolding
Automated feedback; intelligent agents
Unfamiliarity with the domain
Timely feedback; find ways to ensure that students are focusing on conceptual cohesion
Different priorities among faculty as to the quality of a piece of writing -- two people may grade differently / may evaluate arguments differently
Can create rubrics to create a common ground between the different point of views on argumentation
Benchmarking / calibrated peer review tasks
Detect change between drafts
Create graphs to identify “binding terms” / terms that appear to be central to a concept or group of concepts
Identifying the right tools for measurement - making sure we’re measuring what we want to measure
Analytics that consider both the “what” and the “when”
Interpreting data is a challenge - use data that makes sense, not what is most convenient to collect
Design analytics that have theoretical backing in learning science.
Analytics should provide guidance for the next step without overwhelming the learner.
Writer’s block. Not knowing what is expected or the style for a specific genre. Weak at building the argumentative structure of the text.
Prepare checklists/How-Tos/scaffolds for students. Hands-on training with existing articles. Peer-review (on already published articles) as class/course exercise. Reflection on the feedback comments.
Writing tools plugins, fuzzy analytics of correspondence between different rhetorical elements of a document (claim matching results, contributions appearing in abstract, main body, conclusion).
Make use of peer review
Analytics can facilitate peer review
Know all of the best words (stylistic vocabulary)
Know all of the best words (domain vocabulary)
Appropriate depth and style (short v long sentences, over v under elaboration)
The structure of a particular genre of text and cultural differences therein
What is an “analytic”? -- Anything you can count
How can these tools be deployed in practical pedagogic contexts?
What knowledge is needed to do that successfully?
A3R Project using AWA - Gibson
Provides near instant reflective writing feedback on submission (i.e. not during writing)
Best used in contexts where reflection explicitly taught, and really embedded in teaching.
Baseline reflective writing knowledge
Ability to take action given lack of highlighting (i.e., no reflective moves identified)
Bolo interactive writing analytics
Content knowledge/background knowledge gap. Large number of articles i.e. ‘searching for and filtering information sources’
Provides real-time keyterm suggestions to recommend new resources to learners, based on instructor list of terms + associated resources.
Has interface to show paragraphs that relate to particular keyterms identified.
Instructor must have key-terms and associated articles
How to integrate the new info into paper
Elena Cotos (Iowa State University) – Research Writing Tutor (RWT) – A corpus-based platform for data-driven writing (webinar discussing RWT)
Data-driven learning approach in writing instruction
Students lack genre awareness
Understanding of genre conventions (communicative moves and rhetorical strategies) in disciplinary writing
M-write David Harlan, Tim Kay, et al.,
Anne Gere, Ginger Shultz, Chris Teplovs, Dave Harlan (University of Michigan) – M-Write: A Large-Scale Laboratory for Writing Analytics Research (pdf)
Actionable feedback based on large corpora of student texts.
How do we use the corpus to provide diagnostics to instructors and students?
Christian Rapp, Otto Kruse, Madalina Chitez, Jakob Ott. (Zurich University of Applied Sciences) – Thesis Writer (TW) – an Intelligent Tutoring System for Writing Instruction and its Study (Thesis Writer abstract (pdf); and webinar discussion of the tool)
www.thesiswriter.eu available in German and English.
I would like to draw your attention to that conference http://eataw.eu/conferences.html where we will have a 3 (!) hour symposium on tools supporting writing. Most of the European tools will be represented.
“Proposal wizard”: Structured writing template for writing a proposal with brief guidance at each stage - designed to get it written fast first.
“Thesis editor”: Then more feedback on submission - Phrase book, examples, and tutorial, and a corpus tool/concordancer for keywords.
Gives holistic overview, and allows collaboration
User guide, overview intro to text genre
Bahar Sateli and Rene Witte (Concordia University) – Personal Research Assistants for Young Researchers (pdf).
How do students and academics make sense of vast amounts of literature?
System to deal with corpus of (computer science) articles.
The presented research is more an “automated workflow” than a readily available tool.
Noureddine Elouazizi, et al. (University of British Columbia) – A Formal Semantics-informed Learning Analytics Technology for Analyzing Written Argumentation
How do students learn to argue in the genre?
Give instructors feedback on a corpus of student essays - instructors get a dashboard showing how students are doing on each
Give students feedback on their own data
Given gauges/visual traffic lights on quantity of particular things within a text
How do we encourage students to draft? (limit numbers of drafts to submit, but give feedback per-graph)
What does good argument look like?
What do argumentative moves look like?
What’s the right argument (e.g. some issues there’s a correct(er) side to be on!)
What’s the ‘correct amount’ of a particular move?
Danielle McNamara and Laura Allen (Arizona State University) – The Writing Pal: A Writing Strategy Intelligent Tutoring System (ITS) (SoLET Lab)
Persuasive writing (not source based) designed to support basic argumentative writing (structure of a text, notion of a ‘thesis’, etc.).
Instructors can direct students to particular tasks
Feedback doesn’t focus on errors or line-by-line feedback but the strategy they need to focus on
Blended activities workbook to support modeling
Lower engagement if tools are seen as ‘separate’ online only
Preferably, some understanding of the strategies
Materials to support the skill of writing vs quality of the content
Is a tool trying to improve academic thinking, or the student’s understanding of a particular genre of academic writing? (Or both?)
Both tools focus on the latter only (the argument or reflection might be weak/nonsensical) in order to help the student reflect on whether they have communicated their ideas in an appropriate way. Help them reflect on “Did you really mean to say this?” (because that’s how it will be read in this genre...)
In coming to understand the academic moves that define a genre, this may in turn help them improve their thinking (in the way that any good visualisation/symbol system can ‘talk back’ to the user). But the student will have to learn how to “read” the analytics feedback (as with any visualisation/symbol system)
Formative feedback tools like these cannot (or aren’t designed to) grade the overall quality. The tools don’t have a model of “good” per se, but rather, are built based on detecting the hallmarks of a genre, and based on confidence that “giving this kind of feedback is generally helpful” — beyond the NLP, feedback design is a critical step.
High performing writers will invariably break any set of genre rules — but hopefully they understand that they’ve done so.
The importance of framing the tool for students:
Briefings are important so the students don’t come to the tool cold. But we can’t depend on students attending/remembering such briefings: the tools themselves do also need to convey how they should be used.
Writing tasks will fundamentally shape student use of the tool
What is “good”?
There are NLP models of writing (eg rhetorical parser), and models of how the learner will engage with these models will shape the UI
Sculpture metaphor - you work the stone until the inner figure is revealed “clearly enough” — requires a lot of revision
What learning problem/task would produce these?
What interventions would support writing in context of that learning?
What tools or data would give insight for those interventions?
What pre-requisite knowledge is required to engage with that tool/data
What would a user (student, teacher, other?) actually do with the tool/data?
Feel free to share thoughts here
Graham, S., & Perin, D. (2007). Writing next: Effective strategies to improve writing of adolescents in middle and high schools –A report to Carnegie Corporation of NewYork. Washington,DC:Alliance for Excellent Education.
Evidence Centred Design https://www.ets.org/Media/Research/pdf/RR-03-16.pdf
Where do we go next?...
Want to make a note of something? Nowhere else to put it? Here’s the place to put it…
Please put any relevant information below, we’ll convert this to a ‘view only’ doc and link from the workshop website after the day.
I’m one of the workshop organisers, and a lecturer at the University of Technology Sydney. I am primarily interested in student epistemic cognition (their understanding of where knowledge comes from, how it is justified, and how claims are inter-related). One means through which to investigate epistemic cognition is via writing practices and the written products that students produce in developing evidence-based texts. I’m interested in open education, and development of resources to build capacity in using evidence and high quality assessment in education.
Workshop organiser, Professor of Learning Informatics at the University of Technology Sydney, and Director of the Connected Intelligence Centre. I’m interested in analytics for building the critical competencies that students will need for the future of work, and citizenship. In this context, analytical and reflective writing are framed as windows onto the mind of a learner’s capacity for critical thinking and deep reflection. My HCI background and work on visualizing argumentation have shaped how I think about the user experience, and the role that visual feedback can play in shaping cognition.
Workshop organiser, Research Fellow in Writing Analytics with the Connected Intelligence Centre. I’m particularly interested in Reflective Writing Analytics, and in student focused Learning Analytics. More generally I have an interest in the ways we bring together computational techniques and human problems.
Danielle McNamara - Arizona State University - Professor in Psychology and the Institute for the Science of Teaching and Learning. Conduct research in writing, comprehension, learning, and language. Developed the Writing Pal, Coh-Metrix, iSTART, and other natural language processing tools. Examine language in multiple contexts.
Workshop organizer, PhD student in Cognitive Science at Arizona State University. I am interesting in examining the cognitive and affective processes involved in language comprehension, writing, and knowledge acquisition, and to apply that understanding to educational practice by developing and testing educational technologies.
My background is 30+ years in Learning Science, AI, Intelligent Tutoring Systems, Computer Science - research and product. My role at Intel is largely “thought leadership” and “ecosystem development” - so I spend a lot of time learning about emerging technology trends and analyzing how they might impact education K-20 globally. LinkedIn profile.
I am a PhD Candidate (ABT) at the Department of Computer Science and Software Engineering, Concordia University (Montréal). My research focuses on applications of semantic technologies, in particular text mining and semantic web, in knowledge-intensive domains. My PhD topic is Semantic Publishing, which examines automatic knowledge extraction and formal modeling of scientific literature, with the ultimate goal of creating a semantically-rich knowledge base of queryable scholarly data. I have several years of academic and industrial work experience, including startups in Montréal, collaborations with the University of Jena, Germany, as well as Concordia's Centre for Structural and Functional Genomics. Most recently, my work won the Semantic Publishing Challenge at ESWC 2015 (Task 2 - Most Innovative Approach), the Semantic Publishing Challenge at ESWC 2016 (Task 2 - Runner up) and a Best Paper Award at the WWW 2015 SAVE-SD Workshop on enhancing scholarly data.
I’m a data scientist, drawing inference from large data sets. My research over the last 25 years has been in two main areas: observational cosmology and higher education, and I am a Professor in the Departments of Physics and Astronomy as well as in our School of Education. I’ve also been an academic administrator, leading an 1800 student Honors Program for the College of Literature, Science, and the Arts at Michigan. These days, I spend my time acting as Director of the Digital Innovation Greenhouse (DIG) within the Office of Academic Innovation at Michigan. Within DIG, I lead development of the ECoach system for computer tailored communication. DIG works on a number of other projects, including MWrite, which is creating technologies which support using writing-to-learn pedagogies at scale, especially in large introductory science courses.
I’m a software developer with the Digital Innovation Greenhouse (DIG) within the Office of Academic Innovation at the University of Michigan. I work full-time on the MWrite project, whose goal is to enable writing-to-learn in large format gateway courses. I’ve been at DIG for about a year, prior to which I worked for University of Michigan’s Information and Technology Services. Before my time at the University I worked in a number of roles for IBM.
I am a cognitive scientists and like to keep my feet in both academia and industry. I am most interested in doing basic research and then translating it into practice and widespread use in education. I currently lead an organization working on data analytics for writing for higher education within Pearson. I’m also a adjoint professor at the University of Colorado Boulder’s Institute of Cognitive Science. I have developed a number of technologies that are widely used (millions of students) and also conduct research ranging from reading comprehension, writing analytics, clinical assessment, and 21st century skills, particularly collaborative problem solving.
Learning Science, data science, peer matching, PhD student (working with Christopher Brooks)
I am a PhD candidate at University of Bergen (started in January 2017), where my project is about developing a Learning Analytics (software) architecture for Higher Education. My focus as of now is primarily on collecting and merging data from different educational sources, where the data can be of varying levels of structure (structured, semi-structured, unstructured text). In my project I will use semantic technologies. I also have an interest for Natural Language Processing.
Currently I am a researcher at the Welten Institute (Research Center for Learning, Teaching and Technology) of the OUNL. I originally studied computational linguistics at the University of Edinburgh and the University of Bonn which is why I am interested in this workshop although I have not done any work or research related to this field of study for quite a while. I previously worked at the Fraunhofer Institute for Applied Information Technology (FIT) focussing on aspects related to technology-enhanced learning. Since 2014 I have been working at the Welten Institute where I was involved in the management as well as the research for the LACE project and now contribute to the SHEILA project, the SafePAT project and the CompetenSEA project. My PhD work focuses on creating an evaluation framework for learning analytics.
I’m from the Centre for Research in Assessment and Digital Learning. I have a background in health professions and higher education as well as computer science. I’m interested in thinking about theoretical and conceptual basis of learning, practices of teaching and how technology intersects with these.
I supervise master students . I am also an institute leader. It represents a challenge to write academic text. I am very interesting in how we can make it easier to write this kind of texts.
Elena Cotos - I am faculty at Iowa State University, Applied Linguistics Program. I am also the director of the Center for Communication Excellence, Graduate College. I do corpus-based ‘move’ analysis for technology-supported genre-based writing pedagogy, also research EAP/ESP, AWE, CALL. I developed the Research Writing Tutor. (see https://works.bepress.com/elena_cotos)
I am a 3rd year PhD student at SFU in the Educational Technology and Learning Design programme. I am also a Secondary School French Immersion teacher. I am new to the area of writing analytics but am eager to learn more and apply this to my profession and research endeavors surrounding second language acquisition with a particular focus on writing.
I am working on publishing paper about Educational policy. Sometimes working with researchers from other institution, universities.
And I am also 2nd year PhD student at KNU in Computer Science, so interested in how to deal with good writing.
PhD Graduate student in Educational Technology and Learning Design - Faculty of Ed. I also work for SFU Student Learning Commons to support students and would like to learn more about learning and writing tools.
PhD student in Educational Technology and Learning Design. Also working as researcher at New York University.
Dr. Thomas Ullmann is Lecturer at the Institute of Educational Technology at the Open University, UK. He researches especially in the area of Technology Enhanced Learning, with his background in empirical educational science, and computer science. His current focus of research is text analytics for learning. He is especially interested in the automated detection of reflection in writings.
Check out ReflectR: http://qone.eu/reflectr
Lauren Barrows- Paragon Testing Enterprises (Vancouver, Canada) www.paragontesting.ca
Paragon Testing creates English language tests for high-stakes purposes, Canadian immigration and Canadian citizenship (CELPIP), and post-secondary entry (CAEL). The tests assess reading, writing, listening, and speaking proficiency. The CAEL test is an integrated skills test. The speaking and writing performances are rated by trained human markers. The reading and listening sections are machine scored.
I am interested in assessing writing, training writing and speaking raters and item writers, automatic scoring, and creating writing prompts to elicit writing that targets a particular genre.