1 of 18

PNC 2026 | Intelligence in Context: Practices, Impact, and Connection

Toward Humanistic AI in Digital Humanities Education:

Assessing Conceptual Transfer in an AI-Supported Linguistics Classroom

Mei-Rong Alice Chen,

Associate Professor,

Soochow University, TW

When our students learn a new concept, do they truly know what it means, how to show it, and how to actually use it? Which one can they do it better?
And how often are they using AI to help them get it done ?

2 of 18

Outline

Study background
Research Questions
Educational framework-The DEA Scaffold
Method

3 of 18

When our students learn a new concept, do they truly know what it means, how to show it, and how to actually use it? Which is better?

And how often are they using AI to help them get it done ?

Background

In my English-major classes, I noticed something interesting.

Some students are very good at remembering definitions, but when I ask them to give an example or think about how the concept might matter in an EFL classroom, they are not always sure what to do.Other students are almost the opposite. They can give very good examples, or even suggest interesting teaching applications, but when I ask them to explain the concept itself, the definition may still be vague or incomplete.

So I became interested in the relationship among three things: Can students define a concept? Can they show it with an example? And can they actually use it?

In this study, I looked at one chapter on language acquisition and used a three-column worksheet: Definition, Example, and Application. I wanted to see what kinds of gaps appeared across these three parts and whether some linguistic concepts were easier to transfer into teaching ideas than others.

There was also another classroom reality. In our English department, students receive different messages about generative AI. Some instructors do not allow AI use in their courses, so even though I did not require or prohibit AI in this activity, some students chose to complete the worksheet entirely on their own, while others may have used AI as part of their work. Rather than treating this as an AI experiment, I wanted to observe what happened in an ordinary classroom setting. So the second question was whether responses showing different levels of observable AI-assistance traces also differed in rubric-based response quality.

4 of 18

The challenge

Students could often say “what it means”, but not yet “what to do with it”.

Linguistic concepts are useful but abstract.
Definitions and textbook examples do not automatically become teaching decisions.
The gap became visible when students had to write an EFL classroom application.

Actual student response

Overgeneralization happens when learners apply a grammar rule too broadly. Example: goed instead of went.

Stronger application example

Teachers should see this as progress and model the correct form naturally, such as ‘Yes, you went there.

For example, take a familiar concept such as overgeneralization.

A student may write: “Overgeneralization happens when learners apply a grammar rule too broadly. For example, goed instead of went.”

This is correct. If I only ask for a definition and an example, I may conclude that the student understands the concept quite well.

But then I ask one more question:

“So what should an English teacher do when a learner says goed?”

Now the task changes. A stronger answer might be:

“Teachers should see this as progress and model the correct form naturally, such as ‘Yes, you went there’”

This response does something different.

The student is no longer simply recalling a linguistic concept. The student is using that concept to make a pedagogical decision.

And this was the key issue for me.

Definitions and textbook examples do not automatically become teaching decisions.

At the same time, I also did not want to assume that every student would show the same pattern.

Some students might have strong definitions but weak applications.

Others might have weak definitions but surprisingly good applications.

So rather than asking only, “Do students understand the concept?”, I wanted to see where the understanding was strong, where it was weak, and whether the direction of the gap changed across different concepts.

[Transition]

That led to the two research questions.

5 of 18

Research Questions

One learning problem viewed through two lenses.

Conceptual transfer

RQ1. How do EFL undergraduates' responses to a three-column concept worksheet reveal gaps between conceptual understanding and pedagogical transfer, and how do these gaps vary across linguistics concepts?

AI in context

RQ2. How are observable traces of AI assistance associated with rubric-based response quality, and what do these patterns imply for worksheet design and AI-supported concept learning?

6 of 18

Educational framwork-The DEA Scaffold

Three simple questions turn thinking into visible evidence.

What is it?

Definition

Can I show it?

Example

Can I use it?

Application

Conceptual recall

Contextual transfer

Revised Bloom's Cognitive Processes Taxonomy (Anderson & Krathwohl, 2001)

I call this the DEA scaffold./DEA stands for Definition, Example, and Application.

The three questions are very simple../Definition:/What is it?/ Example: Can I show it? Application: Can I use it?/The first two are familiar in linguistics classes.

We often ask students to define a term and give an example./ The third question is the important addition.

Can I use it?/ For example, if students learn about the silent period, can they connect that idea to what a teacher might do with a beginner who understands English but does not yet want to speak?/ If students learn about inversion, can they connect it to a difficulty that Mandarin-speaking EFL learners might have when forming English questions?/ So the three columns are not three different tests.

They are three views of the same concept./ And by separating them, I can see where students’ understanding begins to change.

[Transition] I then placed this scaffold into a collaborative online activity.

7 of 18

Method: Web-Based Collaborative Matrix

Individual responsibility:

every student had a designated response space.

Peer visibility:

group members could compare responses in real time.

Structured data:

final responses became concept-level records for analysis.

AI as part of the ecology

Students were allowed to use generative AI tools, reflecting authentic classroom conditions rather than an artificial no-AI setting.

Connection in context

The matrix made reasoning visible to peers, the teacher, and later the researcher.

Slide 5. Web-Based Collaborative Matrix/ The activity was hosted on Google Sites and connected to a shared Google Sheet.

Each student had a designated space for responding.

At the same time, students could see the responses of their group members.

So the activity had three functions.

First, individual responsibility.

Each student still had to contribute.

Second, peer visibility.

Students could compare how other people defined or applied the same type of concept.

Third, the responses became structured classroom data.

Instead of having one final score for the entire worksheet, I could examine each concept-level response separately.

And AI was simply part of this learning ecology.

Students were not required to use generative AI.

They were also not prohibited from using it in this activity.

This was important because I wanted to observe what actually happens in a normal classroom where students have different habits, different beliefs, and different experiences with AI.

[Transition]

Let me show you what these responses actually looked like.

8 of 18

https://sites.google.com/view/data-lab-mrc/1st-language-acquisition?authuser=0

Students’ responses

Concepts

9 of 18

Method: Data, Scoring, and Reliability

Directed content analysis, Hsieh & Shannon, 2005

Coders reliability

Definition Q: 37.8%, κ = .068

Example Q: 80.3%, κ = .409

Application Q: 53.2%, κ = .239

Application AI Trace: 55.9%, κ = .186

Disagreements were resolved by 3^rd coder

Coder 3 reread the original responses and applied the same codebook.

Participants	Data
42 EFL undergraduates, mean Age 20.5	188 concept-level records
RQ1	RQ2
Rubric dimensions: Definition Quality, Example Quality, Application Quality	AI Traceability: No, Possible, Strong, coded as observable text features

Messick (1995) introduced the concept of "construct underrepresentation" in his influential validity theory.

Slide 7. Data, Scoring, and Inter-Coder Reliability

The dataset contained 188 concept-level records from the class activity.

Each response was evaluated separately for Definition Quality, Example Quality, and Application Quality.

The quality scale used three levels: 1, 2, and 3.

For application, for example, a high-quality response needed more than a general suggestion.

It needed a reasonably concrete classroom action and a meaningful connection to the linguistic concept.

AI Trace was coded separately as:

No,�Possible,�or Strong.

Again, this was not a judgment of actual authorship.

It was only a judgment of observable textual features.

We also wanted to make the coding process transparent.

Two coders first coded the responses independently.

The agreement was not equally strong across all dimensions.

For Example Quality, exact agreement was relatively high, about 80%.

But Definition Quality was much more difficult, with about 38% exact agreement.

Application Quality was about 53%.

Application AI Trace was also challenging, at about 56%.

This disagreement was actually informative.

It showed us that deciding whether an answer is “partially correct” or “fully developed” is not always straightforward.

It was also difficult to distinguish between a polished human response and a response that might contain observable AI-related features.

So instead of simply averaging the two coders, we introduced a third coder for adjudication.

Coder 3 returned to the original student response and the coding guide, without automatically privileging either Coder 1 or Coder 2.

The final adjudicated scores were then used for the results I will show next.

[Transition]

And after adjudication, the first result was actually different from my original expectation.

10 of 18

Result 1-1

Interpretation

Understanding was not a single skill.

Students could be stronger at defining, showing, or using the same concept.

RQ1 answer: Definition VS Example

Pattern	%	interpretation	Concepts
Example > Definition	50.5%	Example stronger than Definition Students can show it, but cannot fully explain it.	Input, Acquisition schedule, Developing syntax, Inversion
Example = Definition	41.5%	Similar performance�Students can explain it and show it.	Cooing, Babbling
Definition > Example	8.0%	Definition stronger than Example Students can explain it, but struggle to find a good example.	First language acquisition Telegraphic speech Holophrastic speech

When we compared Definition and Application scores across the 188 responses, three patterns appeared.

The largest group, 56.4%, showed balanced performance:�students explained the concept and applied it at about the same level.

But almost one-third of the responses, 29.3%, showed something more interesting:�students were better at applying the concept than defining it.

Only 14.4% showed the pattern we originally expected:�students could explain the concept better than they could use it.

So the story is not simply:

“Students know the theory, but cannot apply it.”

Instead, students showed different kinds of understanding.

Some students could explain a concept clearly but struggled to turn it into a teaching decision.

Others could suggest a useful classroom application even when their formal definition was incomplete.

In other words, knowing how to define a concept and knowing how to use it are related, but they are not the same ability.

[Transition]

The second finding helps explain why this happens.

11 of 18

Result 1-2

RQ1 answer: Definition vs Application

Pattern	%	Plain-language interpretation
Definition = Application	56.4%	Similar performance between Definition and Application can explain it and use it at a similar level.
Application > Definition	29.3%	Application stronger than Definition can use it better than can explain it.
Definition > Application	14.4%	Definition stronger than Application can explain it better than can use it.

When we compared Definition and Application scores across the 188 responses, three patterns appeared.

The largest group, 56.4%, showed balanced performance:�students explained the concept and applied it at about the same level.

But almost one-third of the responses, 29.3%, showed something more interesting:�students were better at applying the concept than defining it.

Only 14.4% showed the pattern we originally expected:�students could explain the concept better than they could use it.

So the story is not simply:

“Students know the theory, but cannot apply it.”

Instead, students showed different kinds of understanding.

Some students could explain a concept clearly but struggled to turn it into a teaching decision.

Others could suggest a useful classroom application even when their formal definition was incomplete.

In other words, knowing how to define a concept and knowing how to use it are related, but they are not the same ability.

[Transition]

The second finding helps explain why this happens.

12 of 18

Result 1-3

Example vs Application

Pattern	%	Plain-language interpretation
Example = Application	60.6%	Similar performance between Example and Application can show it and use it.
Example > Application	33.5%	Example stronger than Application can show it, but less sure how to use with it.
Application > Example	5.9%	Application stronger than Example can use it even when example is less developed.

When we compared Definition and Application scores across the 188 responses, three patterns appeared.

The largest group, 56.4%, showed balanced performance:�students explained the concept and applied it at about the same level.

But almost one-third of the responses, 29.3%, showed something more interesting:�students were better at applying the concept than defining it.

Only 14.4% showed the pattern we originally expected:�students could explain the concept better than they could use it.

So the story is not simply:

“Students know the theory, but cannot apply it.”

Instead, students showed different kinds of understanding.

Some students could explain a concept clearly but struggled to turn it into a teaching decision.

Others could suggest a useful classroom application even when their formal definition was incomplete.

In other words, knowing how to define a concept and knowing how to use it are related, but they are not the same ability.

[Transition]

The second finding helps explain why this happens.

13 of 18

RQ1 story across all three DEA dimensions:

Definition → Example: students often knew what a concept looked like before they could define it precisely.
Example → Application: once they could show the concept, moving from an example to a pedagogical decision was still a challenge for about one-third of responses.
Definition → Application: the relationship was more mixed and could go in either direction.

14 of 18

Result 2: AI Trace Was Descriptive

Most responses showed little or no observable AI trace.

Application Quality increased as AI trace became stronger

AI Trace	Application Quality	What we can observe
No 66.5%	2.32	More direct or less elaborated responses
Possible 21.8%	2.71	More developed explanations
Strong11.7%	2.86	More polished, structured, and pedagogically elaborated responses

15 of 18

Conclusion

Conceptual learning did not develop as a simple

Definition → Example → Application sequence.

González-Fernández and Schmitt (2020) found that recognition and recall aren't just "easy" and "hard" versions of the exact same brain power.

It proves that knowing about a concept and being able to produce it draw on different mental strengths. The three tasks are related, but they tap different aspects of conceptual understanding and pedagogical transfer.

Better-looking answers ≠ deeper understanding.

Final products show quality, but not the learning process.

16 of 18

Design Implications and Next Step

The DAE task should make the reasoning link visible, not only the final answer fluent.

Use two kinds of bridging prompts:

1. Concept to practice: For cooing or consonant development, ask “What EFL problem does this concept help explain?”

2. Practice to concept: When students give a good activity, ask “Which concept justifies this action?”

3. For AI-supported learning, ask AI to critique the concept-application link instead of generating the answer.

Slide 11. Design Implications and Next Step

The first implication is that we probably need to assess more than definitions.

If I only ask, “What is overgeneralization?”, I may miss a student who has difficulty with formal wording but actually understands how the concept affects teaching.

Likewise, if a student gives me a beautiful classroom application, I should not automatically assume that the student understands the linguistic principle behind that activity.

So future worksheets should make the connection itself visible.

For example, instead of asking only:

“What would you do?”

we can ask:

“What learner difficulty does this concept help explain?”

“What would you do as a teacher?”

“And why does this action match the linguistic concept?”

The second implication concerns AI.

I do not think the best question is:

“Did this student use AI?”

That is becoming increasingly difficult to answer from a final product.

A more useful question is:

“Can I see the student’s reasoning?”

So in the next version of the activity, I would like to collect more process evidence.

For example:

the first response,

the prompt given to AI,

AI feedback,

the student’s revision,

and perhaps a short explanation of what the student accepted or rejected.

This would allow us to distinguish between AI simply polishing an answer and AI actually participating in conceptual learning.

[Transition]

So I would like to end by returning to the conference theme: Intelligence in Context.

17 of 18

Conclusion: Intelligence in Context

Practices, impact, and connection in one classroom matrix.

Practice: DEA made students’ definitions, examples, and applications visible.

Impact: adjudicated evidence showed that conceptual understanding and pedagogical transfer were unevenly aligned.

Connection: AI trace was treated as part of the learning ecology, not as proof of authorship.

Assessment: the key question became “Can we see the reasoning link?”

Thank you. I welcome your suggestions.

This study started from a very ordinary classroom observation./Some students can define a concept but cannot easily use it.

Some students can use a concept but cannot clearly define it./And in the same classroom, some students may work independently while others may use generative AI as part of their learning process./The three-column DEA worksheet made these differences visible.

Definition showed one kind of understanding./Example showed whether students could recognize or instantiate the concept.

Application showed whether they could move toward pedagogical reasoning./And the AI Trace analysis reminded us that a polished final answer is not the same thing as visible learning.

So for me, the main message is this:

Intelligence is not located only inside the student, and it is not located only inside the AI tool.

It appears in the interaction among the learner, the linguistic concept, the classroom task, peers, AI tools, and the way we assess learning.

The key question is therefore not simply:

“Did the student get the answer right?”

And perhaps not even:

“Did the student use AI?”

The more important question is:

“Can we see how the student connected the concept, the example, and the classroom decision?”

That is what I mean by making pedagogical transfer visible.

Thank you very much.

I look forward to your questions and suggestions.

18 of 18

Score	Definition Accuracy	Example Appropriateness	Classroom Application Quality
Strong	Concept is restated in the student’s own words; all key attributes are accurately captured; clearly distinguishable from related concepts.	Example clearly instantiates the concept’s defining attributes; distinguishable from neighboring concepts; self-generated examples demonstrate transfer beyond the textbook.	Specific, operational classroom action is proposed; rationale connects the activity to the concept; demonstrates awareness of EFL learner characteristics, such as L1 transfer, anxiety, or limited lexicon.
possible	Core meaning is correct but missing one key attribute, or is overly generic; minor errors do not undermine the concept.	Example is in the right direction but missing a defining attribute, or is partially conflated with a related concept.	Suggestion is reasonable but underspecified; correct in direction but lacks concrete activity, materials, or sequencing; may read as a generic teaching suggestion.
Poor	Definition is inaccurate, conflated with another concept, or copied nearly verbatim from the textbook without restatement.	Example does not match the concept, contains a conceptual error, or is misplaced, such as being placed in the wrong column.	Suggestion is not substantively connected to the concept, restates the definition, or consists only of a generic teaching cliché, such as “encourage students to practice more.”
0 Missing	Column left blank.	Column left blank.	Column left blank.