1 of 15

When Chain-of-Thought Backfires

Evaluating Prompt Sensitivity in Medical Language Models

Binesh Sadanandan · Vahid Behzadan

SAIL Lab · University of New Haven

OPENING HOOK (before the title lands):

Quick show of hands before I start, keep them up, who here has told someone, "just add chain-of-thought, it'll reason better"? Most of this room. Now keep your hand up if you've ever checked whether that's true on a medical model. Right, almost everyone's hand just dropped. For the next fifteen minutes I'm going to be the person who checked. And I owe you a warning: the answer cost us six points of accuracy, and in a couple of cases, the right drug.

[~25s] SCRIPT:

Good morning. I'm Binesh Sadanandan, with Vahid Behzadan, from the SAIL Lab at the University of New Haven. The title says it plainly: when chain-of-thought backfires. We took a celebrated prompting trick everyone reaches for, and on a domain-tuned medical model, it didn't help. It cost us nearly six points of accuracy. Today I'll show you why, across two medical benchmarks.

TRANSITION to next slide:

But before we dig into why chain-of-thought failed, we need to confront a deeper problem: the headline accuracy number on a benchmark tells only part of the story about how these models actually behave when deployed.

2 of 15

Benchmark Accuracy Tells Part of the Story

2 / 15

THE BENCHMARK WORLD

clean, fixed exam format

86.5%

Med-PaLM 2 on MedQA

GPT-4 clears the passing bar by 20+ points

“LLMs are ready for the clinic”

THE DEPLOYMENT WORLD

inputs are never formatted identically

Reworded prompts

Chain-of-thought added

Few-shot examples

Shuffled answer options

Truncated / retrieved context

Accuracy? Reliability?

same�model

On the left of this slide is the world the benchmarks live in: a clean, well-formatted medical exam. On those leaderboards, frontier general models clear the human passing bar by a wide margin, and published medical systems like Med-PaLM 2 report around 86.5 percent on MedQA. That is the number I want you to hold onto: 86.5. It looks like a model that's ready for the clinic, and understandably, that fuels a lot of enthusiasm for clinical decision support. But now look to the right. This is the world your model actually meets in deployment: prompts reworded by a busy clinician, chain-of-thought added because someone read it helps, a couple of examples pasted in, options in a different order, a chart note that got truncated. None of that looks like the exam. So the real question isn't "can it pass the test" - it's how well that 86.5 holds up when the input format, the option order, and the context all shift underneath it. That gap, between benchmark accuracy and real-world reliability, is what this whole paperis about.

TRANSITION to next slide:

So if the published exam scores don't survive contact with deployment, we have to ask the question this talk is built around: do the prompt-engineering best practices we all rely on - chain-of-thought, few-shot examples, careful prompting - actually transfer to medical language models, or do they quietly make things worse?

3 of 15

Do Prompt-Engineering 'Best Practices' Transfer to Medical LLMs?

3 / 15

▸ CoT and few-shot are treated as universal 'best practices'

▸ But a model trained on medical text may have internalized different patterns

▸ We stress-test MedGemma across four axes (prompt, option order, context, scoring)

▸ Thesis: these techniques do NOT transfer, but reliable alternatives exist

4 of 15

Setup: Two Models, Two Benchmarks, Four Experiments

4 / 15

Prompt Ablation

MedMCQA · 4,183 · 10 conditions

Do CoT & few-shot help or hurt?

Option Order

MedMCQA · 4,183 · 5×3 seeds

Does shuffling options change the answer?

Evidence Conditioning

PubMedQA · 1,000 · 11 conditions

How does context truncation affect it?

Robustness Baselines

MedMCQA · 3 scoring methods

Can better scoring fix the fragility?

MedGemma-4B & 27B · frozen inference protocol · seeds {42, 123, 456}

2 / 15

So how do we actually test that? Two models, two benchmarks, four experiments. The models are MedGemma, the domain-tuned one, in both its four-billion and twenty-seven-billion parameter sizes, run on a single A100. We test them on MedMCQA, forty-one hundred medical board questions, four options each, and PubMedQA, a thousand yes-no-maybe research questions. And here's the thing I want you to trust me on: everything downstream is frozen. Same decoding, three fixed seeds, Wilson confidence intervals, McNemar tests. The only thing we ever change is the input. On screen, the four experiments. Top-left, does chain-of-thought help or hurt? Top-right, does shuffling the options change the answer? Bottom-left, how does truncating the evidence matter? Bottom-right, can smarter scoring fix the fragility? Four clean questions. The number to hold onto: forty-one eighty-three. Every MedMCQA result I show you is on that full validation set, not a cherry-picked slice.

TRANSITION to next slide:

And the very first thing that full set reveals is uncomfortable: every fancy prompting trick we tried lost to plain, direct prompting.

5 of 15

Finding 1: Prompt 'Best Practices' Hurt

5 / 15

▸ Zero-shot direct is BEST at 47.6%

▸ Chain-of-thought: −5.7 pts (41.9%)

▸ Few-shot examples: −11.9 pts (35.7%)

▸ All gaps significant (McNemar, p<0.001, Bonferroni-corrected)

CoT −5.7 pts Few-shot −11.9 pts (vs. direct 47.6%, random 25%)

5 / 15

We took the prompting tricks everyone reaches for, chain-of-thought, few-shot examples, and ran them head to head against the plainest thing you can do, just ask the question directly. On the left you can see the result. The green bar, zero-shot direct, sits highest at 47.6 percent. And every single fancier strategy falls below that dotted line. Chain-of-thought, the technique we tell everyone to use for hard reasoning, drops us 5.7 points, down to 41.9. Curated few-shot examples are even worse, almost twelve points down, at 35.7. These aren't noise. Every gap is statistically significant, McNemar's test, p below 0.001, Bonferroni-corrected across the board. So here's the one number I want you to walk away with: minus 11.9. That's what best-practice few-shot prompting cost us on a model built for medicine. The harder we tried, the worse it got.

TRANSITION to next slide:

But why would more reasoning make a medical model worse? Watch what happens when chain-of-thought tackles a poisoning case: it correctly names atropine as the antidote, then talks itself, step by step, into choosing neostigmine, the one drug that makes the patient worse.

6 of 15

Why CoT Backfires: It Talks Itself Out of the Answer

Q (Pharmacology): “Which drug is used for the treatment of acute gout?”

A) Allopurinol B) Colchicine ✓ C) Probenecid D) Febuxostat

DIRECT PROMPTING

Colchicine

correct ✓

CHAIN-OF-THOUGHT

“Colchicine treats acute flares… but NSAIDs are first-line… narrow therapeutic index… given side-effects…”

→ A (Allopurinol)

contraindicated in acute gout ✗

It identifies the right answer, then reasons its way out of it. Verbose reasoning in 90.7% of failures; self-contradiction in 25.6%.

2 / 15

[~70s] SCRIPT:

So why does chain-of-thought backfire? Here's the mechanism, and it's almost human. Across the full set, reasoning changed the model's answer on twelve hundred and sixty-two questions. It fixed five hundred and twelve. But it broke seven hundred and fifty. Net, we lost two hundred and thirty-eight answers we already had right. Look at the example on screen. Acute gout. On the left, direct answering picks colchicine. Correct, done. On the right, chain-of-thought actually names colchicine in step one, then talks itself out of it across its reasoning steps and lands on allopurinol, which is contraindicated during an acute attack. The knowledge was there. The reasoning destroyed it. And this isn't a fluke. In our failure sample, ninety-one percent of wrong answers came wrapped in long, verbose reasoning. A quarter literally contradict themselves mid-thought. So the number to hold onto here: minus two hundred thirty-eight. Adding reasoning to a model that already knew the answer made it worse.

TRANSITION to next slide:

If reasoning was the first way these models break, the second is even more insidious: few-shot examples don't teach the model the content at all, they just teach it a position habit.

7 of 15

Finding 2: Few-Shot Triples Position Bias

7 / 15

▸ Few-shot predicts 'A' 76% of the time; A is correct only 32%

▸ Position bias jumps 0.14 → 0.47

▸ The model copies the SHAPE of the examples, not their usefulness

Few-shot predicts “A” 76% of the time; A is correct only 32%.

7 / 15

So if chain-of-thought backfires, surely few-shot examples help, right? We show the model a few worked questions, it learns the pattern. Well, it does learn a pattern, just not the one we wanted. Look at the figure. On the left, the ground-truth answers are spread fairly evenly across positions, A through D, with A the correct answer about 32 percent of the time. Zero-shot, in the middle, is already a little A-happy at 46 percent. But watch the few-shot bars on the right. That A bar towers at 76 percent. The model now picks A more than three times out of four, when A is right less than a third of the time. Our position-bias score jumps from 0.14 to 0.47. Here's what's really happening: the model copied the shape of my examples, not their content. If my demos started with A, it learned to start with A. Remember 76 percent. Few-shot taught a habit, not knowledge.

TRANSITION to next slide:

And if you suspect that latching onto position rather than content is the real problem, the next experiment makes it undeniable: when we simply shuffle the answer options, the model's prediction flips on roughly 59 percent of questions.

8 of 15

Finding 3: Shuffling Options Flips the Answer 59% of the Time

8 / 15

▸ Mean flip rate 59.1% (rotate-1 up to 72.9%)

▸ Rotations drop accuracy up to 27.4 pts (down to 20.2%)

▸ Distractor swap (content changes, positions fixed): ZERO drop

▸ → The model follows POSITION, not content

59.1%

answers flip

mean flip rate when options are shuffled (up to 72.9%)

8 / 15

[~70s] SCRIPT:

So far the failures have been about how the model reasons. This one is scarier, because it's about whether the model is reading the question at all. Here's the test: I take the same question, the same four options, and I just change the order they appear in. The content is identical. On the left you can see what happens to accuracy. We start at 47.6 percent, and a single rotation of the options drops us all the way down to 20.2 percent. That's a 27-point collapse from doing nothing but moving text around. Now look at the donut on the right. Across these orderings, on average the model's answer flips on 59 percent of questions. Fifty-nine percent. And here's the control that proves it: when I swap out the distractors, changing the content but keeping the correct answer in its slot, accuracy doesn't move at all. Zero drop. So the model isn't choosing the right medicine. It's choosing a position on the screen, and dressing it up as an answer.

TRANSITION to next slide:

If position can hijack the answer, the next question is what happens when we feed the model real evidence and start cutting it away, and it turns out that where you cut matters far more than how much, because front-truncation is catastrophic while back-truncation is nearly free.

9 of 15

Finding 4: How You Truncate Matters More Than How Much

▸ Front-truncate 50% → 13.8%, WORSE than no context (34.5%)

▸ Back-truncate 50% → 44.5%, keeps 97% of full context (45.8%)

▸ 30.7-pt gap at the SAME cut = the largest effect in the study

▸ RAG lesson: truncate from the END; salient top-5 keeps 88% at half the tokens

30.7-pt gap between back-trunc (44.5%) and front-trunc (13.8%): the largest effect in the study.

9 / 15

[~70s] SCRIPT:

So far we've abused the question. Now let's abuse the evidence. In our PubMedQA experiments we gave the model a real abstract, then cut it in half two different ways. Same model, same task, same number of tokens thrown away. The only thing that changed was which half we kept. Look at the chart. Up top, full context lands at 45.8 percent, and back-truncation, keeping the end, holds at 44.5. It barely flinches, 97 percent of the full signal. Now drop to the red bar at the bottom. Front-truncation, where we keep the beginning and cut the end, collapses to 13.8 percent. That's below the dashed line, which is what the model scores with no context at all, 34.5 percent. We didn't just waste the evidence, we actively poisoned it. So here's the one number for this slide: 30.7 points. Same cut, opposite direction, the single largest effect in our entire study. For anyone building RAG, the lesson is blunt: truncate from the front, not the end, or just retrieve the salient sentences and keep 88 percent of the signal with far fewer tokens.

10 of 15

Finding 5: Scale ≠ Robustness Until You Read the Log-Probs

▸ 27B LOSES to 4B when generating (38.2% vs 45.8% full context)

▸ Cloze scoring flips it: 27B = 64.5%, 4B = 51.8%

▸ Cloze recovers +26.3 pts over 27B's own generation

▸ The model knows more than its generated text shows

27B loses on generation, then wins big on cloze (+26.3 pts). The model knows more than it shows.

2 / 15

[~70s] SCRIPT:

Here's the finding that surprised me most. We assume a bigger model is a safer model. Look at the left pair of bars. With full context and asked to generate an answer, our 27-billion-parameter model scores thirty-eight percent. The small 4-billion model? Forty-six. The big model loses. On its face, scale made things worse. But that conclusion is wrong, and the right-hand bars show why. Instead of letting the model write a sentence and parse it, we just read the log-probabilities of A, B, C, and D and pick the highest. Same model, same weights, same questions. Watch the 27B bar jump to sixty-four-and-a-half percent. The knowledge was there the whole time. Generation was hiding it. That recovery is the number to keep: plus twenty-six points, just from reading log-probs instead of parsing text. The 27B model isn't worse. We were measuring it wrong. Cloze scoring is the single best configuration in our entire study.

TRANSITION to next slide:

And that gap between knowing the answer and saying it isn't unique to scale, it's the common thread behind every result so far, which points straight at the generate-then-parse pipeline itself as the real bottleneck.

11 of 15

The Bottleneck Is the Generate-Then-Parse Pipeline

11 / 15

MODEL'S INTERNAL KNOWLEDGE

log-probs over A / B / C / D

autoregressive�GENERATION

regex�PARSE

answer

noise added at the generation step; only ~47.6% expressed

CLOZE: read it directly → 64.5%

bypass generation + parsing

The knowledge is there; generation and parsing leak it.

11 / 15

[~58s] SCRIPT:

Step back and look at what ties all of this together, because every failure we've seen is really one mechanism. The box on the left is the model's knowledge: log-probs over A, B, C, and D. That signal is good. But to produce an answer, it has to travel the red path across the top, autoregressive generation, then a parse. The noise comes from the generation step, not the parse; we checked, and parse errors are under two and a half percent, so they can't explain the gap. By the time we read an answer out of the generated text, only about forty-seven percent of what the model knows survives. Now follow the green shortcut along the bottom: cloze scoring skips generation entirely and just reads those probabilities directly. Same model, same question, but the position bias collapses to almost zero. The knowledge was never the problem. The pipeline was.

TRANSITION to next slide:

So if the pipeline is the bottleneck, the fix isn't a cleverer prompt, it's better extraction, and that's exactly what we hand you next: three deployable mitigations you can drop into production today.

12 of 15

Fixes That Work: Better Extraction Beats Better Prompting

12 / 15

▸ Cloze scoring: highest accuracy + near-zero bias (0.137→0.013)

▸ Permutation voting: +4 pts (49.0% vs 45.1%); 70% agreement = abstention signal

▸ Back-truncate context; prefer salient, high-density retrieval

Cloze cuts position bias from 0.137 to 0.013; permutation voting adds +4 pts and an abstention signal.

2 / 15

[~60s] SCRIPT:

So here's the good news. The knowledge is already in the model, so let's stop fighting the prompt and fix the extraction instead. On the left you see our zero-shot baseline at 47.6 percent. Now watch what happens. Permutation voting, where we ask under four different orderings and take the majority, nudges us to 49 percent, about four points over the per-permutation average, and the 70 percent agreement rate doubles as an abstention signal: when the votes split, the model is telling you to defer to a human. But the headline is cloze scoring. One forward pass, just read the log-probs of A, B, C, D. The 4-billion model jumps to 51.8 percent, and the 27-billion model hits 64.5. And look at the bias labels inside the bars: 0.137 down to 0.013, basically gone. The single number to take home: 64.5 percent, from better extraction, not a bigger prompt.

TRANSITION to next slide:

But before we celebrate that fix, I want to be clear about why this matters so much, because a flipped answer here isn't just a wrong percentage point, it can mean handing a patient a contraindicated drug.

13 of 15

This Is a Clinical Safety Problem, Not Just a Statistical One

13 / 15

⚠ Prompt choice alone can select a contraindicated drug

• Among the 750 CoT-broken answers, some are high-risk

• Organophosphate poisoning → CoT selects neostigmine (worsens it)

• Acute gout → CoT selects allopurinol (contraindicated)

• A system that flips on option order gives clinicians no reliable signal

2 / 15

[~55s] SCRIPT:

I want to slow down here, because this is the part that keeps me up at night. Everything I've shown you so far has been about accuracy and statistics. But look at the red panel on screen. Out of the 750 questions where chain-of-thought broke a correct answer, some of them are genuinely dangerous. Organophosphate poisoning: the model correctly identified atropine in its reasoning, then talked itself into neostigmine, which makes the patient worse. Acute gout: it selected allopurinol, which is contraindicated during an acute attack. Nobody changed the medicine. Nobody changed the patient. The only thing that changed was the prompt. And here's the number to hold onto: a system that flips on something as trivial as option order gives a clinician zero reliable signal about when to trust it. That's not a statistics problem. That's a safety problem.

TRANSITION to next slide:

So if the prompt itself can turn a correct answer into a dangerous one, what do we actually do about it? Let me pull this all together into five findings and four concrete recommendations for anyone deploying these models.

14 of 15

Five Findings, One Message

14 / 15

CoT

−5.7 pts

Chain-of-thought hurts

Few-shot

−11.9 pts

Triples position bias

Order

59.1%

Shuffling flips answers

Context

13.8%

Front-trunc < no context

Cloze

64.5%

Reading log-probs wins

Standard prompt-engineering “best practices” do not transfer to medical LLMs

✔

Default to zero-shot direct prompting or cloze scoring

✔

Test option-order sensitivity; vote across orderings and abstain on disagreement

✔

Truncate retrieved context from the END, not the beginning

✔

Validate per use-case: benchmark accuracy ≠ deployment robustness

2 / 15

[~55s] SCRIPT:

So here's where we land. Five findings, one message. On the left you can see the five chips that summarize the whole talk: chain-of-thought cost us 5.7 points; few-shot cost 11.9 and tripled position bias; shuffling options flipped 59 percent of answers; front-truncated context dropped to 13.8, worse than giving no evidence at all; and cloze scoring climbed to 64.5. Every one of those is a standard best practice that simply does not transfer to a medical model. So four things to take home. Default to direct or cloze scoring. Test your option order, then vote or abstain. When you must truncate, cut from the end, not the front. And validate per use-case. Because the deepest lesson is this: benchmark accuracy measures your prompt-generate-parse pipeline, not the model's knowledge. The knowledge was there the whole time.

TRANSITION to next slide:

That's the whole story, so let me close by thanking you for your attention, pointing you to the full code and data at our GitHub repo, and opening the floor for your questions.

15 of 15

Thank You

Standard prompt-engineering “best practices” do not transfer to medical LLMs.

github.com/UNHSAILLab/MedMCQA-Robustness-Study

Questions?

https://bineshkumar.me/research/