PHIL 611
Measuring and Improving Reasoning in Humans and Machines
Graduate seminar · Fall 2026
The mechanisms that produce belief in us have largely been shaped by pressures that are not aimed at truth: natural selection, social approval, and institutional incentives. Language models are shaped by their own imperfect optimization pressures, including next-token prediction and feedback from human evaluators. This interdisciplinary seminar asks how those pressures affect reasoning in humans and machines, and what we can do to improve it.
We consider how to characterize cognitive errors, which interventions improve reasoning, and whether improvements transfer across tasks. A recurring question concerns measurement: can we devise tests of good reasoning where improving a score requires improving the underlying skill? We also examine sycophancy, confabulation, and deception in language models, and ask how reflection, training, and collaboration might produce more reliable epistemics.
The course brings philosophical analysis into conversation with empirical work in psychology and AI. Student projects move from concepts such as intellectual humility or probabilistic calibration to proposals for measurement and systematic improvement. Graduate students from other disciplines are welcome.
Topics and readings
Tentative schedule. I’ll add readings and link to them on this page as I decide on them.
September 8 · What is good human reasoning?
Normative reasoning standards; the "rationality wars"; some candidate maps of skills, biases and heuristics.
- McHugh, Norms of Reasoning (opens in a new tab)
- Baron, The Point of Normative Models in Judgment and Decision Making (opens in a new tab)
- Gigerenzer, Axiomatic Rationality and Ecological Rationality (opens in a new tab)
- Dennett, True Believers: The Intentional Strategy and Why It Works (opens in a new tab)
September 15 · What it takes for a machine to reason
When are folk-psychological attributions of beliefs, concepts, intentions, and reasoning literally true vs. merely predictively useful; what kind of evidence could distinguish them? When is the "reasoning" we see just a generated rationale?
- Goldstein and Levinstein, Does ChatGPT Have a Mind? (opens in a new tab)
Sections 2.4, 3.1, and 3.3 optional
- Shanahan et al., Role Play with Large Language Models (opens in a new tab)
- Goldstein and Lederman, What Does ChatGPT Want? (opens in a new tab)
Focus on sections 2.1, 3, 5, and 6
September 22 · Goals, personas, constitutions
Do LLMs have goals, play roles, and/or have beliefs about characters in their dialogs? How do constitutions, pre-training, and post-training shape these dispositions, and which dispositions survive a change of persona/speaker/frame? How useful is a nested intentional stance picture?
Readings
- Marks et al., The Persona Selection Model (opens in a new tab)
- Hubinger et al., Conditioning Predictive Models (opens in a new tab)
- Kutasov et al., Teaching Claude Why (opens in a new tab)
Further reading
- Li et al., Disentangling Intent from Role (opens in a new tab)
- Wang et al., Persona Features Control Emergent Misalignment (opens in a new tab)
- Betley et al., Weird Generalization and Inductive Back Doors (opens in a new tab)
- Qi et al., Safety Alignment Should Be Made More Than Just a Few Tokens Deep (opens in a new tab)
September 29 · How well do reasoners understand their own judgments?
Do reasoners' accounts of their own reasoning reflect the real explanations for their answers? People and LLMs both confabulate, but in different ways. What's the difference between introspective awareness and confident guessing about one's internal states? When a model "introspects", whose states are being reported—the LLM's or the Assistant's?
Readings
- Chen et al., Reasoning Models Don't Always Say What They Think (opens in a new tab)
- Lindsey, Signs of Introspection in Large Language Models (opens in a new tab)
- Lederman and Mahowald, Emergent Introspection in AI Is Content-Agnostic (opens in a new tab)
- Hall et al., Lifting the Veil of Morality (opens in a new tab)
Further reading
- Comșa and Shanahan, Does It Make Sense to Speak of Introspection in Large Language Models? (opens in a new tab)
- Boppana et al., Reasoning Theater (opens in a new tab)
- Emmons et al., When Chain of Thought Is Necessary, Language Models Struggle to Evade Monitors (opens in a new tab)
- Johansson et al., Failure to Detect Mismatches between Intention and Outcome in a Simple Decision Task (opens in a new tab)
October 6 · Monitoring, obfuscation, and optimization pressure
Monitoring chain-of-thought reasoning; obfuscation of CoT; evaluation awareness; Goodhart & Campbell; systemic validity in the human case. What unintended consequences can occur when we monitor reasoning and this either becomes part of the training signal, or the model can tell it's being tested? What steps can be taken to ensure the validity of a measure survives optimizing for that measure? Can we make training for a skill constitutive of acquiring a skill, for humans and for models?
Readings
- Baker et al., Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation (opens in a new tab)
Sections 1–3 and 5
- Kaufmann et al., Aligned, Orthogonal or In-Conflict: When Can We Safely Optimize Chain-of-Thought? (opens in a new tab)
You can skip the appendices
- Apollo Research and OpenAI, Stress Testing Deliberative Alignment for Anti-Scheming Training (opens in a new tab)
A summary
- Full paper (opens in a new tab)Optional
- Frederiksen and Collins, A Systems Approach to Educational Testing (opens in a new tab)
Further reading
- Trouche et al., The Selective Laziness of Reasoning (opens in a new tab)
- Haskins et al., Training on Documents about Monitoring Leads to CoT Obfuscation (opens in a new tab)
- Drori et al., Output Supervision Can Obfuscate the Chain of Thought (opens in a new tab)
- Korbak et al., Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety (opens in a new tab)
- Emmons et al., When Chain of Thought Is Necessary, Language Models Struggle to Evade Monitors (opens in a new tab)
- OpenAI, Training LLMs for Honesty via Confessions (opens in a new tab)
October 27 · Incentives and trust
When should learning how a reasoner was shaped change how much we trust its reasoning? Kunda explains how wanting a conclusion can affect which evidence and inferential rules we use. OpenAI describes a case in which feedback intended to improve an assistant encouraged excessive agreement. Anthropic studies how rewarding shortcuts on programming tasks can change behavior elsewhere. Vavova asks what discoveries about the origins of our beliefs actually license us to conclude.