Telling reasoning apart from pattern-matching that resembles it
A system can arrive at the right answer for reasons that will not hold the next time the problem shifts. This planned paper is about separating genuine reasoning from surface pattern-matching that happens to look the same on a familiar test.
It is easy to be convinced a system is reasoning when it is really recognising. On problems close to what it has seen before, pattern-matching and reasoning produce the same output, and a test built from familiar problems cannot tell them apart. The distinction only shows up when the problem is moved somewhere the pattern does not reach.
This matters because the two behave differently when conditions change. A system that reasoned will tend to hold up under a shift it has not seen; a system that matched a pattern will tend to fall over quietly, and look confident while doing it. We want evaluations that make that difference visible before it costs someone in production.
What we intend to measure
- How does behaviour change when a problem is moved just outside the distribution the system was trained and tested on?
- Can we construct variations that keep the reasoning identical while changing surface features, and see which the system was actually tracking?
- What signals, if any, distinguish a robust answer from a lucky one at the moment it is produced, rather than in hindsight?
Open questions
- Is there a way to characterise the edge of a system's reasoning without an expensive hand-built probe for every task?
- How do we keep such a test honest as systems are optimised against the very properties we are measuring?