Compiler Error Messages and Feedback Lift GPT-5 from 39 percent to 96 percent in Idris, USC Study

AI models typically pick up programming languages by ingesting vast quantities of examples. The challenge is obvious when the language is uncommon and there are very few code samples to learn from.

New research indicates that the quality of feedback can outweigh the amount of data. The researchers showed that a language model could be steered into producing correct code in Idris - a niche programming language it had scarcely seen previously.

The key was an iterative loop: the team repeatedly returned compiler error messages to the model and instructed it to repair its own output. Using this approach, they lifted performance from 39 percent to 96 percent.

Taken together, the findings imply that, when an AI system receives precise signals about what failed, it can move well beyond what its original training alone would suggest.

Where the system stumbled

Using a set of Idris coding exercises hosted on Exercism, the researchers could quickly see the mismatch between the model’s initial answers and Idris’s strict requirements.

At the University of Southern California (USC) Viterbi School of Engineering, Minda Li captured every compiler error and passed each message back to the model so it could revise its solution.

With each correction cycle, the model had to grapple with the specific rule it had violated, gradually converting non-compiling attempts into functioning programs.

This repeated pattern pointed to a broader takeaway: progress relied less on adding fresh training data and more on delivering clearer, more targeted information about exactly what went wrong.

Picking a difficult language

Bhaskar Krishnamachari, a professor of electrical engineering and computer science at USC, deliberately chose a language obscure enough to put the model under pressure.

Idris suited that purpose because it is dependently typed - its type system can encode additional constraints, and those constraints penalise imprecise code early in the process.

The study reported roughly 2,275 public Idris repositories, versus more than 24 million for Python, leaving dramatically less material available for training.

Neither of the researchers had prior knowledge of Idris, keeping the focus on whether the system could leverage feedback rather than being guided through the language.

The model hits a wall

On the same set of exercises, GPT-5 completed 90 percent in Python and 74 percent in Erlang, yet managed only 39 percent in Idris. The contrast matters: the model was already a capable coder when working in widely used, more forgiving languages.

In Idris, errors such as undefined names, missing pattern-match branches, and type mismatches were surfaced before the underlying logic could even be evaluated. As a result, poor Idris performance reflected getting stuck on rigid local constraints more than any lack of general capability.

Li initially experimented with softer forms of support, including feedback from the platform, an error guide she wrote herself, and the official Idris documentation. These helped somewhat, but overall Idris performance still failed to rise above 61 percent.

General guidance could reinforce syntax and common rules, but it often did not reveal why a particular submission failed in that exact case. That plateau underscored a tougher point: broad advice was less effective than immediate, instance-specific correction.

AI fixed mistakes with feedback

The successful method relied on local checking: a compiler - software that verifies and translates code - can identify the precise line that breaks the rules. Li returned those compiler error messages to GPT-5, requested a corrected version, and ran the cycle again, repeating up to 20 times.

Before trying the full loop, Li anticipated only a small gain. Instead, the success rate surged to 96 percent.

“I was surprised that just that alone, seemingly one simple thing, just keep recompiling, keep trying, was able to get to 96 percent,” said Li.

The explanation was straightforward: compiler output pinpoints the specific fault in each program. Under the weaker baseline approach, missing names showed up 123 times, and Idris frequently could not infer which meaning the code was intended to express.

A manual can highlight typical hazards, but only the compiler can state what is wrong in that particular program, at that particular moment.

“Our AI tools are now able to transcend their initial training,” said Krishnamachari.

Beyond software alone

Highly specific feedback of this kind is not unique to programming. In areas such as mathematical proofs, legal reasoning, and other work governed by strict rules, it is often possible to clearly identify when a step is invalid.

In those contexts, the same correction loop could let an AI system iterate on its own output before a person ever reads the first draft.

The technique could also be relevant to machine translation for low-resource human languages, where a shortage of written examples has long made dependable training difficult.

For the approach to succeed, however, the feedback has to be unambiguous, correct, and consistently tied to the exact mistake each time.

Not every problem was solved

The method is not flawless. Even after 20 correction rounds, two Idris tasks still could not be solved, suggesting that repeated patching can sometimes introduce fresh errors.

The researchers also caution that some Idris snippets or patterns may already have been present somewhere in the model’s training data.

If so, some of the improvement might reflect recall rather than learning driven purely by feedback.

To measure genuine learning more cleanly, future evaluations would need entirely new problems written after the model’s training finished.

Teaching AI to remember

Li’s next goal is to make the system retain what it learns from one exercise so it can apply those lessons to the next, rather than beginning from scratch each time. As things stand, the process often advances through trial and error even when the same class of mistake recurs.

A model that carries forward earlier fixes could converge in fewer attempts and consume less compute by avoiding repeated missteps. That change would transform a neat debugging loop into something more like sustained skill acquisition.

More broadly, the work suggests that the biggest constraint may not be missing data so much as the lack of sharp, dependable correction. Better feedback will not remove every hard limit, but it may make it far easier for AI to work with smaller languages and more restrictive, rule-bound domains.