Putting gen AI confidence scores in context

Large language models (LLMs) are increasingly regarded as something closer to a natural phenomenon or a living organism than a piece of software. They have even been compared to aliens.

 

Systems such as GPT-4 are not developed in the manner of predictable symbolic AI, carefully programmed by humans with rules, facts and tidy logic structures: less built in the traditional engineering sense than “grown”.

 

To understand the workings of LLMs – what routes inputs take, how specific parameters influence outputs, what constellation of weights nudges a model toward one decision rather than another – researchers had to invent entirely new methodologies.

 

The drive to look inside is not mere scientific curiosity. Gaining visibility into how LLMs function is essential to making them more reliable and trustworthy and, by extension, to accelerating their adoption.

 

Interpretability, often described as the holy grail of LLM transparency, remains elusive despite becoming  a research field in its own right, as well as a subject taught at universities under mechanistic interpretability.

 

Some researchers even doubt that LLMs will ever be fully understood by humans thanks to their enormous scale and the billions upon billions of parameters interacting to deliver an answer to a single query.

 

While interpretability remains beyond reach and explainability, relying on approximations and simplification is still largely confined to research labs – and the closest businesses can come today to peering under the bonnet of a generative AI system is through observability.

 

Supported by the telemetry capabilities of observability tools, they can see application and infrastructure behaviour in real time. By keeping an eye on the dashboard and acting on surfacing anomalies, they can ensure that no gen AI model quietly goes astray, hallucinates excessively, turns out toxic content, produces high latency output or drifts from its original performance baseline.

 

Observability dashboards provide a comprehensive view of an LLM’s performance, augmenting traditional core performance indicators already known from ML monitoring with metrics specific to gen AI. These dashboards don’t explain how the model thinks, but they can reveal how it behaves.

 

Confidence scores: AI marking its own homework

 

With the rise of gen AI, observability tools increasingly offer confidence score capabilities.

 

In classical machine learning, confidence scores feel intuitive. A classifier outputs not only a prediction but also the likelihood associated with it. Probabilities are embedded in the architecture of such systems, whereas LLMs, as next-token predictors, continuously assess which of the tens of thousands of tokens is most likely to follow in the sequence that forms a response. To produce an overall confidence metric, the raw probability scores of individual tokens must be aggregated and then converted into percentages.

 

Mathematically complicated as they may sound, these calculations can be performed with modern software in a fraction of a second.

 

But the deeper issue is not computational but conceptual. To what extent can an LLM’s self-assessed confidence score be trusted? Can a tool designed to detect anomalies in a model’s operation rely exclusively on the evaluation provided by that very model?

 

The dilemma is far from hypothetical. Research has shown that some reasoning-focused generative models, when asked to articulate their train of thought, can fabricate plausible but fictitious reasoning processes. If a model can invent a convincing explanation of how it arrived at an answer, what confidence should we place in the sincerity of its expressed certainty?

In an article on the hidden perils that overconfident AI systems pose, technologist Andre Jay goes as far as to compare the reliability of raw probability scores to forecasting weather from a Magic 8 Ball.

 

He is not the only expert maintaining that LLM probability scores can’t serve as a meaningful metric unless they are calibrated to ensure that a model’s predicted probability matches its actual accuracy. If well calibrated, a model’s 90 per cent confidence score guarantees that it is in fact correct 90 per cent of the time.

 

The technical methods for achieving this – temperature scaling, reliability diagrams and isotonic regression – are numerous and complex. But developers are not left unaided thanks to a growing range of tools and libraries that now exists to support calibration efforts.

 

Identifying when a human must stay in the loop

 

Confidence scores with the above caveats firmly in place are also key to gauging the right level of automation for specific workflows. Drawing the line between predictions that can run fully automated and those requiring a human in the loop is critical to the economics of AI deployments.

 

As generative models are probabilistic rather than rules-based, the wheat – the benefits they produce – always come with some chaff: false positives and false negatives.

 

What businesses must decide based on the individual use-case and their risk appetite is where the threshold should be set: whether they should prioritise certainty and high precision even if automation rates drop; or embrace higher automation while accepting that more errors may slip through.

 

In fraud detection, for example, a low confidence threshold would allow the system to act even when relatively unsure, increasing automation. But false negatives that evade detection may lead to fines and reputational damage. So unsurprisingly, fraud detection tends to be a high-confidence, high-precision use case, where less-certain predictions trigger human review.

 

Meanwhile, in defect detection, a low confidence threshold means that any defect, however slight, is flagged up and, therefore, can’t slip through an automated system. Here, the cost of over-flagging may be far lower than the cost of a missed defect.

 

A similarly low threshold in an email moderation system may capture more spam but also generate an avalanche of false positives. Newsletters, community updates and legitimate communications may end up, annoyingly, in the spam folder, compromising user experience.

 

Establishing a boundary between automation and human review is always a matter of trade-offs. The question is whether precision and safety or speed and a frictionless user experience is the top priority – even at the cost of occasional false negatives.

 

These trade-offs vary and must be considered separately for every use case. In the context of self-driving cars, for example, a low threshold allows the system to brake even when the object detection system is only vaguely sure that circumstances necessitate it.

 

But opting for unjustified braking rather than high confidence thresholds, where the car only brakes when it’s 95 per cent sure that what it senses is brake-worthy – is a no-brainer. Overcautius braking is much less consequential than a pedestrian being hit.

 

Context as a make-or-break factor

 

As with so many aspects of gen AI performance, context determines whether confidence scores eventually prove insightful or deceptive.

 

If an LLM consistently signals low confidence in its token choices – particularly when its self-doubt is paired with an authoritative tone – the discrepancy is an obvious red flag calling for immediate attention and troubleshooting.

 

But those who set the confidence thresholds of their automated systems without aligning self-reported confidence with empirical accuracy may let the model get away with its hallucinations and drifts until realty sends a costly wake-up call.

 

Meanwhile, users adopting the motto “trust but verify” will benefit the most from these tracking tools.

 

Taken with a pinch of salt and cross-referenced with other observability metrics, gen AI’s self-reported confidence can finally offer a nuanced glimpse into an awe-inspiring system whose genesis and operation are steeped in obscurity. User confidence in these models won’t emerge from mystique but must be earned through calibrated probabilities, carefully chosen thresholds and a conscientious weighing of trade-offs.