Designing Reliable A.I. Systems

LLMs Are Inherently Unreliable — Here’s Why

… And How To Build For Resilience Anyway

Ron Diamond (dev)

10 min readMar 7, 2026

Large language models (LLMs) — now used in Artificial Intelligence chatbots — are remarkable tools.

But they should not be trusted to enforce critical rules or security boundaries. We’ll explore why.

It’s been fun experimenting with LLMs over the past few years, to see what they’re capable of doing.

Though recently, I had a very interesting chat with ChatGPT.

The specific prompt isn’t important to the point here, so I’ll spare you the details.

But it boiled down to this:

I gave it an explicit command NOT to use a certain piece of information in its responses.¹

This was a very binary instruction: simple, and absolutely clear.

But when I tested it, to make sure it’d comply … it went ahead and included that info anyway.

This seemed a little weird, so I tried other prompt variations … none of which worked. It kept screwing up. And then eventually, I fell back to a technique that I’d already found useful:

I asked the LLM to tell me what prompt to use, going forward, so it would behave the way I wanted.

But even that didn’t work! And so, I asked for variations on that prompt … over and over again.

And each time, it promised that the new version of the prompt would surely do the trick — but instead, it kept doing the very behavior I was asking it not to do.

And that really became annoying. 😬

All told, this clown car of a conversation went on and on, back and forth, for a full hour.

Until finally …

HAL Lives!

I was suddenly reminded of HAL 9000, from (my all-time favorite film) 2001: A Space Odyssey.

In the film [Spoiler Alert!] … HAL suffers a kind of psychotic breakdown, and sabotages the mission, big-time …

And though it isn’t really explained until the (not quite as epic) sequel 2010: The Year We Make Contact, the reason turned out to be … that HAL had actually been given fundamentally conflicting instructions — the equivalent of two very different System Prompts — and then did its best to reconcile those instructions … with disastrous results.

And my experience reminded me of that — though for different reasons.

HAL’s outward behavior became unpredictable for the crew.

Modern LLMs can sometimes surprise us, too. Not because they suffer from HAL’s fictional malfunction … but because complex systems might behave in ways that users don’t expect.

And in the process, it suddenly struck me: that we’re now leaving our first era of computing … and entering a new one …

… A brave new world, operating under a very different set of rules.

People aren’t strictly “logical” … and so now, in a sense, neither are our machines.

“The Times, They Are a’ Changin’ ”

Since the beginning of modern computing, some 75+ years ago, these machines have operated on — what ultimately boils down to — simple Boolean logic.

It’s been manifested in tubes & transistors & integrated circuits … which represent a series of “On” and “Off” states … which are then incorporated into the machine’s outputs.

And even over the decades, with many added layers of hardware and software abstraction, practically anything a computer did … could still be traced back to simple binary logic, in the form of millions or billions of discrete transistors. Even when etched microscopically onto a chip.

Photo by Pdesousa359; via Wikipedia, CC BY-SA 3.0

But this new era is different.

Of course, computers still operate mechanistically, under the hood.

But our new LLMs do not appear to be so deterministic.

Because instead of explicit Off/On logic rules, the neural network in an LLM consists of millions or billions of floating point variables [the “model weights”] — effectively, analog values.

And all the zillions of permutations of those values are about as close to seeming-infinity as you’re gonna see in a computer. Kind of like the dendritic connections in a human brain.

Which, when you tally them up, effectively turns the whole thing into a big ball of fuzzy-like logic.²

But you say:
That logic is still deterministic, just much more complicated. And technically, that’s true.

Though in practice …

First: LLMs usually have a parameter for “Temperature” [i.e., randomness].

But even if we turn that randomness parameter down to zero … the results can still be unpredictable.

Why? Because the context in which they do all those zillions of computations … changes.

Even over the course of a single chat with an LLM, its “context window” [the short-term memory of the chat so far] changes with every back & forth exchange.

And the text [tokens] in that context window literally become part of the input for its next response [token generation] — thus creating a feedback loop, which steadily changes the state of the LLM.

And so, as the conversation progresses, that history keeps getting appended.

And even though the LLM itself behaves like a pure, stateless function … the values of its input argument(s) are constantly changing.

From the user’s perspective, that makes its outputs less predictable.

“Can’t Trust — Can’t Verify”

OK, so an LLM’s output is more variable than we might have realized.³

So, for incorporating an LLM into critical systems, what about the old adage: “Trust, But Verify”?

Well, we’ve already looked at why an LLM may give inconsistent results over time (and context).

And if you’ve ever experienced an LLM’s “hallucination” … and then seen it backtrack and correct itself … you’ve already had a sense of this.

So we can’t really trust an LLM to give us what appear to be a consistent set of outputs.

No worries, you say: let’s create a really good automated test suite that verifies the LLM’s behavior across a wide range of scenarios. If it’s comprehensive enough, we can verify the results.

Not so fast!

Remember what we just said about that context window?

How do we test for all those possibilities?

Practical answer: we can’t.

“To Infinity — And Beyond!”

That long string of text is a data structure with a mind-bendingly large number of possible states. (And that’s only one variable!)

And suppose we somehow did (what we consider) extensive testing, to try to ensure that the LLM (with some carefully-crafted system prompt) behaves consistently, according to the rules we give it.

We test it 100 times, and it passes every time. Great! 🥳

But we haven’t covered every permutation, if only because of the variability of that context window. That contributes to an effectively infinite number of permutations — which we can never fully test (alright, maybe before the heat death of the universe⁴).

So even if we test a scenario 1,000 times, and the results are consistently what we expect, and we think the problem is solved … the 1,001st time could still be different.

And that’s barely scratching the surface of those possibilities. 🙄

How does this play out in practice?

Recently, I had a chance conversation with an executive from a large Silicon Valley tech company.

He described how they were beginning to use an LLM as a natural-language querying system for a large internal database, accessible to a variety of personnel.

Those personnel have different levels of access, though, depending on their roles.

And so, the team had crafted a system prompt to explain to the LLM which types of people get what levels of access to which data.

See the problem?

As we’ve discussed, LLMs are reminiscent of fuzzy logic machines. And from the user’s perspective, they’re essentially non-deterministic.

So it doesn’t matter how carefully you craft your system prompt, or how extensively you test those scenarios.

At some point, they just … may not behave the way you expect. No matter how clear your instructions.

Because, with all that variability under the hood — to us humans — an LLM’s behavior can be difficult to predict.

Much like HAL 9000’s.

And depending on the use case — especially for mission-critical applications — that could be bad.

So what do you do instead?

First …

Be careful about trusting your A.I. with anything important.

Hopefully, we’ve begun to make a case for that here.

Many people assume that system prompts function as configuration settings or program instructions. But as we’ve alluded to above …

No amount of prompt engineering will assure the behavior you want, consistently and reliably.

Ask me, I know. 🤓

Instead …

Think of system prompts less like rules for the LLM, and more like … “suggestions.”

They really just skew the model’s probabilities toward the stated behavior, but do NOT assure that.⁵

So: welcome to that brave new world I mentioned.

And if prompts are only suggestions, then something else must enforce the rules.

That’s not as strange as it sounds.

Because LLMs are probabilistic models, not logic gates that enforce hard limits.

The most reliable systems combine both:
➥ Probabilistic models for interpretation;
➥ Deterministic logic for enforcement.

With that in mind, here’s how that enterprise example I mentioned looks, at a high level:

This might work as expected … or it might **not**.

So if you need more definitive restrictions you can actually rely on — put some good old-fashioned, honest-to-goodness Boolean logic in place:

Why? It’s predictable, deterministic … and testable.

In part, because that added component functions as a vastly simpler state machine.

It’s also the Principle of Least Privilege.

And that principle still applies, perhaps more now than ever.

After all, even an A.I. can’t inadvertently reveal information it doesn’t have access to. (Or misfire a system it doesn’t control.)

Very high-profile organizations have come to similar conclusions, especially when it comes to critical systems:

The United Nations,
The Pentagon,
The White House,
and even OpenAI itself.

So it may be worth keeping in mind. 🙌

P.S. There’s already more specific (and thought-provoking) research on some of the ways A.I.’s can in fact behave unpredictably, when we least expect it.

P.P.S. Oh, and back to “2010: The Year We Make Contact” [video, above] …
Seems no one told Roy Scheider that, for “reasons of the highest importance,” um, … Heywood Floyd actually did know about telling HAL to lie: