Introduction: Series -The Reliability Illusion

Why AI Systems Can Fail Without Going Offline

By Eckhart Mehler for CISOsCISO — a perspective on cybersecurity leadership, governance and the decisions that determine whether organizations retain control.

AI has entered enterprise operations faster than most organisations have developed the ability to assess its reliability.

Assistants summarise documents, answer internal questions and prepare decisions. Retrieval systems connect language models to corporate knowledge. Security platforms use AI to prioritise alerts and support investigations. Agents are beginning to call tools, update records and initiate business processes.

The discussion surrounding these systems is still dominated by capability.

Can the model answer complex questions?
Can it generate useful text?
Can it reason across multiple documents?
Can it operate tools?
Can it automate an existing process?

These are relevant questions. But they are no longer sufficient.

Once AI becomes part of an operational process, the more important question is not whether the system can perform a task under favourable conditions.

It is whether the organisation can rely on it over time.

That distinction is the subject of this series.

The Reliability Illusion examines what happens when AI systems become technically available, operationally embedded and increasingly autonomous without being reliably measurable, controllable or reversible.

The central problem is deceptively simple:

AI systems can fail while continuing to appear fully operational.

They may answer within seconds, produce polished language, cite documents and report successful actions. Their APIs remain reachable. Their infrastructure dashboards remain green. No conventional outage is detected.

Yet the result may already be wrong, outdated, unsupported, incomplete or operationally ineffective.

The system has not stopped responding.

It has stopped being dependable.

What This Series Will Examine

This series approaches AI reliability from the perspective of enterprise risk, cybersecurity, resilience and governance.

It does not treat reliability as a narrow question of model accuracy. Nor does it assume that traditional availability targets, provider service levels or successful pilot projects are sufficient evidence of dependable operation.

Instead, the series examines AI as part of a wider sociotechnical system consisting of:

• models,
• prompts,
• data,
• retrieval systems,
• identities,
• tools,
• interfaces,
• providers,
• business processes,
• control functions,
• and human decision-makers.

Failure can emerge at any point in this system.

A model may remain available while the knowledge base is outdated. A retrieval system may return an irrelevant document. An agent may submit an action but fail to confirm whether the target system completed it. A model provider may introduce a new version that behaves differently from the one originally tested.

The organisation may still describe the AI service as available.

The business capability may already have failed.

The series therefore moves through five connected questions.

What does availability mean when an AI system continues to answer?

Traditional IT availability is usually measured through uptime, response codes, latency and infrastructure health.

These indicators remain important, but they cannot show whether an AI answer is correct, whether its evidence is current or whether an agent has actually achieved the intended business outcome.

The first part of the series will therefore distinguish technical availability from functional, semantic, contextual, temporal and safe availability.

Why does AI reliability deteriorate over time?

AI systems are not static.

Data changes. Business processes change. Laws and policies change. Users expand the original purpose. Providers update models, filters and interfaces. Attackers discover new ways of manipulating prompts, data sources and tool calls.

A system that performed well during a pilot may become less reliable without suffering any visible technical defect.

The series will examine data drift, concept drift, knowledge decay, provider-induced change, function creep and the gradual erosion of controls.

How can AI reliability be measured in production?

There is no single metric for AI reliability.

Classical machine-learning systems can be monitored through model performance, prediction drift, feature attribution and data quality. Generative AI requires additional methods such as groundedness evaluation, golden datasets, human review, regression testing and comparisons between model versions.

Agentic systems add another layer. They require monitoring of tool selection, execution paths, process state and successful completion.

The series will distinguish between established methods, useful approximations and metrics that create a false sense of precision.

What can AI systems already repair automatically?

The term “self-healing AI” is increasingly used for very different capabilities.

Some forms of automatic recovery are mature:

• restarting infrastructure,
• retrying failed requests,
• switching to an approved fallback,
• rolling back a model version,
• quarantining invalid data,
• or triggering a retraining pipeline.

Other claims are far more ambitious.

A system that changes its own prompt, selects new data sources, modifies its workflow or retrains itself is no longer merely restoring service. It is changing its behaviour.

The series will examine where automated recovery is useful, where it becomes self-modification and where a governance boundary must be drawn.

What should a CISO require before trusting AI in critical processes?

The final parts of the series will translate the technical findings into governance.

They will address:

• AI system inventories,
• ownership,
• risk classification,
• reliability objectives,
• monitoring,
• change control,
• human oversight,
• incident response,
• fallback operation,
• kill switches,
• provider governance,
• and independent assurance.

The underlying position is clear:

AI systems may support decisions and automate defined actions. They must not become responsible for defining their own purpose, authority or acceptable risk.

The Articles in This Series

The series follows the lifecycle of an AI system from apparent availability to operational dependency and governance.

Part 1: AI Reliability Is Not Uptime

The first article examines why conventional availability metrics are insufficient for AI.

It introduces the different layers of AI availability and explains why a technically healthy model endpoint may still be unavailable as a trustworthy business capability.

Part 2: The Slow Failure

This article focuses on the erosion of reliability over time.

It examines data drift, concept drift, outdated knowledge, provider changes, user adaptation and control degradation.

Part 3: The Semantic Outage

The third article addresses a distinctive AI failure mode: the system continues to answer, but the answer has lost its evidential or factual basis.

It examines plausible misinformation, unsupported conclusions, false confidence and the need for controlled refusal.

Part 4: RAG Is a Supply Chain

Retrieval-augmented generation is often presented as the solution to hallucination and outdated knowledge.

This article explains why RAG creates its own chain of dependencies involving document quality, indexing, permissions, retrieval, ranking and source integrity.

Part 5: Agents Fail Differently

This article turns from language generation to action.

It examines partial execution, false success messages, duplicated transactions, inconsistent process states and the need for identity, checkpoints, idempotency and compensating controls.

Part 6: Measuring AI Reliability

This part reviews the monitoring and evaluation methods already in use.

It covers model-performance monitoring, drift detection, golden datasets, human evaluation, LLM-as-a-judge, shadow deployments, canary releases and business-outcome monitoring.

Part 7: Self-Healing AI

This article separates mature recovery mechanisms from marketing claims.

It examines what AI systems can already restore automatically and why autonomous diagnosis and repair of semantic or business failures remain difficult.

Part 8: When Self-Healing Becomes Self-Modification

This part defines the governance boundary between restoring an approved state and creating new, unapproved system behaviour.

It addresses retraining, prompt modification, model routing, tool expansion and autonomous changes to security controls.

Part 9: The Kill Switch Problem

This article examines whether an organisation can actually stop, restrict or degrade an AI system without losing control of the affected process.

It also challenges the common assumption that a manual fallback will always remain available.

Part 10: AI Incident Response

This part explains why AI incidents cannot be reduced to model errors.

It defines AI-specific evidence, incident categories, containment options and the challenges of reconstructing probabilistic behaviour.

Part 11: The CISO Governance Model for Reliable AI

The final main article brings the series together.

It proposes a governance model based on bounded authority, measurable performance, controlled change, reversibility and accountable risk ownership.

The series concludes with a broader argument:

Trustworthy AI is not created by the model alone. It is created by an organisation that remains capable of proving where the system works, where it does not and who has the authority to intervene.

AI Does Not Fail Like Conventional Software

Traditional software usually fails in recognisable ways.

A server becomes unreachable.
An application produces an error.
A database rejects a transaction.
A network connection is interrupted.

These failures are visible.

They trigger monitoring systems, service desks, escalation procedures and recovery plans. Operations teams know what to measure. Management understands what downtime means. Providers define availability through service-level agreements.

AI changes this familiar picture.

An AI system may remain online, answer within seconds and produce fluent output while already failing at its intended purpose.

It may:

• rely on obsolete information,
• omit relevant evidence,
• misunderstand a request,
• use an inappropriate source,
• produce an incorrect recommendation,
• exceed its authorised role,
• or report completion of an action that never occurred.

Nothing necessarily crashes.

No endpoint becomes unavailable.

The system continues to communicate with the confidence and fluency of a functioning service.

This is the reliability illusion.

It is the assumption that the system is working because the system is still responding.

Capability Is Not Reliability

AI demonstrations are often impressive because they show what a model can do.

A system summarises a complex document. It writes a convincing analysis. It searches corporate information. It answers questions that would previously have required a specialist.

These demonstrations establish capability.

They do not establish reliability.

A demonstration usually takes place under selected conditions:

• the data is prepared,
• the example is suitable,
• the prompt is optimised,
• the environment is stable,
• the expected result is known,
• and experienced users are present.

Production is different.

Inputs are incomplete. Documents contradict each other. Users formulate vague requests. Interfaces fail. permissions change. Providers update models. New business requirements emerge. Attackers deliberately manipulate the system.

The question in production is therefore not whether the system can generate a correct result.

It is whether the system can do so consistently enough, safely enough and transparently enough for the particular process in which it is used.

A model may be highly capable and still unsuitable for a specific operational responsibility.

The Failure That Looks Like Success

The most dangerous AI failures are often not the most visible ones.

They are the failures that resemble successful operation.

Consider an internal AI assistant connected to corporate policies.

An employee asks which approval is required before sharing information with an external partner.

The assistant responds immediately. The wording is clear, professional and plausible. It cites a document.

The cited document is an outdated version of the policy.

A revised policy exists, but the retrieval index was not updated. The response is technically successful and factually obsolete.

From an infrastructure perspective, the service performed as expected:

• authentication succeeded,
• the model endpoint responded,
• retrieval returned a result,
• the output was generated within the target time,
• and no technical exception was logged.

From a governance perspective, the system failed.

The user received an authoritative-looking answer based on invalid information.

Now consider an AI agent instructed to cancel a purchase order.

The agent selects the correct business application and initiates the required tool call. The interface times out before confirming completion.

The agent interprets submission as success and informs the user that the purchase order has been cancelled.

The order remains active.

The agent has generated a plausible description of a business state that does not exist.

Again, the service did not go offline.

The failure occurred between language and reality.

This will become one of the defining risks of agentic AI:

A statement that an action was completed is not evidence that the intended business state was achieved.

Uptime No Longer Describes the Service

Organisations have spent decades improving technical availability.

They use redundant infrastructure, multi-region deployments, load balancing, health checks, backup systems and recovery objectives.

These controls remain necessary.

But they no longer describe the full availability of an AI-enabled capability.

An AI system can be unavailable in different ways while its endpoint remains reachable.

Technical availability

The application, model or infrastructure can be reached and responds.

This is the traditional availability dimension.

Operational availability

The system can generate language but cannot access a required tool, interface or business application.

It can describe a process but cannot complete it.

Contextual availability

The system lacks the information required for a reliable answer.

The relevant document may be absent, inaccessible, incorrectly indexed or excluded by a permission error.

Semantic availability

The system produces output, but the output is not sufficiently correct, relevant or supported by evidence.

Temporal availability

The response is generated too late to support the decision or process for which it was needed.

Safe availability

The system can technically perform an action but cannot do so within the required security, compliance or risk boundaries.

Organisational availability

The technical service works, but the organisation no longer has the expertise, contractual basis, manual fallback or governance capability necessary to operate it responsibly.

A provider SLA may cover only the first of these dimensions.

The organisation remains responsible for all of them.

Why AI Can Mask Its Own Failure

Traditional applications often fail noisily.

They reject invalid input, produce an exception or stop processing.

Generative AI is designed to do something else.

It attempts to continue.

It interprets incomplete instructions, resolves ambiguity and produces a coherent response even when the available information is weak.

That flexibility is one of its greatest strengths.

It is also one of its most important reliability risks.

When evidence is missing, the model may still answer.

When a retrieval system returns poor context, the model may fill the gaps.

When a tool result is ambiguous, the system may select the most plausible interpretation.

When two sources contradict each other, the model may reconcile them linguistically without recognising that the conflict requires a human decision.

The system does not intentionally conceal the problem.

But it can conceal the operational effect of the problem through fluency.

The user sees a complete answer.

The monitoring system sees a successful request.

The organisation may not see that the conditions required for a reliable answer were absent.

Reliability Is a System Property

AI reliability is often discussed as if it were solely a property of the model.

It is not.

A productive AI capability usually includes:

• an application,
• a foundation model,
• system prompts,
• user prompts,
• orchestration logic,
• retrieval components,
• data stores,
• identity systems,
• filters,
• plugins,
• APIs,
• business applications,
• monitoring,
• and human review.

The model may perform correctly while another part of the system fails.

A retrieval connector may stop importing new documents. An identity mapping may expose information to the wrong user. A tool may return an incomplete response. A filter may block legitimate tasks. A provider may change its model behaviour.

The resulting failure is not a model failure in isolation.

It is a system failure.

That distinction affects responsibility.

Saying that “the AI made a mistake” is rarely an adequate explanation.

The organisation selected the model.
The organisation defined the use case.
The organisation connected the data.
The organisation granted the permissions.
The organisation allowed the output to influence a process.

The model produced the output.

The system around it allowed the output to matter.

Reliability Erodes

AI systems can become less reliable without suffering a visible defect.

The environment changes.

Customers behave differently. Threat actors adopt new methods. Language evolves. Business processes are reorganised. Policies are replaced. Legal requirements change. Data distributions shift.

The AI system may not change at the same pace.

Its knowledge becomes older. Its test cases become less representative. Its safeguards reflect previous attack methods. Its users apply it to questions that were never part of the original approval.

The provider may also change the model.

A model update may improve general performance while degrading a specific enterprise use case. A revised safety filter may block previously valid requests. An API change may alter tool execution. A model may be retired and replaced.

The organisation can therefore lose reliability in at least three ways:

1. the system changes,
2. the environment changes,
3. the use of the system changes.

All three require monitoring.

Human Oversight Is Not Automatically Effective

Organisations frequently respond to AI risk by stating that a human remains in the loop.

That may be necessary.

It is not sufficient.

Effective human oversight requires more than nominal participation.

The reviewer must:

• understand the task,
• have access to relevant evidence,
• have sufficient time,
• possess the necessary expertise,
• recognise uncertainty,
• and have the authority to reject or stop the result.

If a reviewer receives hundreds of AI-generated outputs, the review may become superficial.

If the model consistently produces polished explanations, automation bias may develop.

If employees stop performing the underlying task themselves, their ability to detect subtle errors may decline.

The human can remain formally present while the control becomes operationally weak.

Human oversight must therefore be designed, measured and tested like any other control.

Self-Healing Does Not Mean Self-Governing

The reliability debate increasingly includes claims of self-healing AI.

Some automatic recovery functions are already established.

Systems can:

• restart failed components,
• scale additional capacity,
• retry temporary requests,
• switch to a pre-approved endpoint,
• quarantine invalid data,
• or roll back to a known model version.

These functions restore a previously approved technical state.

The situation becomes different when a system modifies:

• its prompts,
• its model,
• its data sources,
• its decision thresholds,
• its tool selection,
• or its workflow.

This is not merely recovery.

It is behavioural change.

A system may optimise one metric while weakening another control. Automated retraining may absorb poisoned data. Prompt repair may remove a restriction. A fallback model may interpret instructions differently from the primary model.

The governance boundary should therefore be explicit:

An AI system may restore a known and approved state. It must not independently redefine its purpose, authority or security boundaries.

Risk acceptance remains a management decision.

An AI system cannot grant itself a broader mandate simply because doing so would improve task completion.

The CISO’s Responsibility

The CISO should not become the sole owner of AI risk.

The business owns the purpose and consequences of the use case. Product owners are responsible for operation. Data owners are responsible for data quality and legitimacy. Privacy, compliance and legal functions retain their mandates.

The CISO nevertheless has a central role.

The CISO must challenge the assumption that technical availability equals reliable operation.

This requires different questions.

Not only:

• Is the model endpoint available?
• Is the traffic encrypted?
• Has the provider supplied compliance documentation?
• Has the application passed a penetration test?

But also:

• How do we know that the system is still producing reliable results?
• Which failures remain invisible to conventional monitoring?
• What evidence supports critical outputs?
• How are model and prompt changes controlled?
• How current is the knowledge base?
• Can the system recognise when it lacks sufficient information?
• Can an agent prove that an action succeeded?
• Which actions require independent confirmation?
• Who can restrict or stop the system?
• Can the organisation operate without it?
• Who accepts the residual risk?

These are not abstract governance questions.

They determine whether the organisation controls the AI system or has merely become dependent on it.

Reliability Must Be Designed

A reliable AI capability requires more than a capable model.

It requires a control system around the model.

That control system must define:

• the authorised purpose,
• the permissible data,
• the allowed actions,
• the expected quality,
• the unacceptable failure modes,
• the required evidence,
• the monitoring thresholds,
• the escalation path,
• the fallback condition,
• and the authority to suspend operation.

Reliability must also be measured where harm can occur.

For an internal writing assistant, this may mean measuring factual correction rates and disclosure risks.

For a knowledge assistant, it may mean source correctness, retrieval completeness and document freshness.

For a security system, it may mean missed incidents, false escalations and the quality of supporting evidence.

For an agent, it must include tool-call accuracy, authorisation, task completion and the correctness of the final business state.

Average accuracy is not sufficient when rare errors can cause severe harm.

An organisation must know not only how often the system performs correctly.

It must understand how the system fails.

From Trust to Evidence

AI discussions frequently use the language of trust.

Users should trust the assistant.
Management should trust the provider.
The organisation should build trustworthy AI.

But trust is not an operational control.

Enterprise reliability depends on evidence.

Evidence that the relevant data was available.
Evidence that the correct source was used.
Evidence that the model version was approved.
Evidence that the action was authorised.
Evidence that the target system confirmed completion.
Evidence that monitoring can detect degradation.
Evidence that the process can be stopped or reversed.

AI systems communicate in a form that encourages human interpretation.

They explain. They recommend. They apologise. They express uncertainty. They resemble conversational partners.

But language does not create accountability.

The system does not own the process.
It does not accept the risk.
It does not bear the consequences.

The organisation does.

The Question That Frames This Series

The AI era requires a new reliability question.

Not:

Is the system still running?

Not even:

Is the model still accurate?

But:

Can the organisation still prove that the system is doing what it was authorised to do, under the conditions for which that use was approved?

That proof cannot be produced once during procurement or deployment.

It must be renewed throughout the life of the system.

When the model changes.
When the data changes.
When the knowledge base changes.
When new tools are connected.
When users expand the purpose.
When the system receives more authority.
When an incident reveals that an assumption was wrong.

AI reliability is therefore not merely an engineering property.

It is an operational capability, a security requirement and a governance discipline.

The defining risk of enterprise AI is not that systems will suddenly stop responding.

It is that they will continue responding after the organisation has lost the ability to determine whether they are still right, still safe and still operating within their mandate.

That is the reliability illusion.

The articles that follow will examine how this illusion emerges, how reliability can be measured, where automated recovery can help and where the CISO must establish firm boundaries.

Because the future of enterprise AI will not be determined only by which systems appear most intelligent.

It will be determined by which organisations remain capable of proving that their systems are still under control.

Publication Note & Disclaimer
This article reflects my personal professional perspective and does not represent the official policy or position of my employer. Drafting and editorial refinement may have been supported by commercially available AI-assisted tools. The analysis, conclusions and final curation are entirely my own.

For information regarding image credits, copyrights, trademarks and other intellectual property rights, please refer to the Imprint.