AI Incidents Need a Rehearsed Response

A convincing AI demo can leave a surprising number of decisions untouched. Who can use it? Which records may it read? Who accepts an incorrect action? Who maintains the integration after the pilot team moves on?

Those gaps are where I would start an AI architecture engagement. Sit with the people doing the work. Follow a case from arrival to completion. Find the handoffs, exceptions and decisions that the demo never encountered.

Give someone responsibility for the connections

The AI Solutions Architect should connect the business process, data flows, model harness, security boundaries and operating model. The role needs enough technical depth to challenge a design and enough business understanding to explain its tradeoffs.

That person does not become the owner of every risk. The process owner still defines an acceptable outcome. Security, privacy and legal specialists retain their responsibilities. Engineering owns implementation and operation. The architect makes the dependencies and unresolved decisions visible, then brings the right people together to resolve them.

In a smaller company, this can be an explicit responsibility within an existing role or an embedded external engagement. A new title alone will not unblock delivery.

Make the first deliverable a decision map

I would begin with one page: the intended outcome, users, data sources, permitted actions, approval points, success measures and named owners. Add the conditions under which the system must stop or hand work back to a person.

Consider an illustrative support workflow. Finding an approved answer, drafting a reply and changing a customer’s account are different capabilities. They need different evidence and permissions. Treating them as one vague instruction to “resolve the issue” hides the important design decisions.

NIST’s voluntary AI Risk Management Framework provides a useful foundation for considering trustworthiness across design, development, use and evaluation. Read the NIST framework.

Test the ownership before scaling

Ask three people independently who can approve a new tool, change the acceptance threshold and pause the workflow. Conflicting answers reveal work to do before expansion.

My approach to embedded AI delivery is to connect these decisions with a working implementation. The result should be a system the business can understand, operate and improve.

Start this week by selecting one pilot and naming an accountable owner for every decision it can make.

A five-year AI budget should explain what the organisation intends to become capable of doing. It should also make clear which assumptions are still uncertain.

Committing to today’s model mix, usage pattern and supplier pricing for five years would create false precision. I would keep the strategic direction longer term and review the delivery portfolio and operating assumptions much more frequently.

Budget for the complete service

The model bill is one component. A useful plan also includes discovery, data preparation, integration, evaluation, security, staff training, human review, support and eventual migration or retirement.

Separate initial delivery costs from recurring operation. Then give each use case an owner and an allocation method for shared infrastructure. This helps finance distinguish a successful service growing with demand from an inefficient workflow generating avoidable retries.

FinOps guidance makes a useful distinction between resource measures, such as token cost, and business measures, such as cost per case resolved. Both can help explain spending. Read the FinOps unit economics guidance.

Use scenarios and staged commitments

For the first year, fund a small portfolio with explicit evidence gates. Establish a baseline, test a bounded workflow and expand only when quality, operational readiness and value justify it.

For years two and three, plan options for shared capabilities: identity, approved integrations, evaluation infrastructure and team enablement. For years four and five, state the business capabilities you want and the assumptions that would change the investment. These are planning horizons, not a prediction of how fast every company will mature.

Build conservative, expected and high-demand scenarios. Change several variables: task volume, model choice, review time, failure rate and integration maintenance. A lower inference price may be offset by more usage or more expensive review.

Give saved capacity a destination

Time saved becomes useful when the organisation decides what to do with it. Faster turnaround, better service and reduced backlogs can all matter. They should be measured separately from cash savings.

Do not count the same hour twice across overlapping automations. Do not assume every released hour reduces payroll. Include the time spent checking and correcting the result.

At the next budget meeting, ask for a one-page investment case per workflow: baseline, full operating cost, expected benefit, uncertainty range and a date for the next decision. That is a budget leadership can steer.

An existing technology strategy can remain directionally sound while some of its delivery assumptions become outdated. AI may change how a task is performed, who reviews it and which integrations are worth building.

The useful response is a targeted review of the work. Look for assumptions that can now be tested more cheaply or challenged by a different architecture. Keep the business outcome visible throughout.

Find the constraint in the real workflow

Start at the desk where the delay occurs. Ask someone to show the last difficult case, including the documents, systems and decisions involved. A polished process diagram rarely captures every workaround.

AI may help interpret unstructured material or prepare an initial draft. It cannot resolve missing authority, unclear policy or unavailable data simply by generating an answer. Sometimes the best first change is a cleaner handoff, a better interface or ordinary automation.

For each candidate, record the current constraint and the evidence that an AI component would improve it. This prevents a strategy refresh from becoming a list of tools to purchase.

Reassess the choices that became cheaper to test

An illustrative internal service might move from manual triage to assisted classification, with uncertain cases still routed to a person. The experiment should measure end-to-end handling time, misrouting, review effort and downstream impact.

Keep a simple baseline in the comparison. A rules-based route may be sufficient for common cases, while a model handles a narrower ambiguous subset. Architectural choice should follow the work and its consequences.

I would also revisit build-versus-buy assumptions. Include configuration, integration, governance and switching effort in both options. A rapid prototype is useful evidence about feasibility; it is not the full operating cost.

Refresh the decision rhythm

Give the strategy a quarterly review: which assumptions changed, which pilots produced evidence, which services are ready to expand, and which should stop? Preserve the longer-term purpose while allowing delivery choices to change.

FinOps describes technology value as a shared responsibility across engineering, finance and business teams. That collaboration is equally useful when deciding where AI belongs in a transformation portfolio. Read the FinOps framework.

My role in embedded delivery is to turn a broad ambition into a practical experiment, then connect the result to architecture and adoption decisions.

Choose one expensive handoff this week. Observe it, establish a baseline and define the smallest credible test.

A team can become fluent at prompting while remaining uncertain about the decisions that matter most. Can this document be uploaded? Is the answer supported? May the agent send the message? What should happen when the result looks plausible but feels wrong?

I would build an AI training programme around those moments. Good judgement needs realistic practice, approved tools and an accessible route to help.

Teach the work people actually do

Give a sales team a proposal with an unsupported claim. Give an engineering team a generated change with a subtle permission error. Give an operations team a document containing instructions that conflict with the approved workflow.

Use synthetic or appropriately sanitised examples. Ask participants to identify the evidence, the data boundary and the action they are authorised to take. Let them practise stopping and escalating, as well as completing the task.

Training should reflect each role’s access and consequences. A person drafting internal notes needs different preparation from someone approving external communications or granting an agent access to production systems.

Make safe practice possible

Publish a short guide alongside the approved tool: allowed data, prohibited uses, verification steps, escalation contact and incident reporting route. Provide a sandbox where people can learn without affecting customers or live records.

Managers need training too. They decide how much time review receives and whether staff feel comfortable questioning an AI-generated result. A policy loses credibility when speed targets make its checks impractical.

The European Commission’s current AI literacy guidance describes context-sensitive measures that support staff and others operating AI on an organisation’s behalf. It does not prescribe one specific literacy level for every individual. Read the Commission’s AI literacy guidance.

Check decisions, then improve the programme

Completion rates show attendance. Short practical exercises reveal whether the guidance works. Can a participant spot a disclosure risk, challenge an unsupported answer and locate the reporting route?

Repeat relevant exercises after a significant tool or workflow change. Review recurring questions with the people maintaining the system. Confusion can be evidence that the interface or policy needs improvement.

The objective is confident, responsible use. Choose three realistic decisions from one team’s daily work and build the first session around them.

Connecting a model to a business system creates a new route through which information and actions can travel. That route needs an identity, a purpose and a boundary.

A confident instruction in a prompt cannot substitute for an access control enforced by the application. The model may misunderstand a task or encounter hostile instructions in material it reads. Design the surrounding system to limit what an error can affect.

Start with the smallest useful permission

Give each workflow only the data and actions needed for its job. Keep reading, drafting and committing changes as separate capabilities where the risk warrants it. Enforce authorisation at the tool or service boundary using the user’s legitimate access context.

Avoid a shared administrator credential that gives every agent the same reach. Review service identities, secrets handling and revocation paths. In an illustrative document assistant, access should follow the user’s entitlement to a document, including during retrieval and citation.

OWASP’s excessive-agency guidance highlights unnecessary functionality, permissions and autonomy as sources of risk. Read the OWASP guidance.

Treat connected content as untrusted input

An external page, email or repository file can contain instructions. Those instructions are data to assess, not authority to change the workflow’s purpose or permissions.

Use layered controls: narrowly defined tools, validated parameters, restricted destinations, approval for consequential actions and useful monitoring. Test whether the system remains within its boundary when retrieved material is misleading or adversarial. No single prompt or content filter should carry the entire security argument.

Retain ordinary engineering disciplines as well: patch dependencies, review changes, protect credentials and prepare recovery procedures. The NCSC’s secure AI development guidance covers security throughout design, development, deployment and operation. Read the NCSC guidance.

Test what the system must refuse

A useful security review includes attempts to reach another user’s records, send data to an unapproved destination or perform an action outside the task. Record the result at the system boundary, not just the model’s verbal refusal.

Start with one agent and list every tool it can call. For each capability, identify its business need, enforced scope, owner and revocation method. Remove access that has no defensible purpose.

The question “Which AI tool may we use?” needs a companion: “Which information may this workflow send to it?”

A supplier approval is useful, but it cannot answer every question about purpose, user access or sensitivity. The same service may be appropriate for public product information and inappropriate for a confidential personnel case under the chosen configuration.

Turn classification into a usable decision

Define practical categories with examples from the organisation: public, internal, confidential and restricted. For each category, specify approved workflows and environments, required handling, and the person who can authorise an exception.

These are organisational design choices, not universal legal classifications. Involve the relevant data, privacy and security owners so the rules match actual obligations and contracts.

Reduce the material sent to what the task needs. A support draft may need a problem description and product details; it may not need an entire customer history. Redaction can help, but context and combinations of details may still identify someone or reveal confidential information.

Follow the full information path

Map the prompt, uploaded files, retrieval store, inference service, tool responses, logs, backups and generated output. Ask where each is processed and retained, who can access it and how it is deleted.

An EU storage location answers only part of this map. Inference location, support access, subprocessors and connected tools deserve separate examination. Obtain service-specific evidence rather than assuming that a regional label covers every component.

OWASP identifies sensitive-information disclosure as an application risk involving both the model and its context. Read OWASP’s sensitive-information guidance.

Make the rule available at the point of use

Put concise handling guidance beside upload and connector controls. Enforce access restrictions and destination rules where possible. Provide a route for legitimate exceptions, with an owner, documented scope and review date.

Review the outputs too. A summary can expose confidential information even when every input came from an approved repository. Retrieval permission and permission to disclose are separate decisions.

My starting deliverable would be a one-page data-flow map for a single use case. Trace a real example from entry to deletion. Any unanswered question becomes a specific architecture or supplier-review task.

An AI policy becomes operational when it changes what a system can do. A document saying “use responsibly” leaves too many decisions to interpretation at the moment of action.

My approach is to turn each important rule into a control with a clear owner and a way to test whether it works. The harness is where business policy meets the model’s tools, context and execution path.

Translate rules into specific controls

Take a rule such as “external communications require an authorised reviewer.” The implementation needs a draft state, an identified reviewer, a view of the proposed message and recipients, and a send operation that remains unavailable until approval is recorded.

A spending boundary needs an enforced limit and an approval route for exceptions. A data restriction needs access and destination controls. A requirement to use approved evidence needs source checks and a clear response when evidence is missing.

These examples are design patterns, not claims that one technical control establishes legal compliance. The control set must fit the use case and its consequences.

Give exceptions a controlled path

Some legitimate work will fall outside the standard route. Record the reason, approving authority, scope and expiry of an exception. Avoid silently converting a one-off exception into permanent permission.

Treat changes to models, prompts, tools and approval logic as changes to the operating system of the workflow. Version them, test the affected cases and preserve a recovery path. The same model can behave differently when its context or tool set changes.

The NCSC’s secure AI guidance treats deployment and ongoing operation as security responsibilities alongside development. That supports maintaining controls after launch. Read the NCSC guidance.

Keep enforcement fair and understandable

Staff need approved ways to complete their work and a safe route to report errors. Deliberate bypassing of controls should enter an established, proportionate process developed with HR, legal specialists and employee representatives where applicable. Do not improvise automatic penalties from an AI alert.

Investigate what happened, whether the rule was clear, and whether the system made safe behaviour practical. Technical containment and an employment decision serve different purposes.

Choose one policy statement this week. Identify its enforcement point, responsible owner, exception route and test. If none exists, the next task is implementation.

“A human will check it” can sound reassuring while concealing an unfinished design. Who is that person? What will they see? How much time do they have? Can they reject the proposed action without fighting the system?

I treat oversight as part of the workflow architecture. It needs a defined purpose and enough support to make a decision worthwhile.

Design the review around the consequence

A low-consequence internal draft and a consequential customer decision need different review arrangements. Decide which outputs need review, what triggers escalation and which actions should remain outside the agent’s authority.

The reviewer should see the proposed action, relevant evidence, material uncertainty and the changes the action would make. A generated explanation is not automatically proof that the underlying reasoning or sources are correct. Show the actual supporting records where access and confidentiality allow it.

For an illustrative account update, the review could show the current value, proposed value, source record and affected downstream process. A generic “approve” button provides less useful context.

Give reviewers a workable job

Train people on the specific failure modes they are expected to catch. Protect time for review and provide a route to specialist help. Let reviewers pause, reject or return work with a reason that can improve the system.

Measure review load as the service grows. If automation creates more cases than the team can meaningfully inspect, the original oversight design no longer matches the operation. Change the scope, capacity or controls before treating rushed approvals as evidence of safety.

NIST’s Generative AI Profile discusses risks including problematic human-AI configurations and provides risk-management actions for organisations. Read the NIST Generative AI Profile.

Evaluate the review step itself

Use representative exercises to check whether reviewers detect relevant errors and understand when to escalate. Inspect false alarms too: an exhausting stream of low-value warnings can undermine attention.

Record the decision and enough context for accountability, with appropriate limits on sensitive logging. Review recurring rejection reasons with engineering and the process owner.

For the next pilot, write a short reviewer brief before building the approval screen. Specify the evidence, expected decision, available actions and escalation route. Then test it with the people who will use it.