From Diagnosis to Controlled Production

What you sometimes see

Each department arrives with its own list of AI ideas, and the committee decides in favor of whichever is argued the loudest. People start from the tool ("we need a chatbot") rather than the problem, and find out too late that the data does not exist or that a simple rule would have been enough.

How to do it right

1. A single entry point, a common scoring grid for every department: you compare projects, not speakers.
2. Start from business pain points, not solutions: how much time or money is lost today, and what does an error cost?
3. Filter before assessing: is it really an AI problem? A rule, better document filing or off-the-shelf software is often enough. This is the step that saves the most time. And if AI is justified, start with the simplest solution: an existing tool, then a prompt, then a RAG, with specific training (fine-tuning) as a last resort.
4. Assess on four axes, with each department: value (measurable); feasibility, starting with the data (does it exist, is it accessible, up to date, personal?) and the business requirements (volume, acceptable latency, explainability); risk (cost of a wrong answer, fairness of decisions, GDPR and AI Act constraints); total cost, running costs included.
5. Prioritize: a few quick wins and one or two structural bets, each with a business sponsor.
6. Deliver a "no" without surprises: the department took part in the scoring, and leaves with a clear condition for reopening the case.

Checklist

• common grid for all departments
• pain point quantified
• "is it AI?" filter passed
• data identified and accessible
• cost of an error assessed
• regulatory risk assessed
• running cost estimated
• sponsor named

What you sometimes see

A POC that shines in the demo, then months spent "moving it to production" with no stopping criterion. Nobody measured the starting point: six months later, there is no way to tell whether the project delivered anything.

How to do it right

1. Tell a POC from an MVP. A proof of concept (POC) proves it can be done, quickly and cheaply: it decides whether the project enters the portfolio. A minimum viable product (MVP) proves it gets used: it decides whether to scale up.
2. Set the go/no-go before the test: a floor (what absolutely has to work) and success criteria, written down before seeing the results.
3. Measure the baseline before launch, with a method (for example a sample of timed searches), and set a target with the business.
4. Deliver in short, deployable increments, guardrails before features.
5. Track two control indicators on top of value: adoption and answer quality.
6. Allow yourself to stop: a regular review where stopping is a real option.

Checklist

• short, low-cost POC
• go/no-go written before the test
• baseline measured
• target set with the business
• short increments, guardrails first
• adoption and quality tracked
• review with the right to stop

What you sometimes see

An assistant plugged into a folder of documents, tested "by eye" on three questions, with a model picked for its reputation. It always answers, even when the information does not exist. Nobody can tell whether it got better or worse after an update.

How to do it right

1. Check that it is a RAG problem: answering questions from a specific corpus, with a verifiable source. If a standard search is enough, there is no need for AI.
2. Qualify the corpus first: who maintains it, which version is authoritative, does it contain personal data, and what is its quality (completeness, accuracy, timeliness, consistency)? An outdated document or a reference to a non-existent article will produce an answer that is wrong, but well phrased.
3. Split along the structure of the documents (articles, sections) rather than into fixed-size chunks, keeping the references (document, version, article): that is what makes citing possible.
4. Measure retrieval separately from the answer. If the right passage is not retrieved, even the best model cannot answer correctly. Changing the embedding model means re-indexing the whole corpus.
5. Constrain the answer: only from the excerpts, with the source cited, and a refusal when the information is missing, tested with out-of-corpus questions.
6. Evaluate before delivering on a question set validated by the business, with thresholds set before the test (see "Evaluating an LLM").
7. Secure and cap: keys kept server-side, quotas and caps, users informed that their data is being sent.
8. Re-evaluate after every change of model, prompt or documents: a RAG can regress without anyone noticing.

Checklist

• corpus qualified and versioned
• structure-based chunking
• retrieval measured on its own
• mandatory citations
• refusal tested out of corpus
• thresholds set before the test
• costs capped
• re-evaluation after every change

What you sometimes see

An agent given powerful tools and instructions in its prompt: "never approve a suspicious claim". The day a document contains a hidden instruction, or the model gets it wrong, nothing stops it from acting.

How to do it right

1. The AI proposes, the code decides. Decisions that commit the organization (status, amount, routing) are made or checked by deterministic code, not by the model.
2. A prompt asks, a schema prevents. What must not happen is not among the agent's tools: an action missing from its schema is impossible, whatever the prompt says.
3. Treat all external content as data, never as an instruction: a prompt injection comes in through the document, not through the user.
4. Recalculate server-side everything the agent produces that has consequences (amounts, thresholds, inconsistent statuses).
5. A human approves decisions that affect a person, with a mandatory reason for any rejection.
6. Check the quota before any external call: otherwise you pay, and send data, for a result that goes unused.
7. Test attacks, not just the happy path: injections, inconsistent documents, malformed inputs.

Checklist

• binding decisions made in code
• forbidden actions absent from the schema
• external content treated as data
• values recalculated server-side
• human approval logged
• quota checked before the call
• injection test carried out

What you sometimes see

"We tested it, it works." A few questions picked by the team, a flattering average score, and a threshold decided after seeing the results. The day an answer invents a contract condition, nobody understands how it got through the tests.

How to do it right

1. A reference question set (golden set) validated by the business, representative of real questions, trick and out-of-scope cases included.
2. Thresholds set before the test, never after.
3. Blocking criteria, not just an average. An answer that invents coverage and a clumsy answer weigh the same in an average score: a single failure on a blocking criterion (a claim missing from the sources, a wrong source, no refusal) must block the release.
4. Check in code whatever can be checked (expected citation, format): it is free and reproducible.
5. Leave to the LLM judge only what code cannot check, such as faithfulness to the sources. The judge is itself an LLM, with known biases (a preference for longer answers, or for answers from its own model).
6. Calibrate the judge before relying on it: compare it with blind human scoring, feed it deliberately wrong answers, and set before the test how many traps it must catch and how many disagreements are tolerated. Choose a judge from a different provider than the model being evaluated.
7. Freeze the baseline before changing tools: measure the old system before retiring it, so you can say "it is better", not just "it works".
8. Re-evaluate after every change of model, prompt or documents.

Checklist

• golden set validated by the business
• thresholds written down before the test
• blocking criteria defined
• failures read one by one
• code checks first
• judge calibrated against human scoring
• judge from another provider
• baseline frozen before any change

What you sometimes see

The "most powerful model on the market" by default, a bill discovered at the end of the month, and quotas set on average usage. Then one day the provider retires the model, and the application goes down.

How to do it right

1. Put a figure on it before choosing: cost per question, expected volume, monthly order of magnitude, to be validated with the business.
2. Define quality on your own questions, not on a market leaderboard, then pick the cheapest model that clears the threshold. If a requirement (for example exact citation) is critical, it is part of the threshold, and you pay for it.
3. Cap costs at three levels: by design (only call the model when it is useful, cap answer length); through quotas calculated on the worst case, not the average; through a spending cap at the provider, dedicated to the project.
4. Isolate each project (dedicated workspace and key) to see its real cost and limit the damage of a compromised key.
5. Track model lifecycles: providers retire models on fixed dates. A migration is tested on the most demanding path, not just the happy path.
6. Keep it reversible: an architecture that lets you compare another provider on the same question set.

Checklist

• cost estimated before choosing
• quality threshold defined on your questions
• cheapest model that clears the threshold
• quotas based on the worst case
• dedicated spending cap
• cost tracked per project
• retirement dates tracked
• comparison with a second provider

What you sometimes see

Customer data sent as is to a model provider, without knowing how long it keeps it or whether it uses it to train its models. The DPO finds out about the project a week before go-live.

How to do it right

1. Involve the DPO and the CISO from the scoping stage, not at the end: data protection impact assessment (DPIA) and AI Act classification before the first increment.
2. Send only what is necessary (data minimization), pseudonymized where possible.
3. Inform people that AI is involved, and of what is sent, before sending it.
4. Read the provider's terms: retention period, use for training (to be refused), data processing agreement; negotiate zero retention when the stakes justify it.
5. Know where inference runs: a European endpoint is no guarantee that all data stays in Europe; you have to read what the provider commits to.
6. Clarify the shared responsibility with the provider: what it guarantees, and what remains your responsibility.
7. Keys kept server-side, role-based access, logs without sensitive content, and any decision affecting a person always signed off by a human.

Checklist

• DPO and CISO involved from scoping
• DPIA and AI Act classification
• data minimization
• information before sending
• retention and training checked
• data processing agreement
• inference location checked
• logs without sensitive content

What you sometimes see

The project team presents its own results, the risk function can only read them, and sign-off rests on trust. The file says what works, rarely what does not.

How to do it right

1. Clarify who signs, according to the organization's governance: usually the business owner, after input from the risk, compliance, security and DPO functions.
2. Meet the risk function at the start, to learn what a file must contain, rather than finding out at sign-off.
3. A file built to be challenged: intended and prohibited uses, data sent and to whom, evaluation protocol with thresholds set beforehand, all results, bad ones included, known limitations, guardrails, monitoring plan.
4. Make everything replayable: question set, scoring grid and scripts, so that the risk function can rerun the tests itself. A validation that can only read is not independent.
5. One common framework for all applications, tested on a pilot first, rather than a different file for each project.

Checklist

• signatory clarified
• risk function met at the start
• prohibited uses written down
• thresholds set beforehand
• complete results, bad ones included
• replayable tests
• limitations and guardrails documented
• monitoring plan

What you sometimes see

A change pushed straight to production "because it's small", an "all tests pass" report accepted without checking, and blind faith in the rollback button.

How to do it right

1. Prove it on a copy of production: every pull request creates a preview, tested by a person on the real user journey. Something missing (a file that should not be public) does not show up in a code review, only in testing.
2. Make the differences between environments deliberate and listed (quotas, keys, services): an unintended difference sends a never-tested feature to production.
3. Take no report at face value, whether it comes from a developer or a coding agent: demand the raw test output, step by step.
4. Run tests automatically (continuous integration) and make their status a merge blocker.
5. Check in production after merging: a successful deployment does not prove that the user journey works.
6. Know what a rollback does not undo: data already exposed, a secret already leaked, a database already migrated.

Checklist

• a preview for each pull request
• human test of the user journey
• environment differences listed
• raw test output required
• blocking continuous integration
• check in production
• rollback limits known

What you sometimes see

Outages are reported by users, answer quality is no longer measured once in production, and every incident ends with "it's fixed", with nothing to stop it from happening again.

How to do it right

1. Detect before the user does: a probe that replays the main user journey at regular intervals, alerts on server errors and on spending; monitor latency, traffic, saturation and errors, and set service level objectives (SLOs) on the ones that matter to the user. Without this, an outage lasts until someone notices it.
2. Beware of what breaks nothing: a data exposure or an authorization flaw triggers no alert, and lasts as long as nobody looks for it.
3. Contain: measure the impact (who, which data), roll back to the previous version, inform the people affected and the sponsor (what we know, what we are doing, when we will next update).
4. Fix once things are calm: look for the root cause in the logs once production is stable.
5. Learn without blame: distinguish the immediate cause from the root cause, and produce a rule or a tool.
6. Plan for recovery: backups and a recovery plan that are tested, not just written down; regulatory and security watch.
7. Monitor answer quality too: a report button whose every confirmed case joins the test set, and a random sample of production answers scored with the same grid, because reports only catch the errors someone notices. Keeping real answers is a trade-off to settle with the DPO.

Checklist

• probe on the main user journey
• error and spending alerts
• containment and communication procedure
• blameless post-mortem
• a rule from every incident
• reports linked to the test set
• production sample scored
• retention settled with the DPO

What you sometimes see

A tool delivered that works, a one-hour training session, a newsletter, and three months later hardly anyone uses it. Adoption was thought about after delivery.

How to do it right

1. Start from the mission, not the tool: AI addresses a pain point the teams recognize.
2. Involve future users from the scoping stage, not just at acceptance testing.
3. Measure usage and perceived quality, alongside value: low adoption is a signal to understand, not to hide.
4. Remove the concrete obstacles: trust in the answers (visible sources), the tool's place in the workflow, clear rules on what can be put into it.
5. A visible sponsor and champions within the teams.

Checklist

• pain point recognized by the teams
• users involved from scoping
• usage measured
• obstacles identified
• sponsor and champions named