AI engineering
From AI pilot to a dependable production workflow
Define what an AI feature must prove before people depend on it in their daily work.
An AI pilot is ready for a controlled production release when it can complete a defined task within agreed quality, cost and response-time limits, with permissions, human responsibility and failure handling in place. Assess those requirements across representative tasks before widening access.
Start with one workflow. “Draft a monthly project update from approved activity for staff review” is testable. “Automate our operations” is too broad to evaluate or release responsibly.
Define the job and its limits
Write down the input, expected output, intended user and permitted actions. Decide what the feature must refuse or escalate. For example, drafting an answer, changing a record and sending a message each require different controls.
Record a baseline using the current process: time to prepare and review a result, common errors and the cases that require an experienced person. The purpose is to compare the new workflow fairly, including human review time.
Build a representative evaluation set
Collect approved examples of routine tasks, difficult cases and requests that should fail safely. Include missing information, conflicting documents, stale records and requests from users with different permissions. Keep a separate set of examples for checking changes so that tuning against the development examples does not become the only measure of quality.
For a report-drafting workflow, assess whether each statement is supported, whether important facts were omitted and whether the draft distinguishes planned work from completed work. Fluency alone is insufficient.
Define the release thresholds with the people accountable for the workflow. The right threshold depends on the consequences of an error; there is no universal accuracy percentage that makes an AI application production-ready.
Evaluate the whole workflow
| Area | Question to resolve | Evidence to keep |
|---|---|---|
| Quality | Is the output supported and useful for this task? | Reviewed examples and error categories |
| Access | Can each user retrieve only permitted information? | Permission and cross-account checks |
| Actions | Which changes need approval, and who can approve? | An explicit action and approval map |
| Cost and speed | Does the complete workflow fit the operating budget? | Usage and latency from representative tasks |
| Failure | What happens when a model, source or integration fails? | Tested fallback and retry behaviour |
| Change | How will updates to prompts, models and sources be assessed? | Version history and repeatable evaluations |
Treat retrieved content as source material, not authority to issue new instructions. Keep action permissions in application controls. An instruction in a prompt is useful, but it cannot substitute for enforcing access and approvals in code.
Use a review record, not just a score
Download the AI workflow evaluation record (.txt). For each case, record the permitted inputs, expected behaviour, actual output, source support, reviewer decision and total preparation/review time. Keep model and prompt versions with the result so a later change can be compared fairly.
For example: the source shows a merged pull request but no release record. The answer may describe the merge; it must not claim the change is deployed. A fluent answer that invents a release fails this case even if the rest of the report reads well.
Use different failure categories for unsupported claims, important omissions, permission violations and unacceptable actions. A single average score can hide a severe failure. Decide which categories block release before looking at the results, and retain examples the implementation team has not used for tuning.
Make review a real product step
Show the reviewer the source context and limitations needed to judge the draft. Let them edit or reject it. Keep the reviewed version and record the decision before publishing or sending anything.
In CareOps, our report workflow keeps the original AI draft and evidence, then requires a staff publication step for the client version. In Networkbase, approving an organizer proposal creates a campaign draft; sending remains a separate action. Both workflows make the review decision explicit before information reaches an external audience.
Plan for operation before widening access
Name an owner for failures and quality review. Decide how to disable the feature, continue the work manually, limit usage and recover from an interrupted task. Test changes against the evaluation set before release, then inspect a suitable sample of real outcomes after launch.
Agree how data is selected, retained and shared with providers before using sensitive information. Involve the organization’s security, privacy and operational owners where their requirements apply. The NIST AI Risk Management Framework provides a voluntary reference for a broader discussion of AI risk management.
Make a release decision, not just a demo decision
A pilot should finish with one of three clear outcomes: release to a defined group with monitoring, continue after resolving named failures, or stop because the approach does not meet the need. Keep that decision tied to evidence, not the appeal of the demonstration.
Discuss an AI engineering project with RapidIteration to define the workflow, evaluation plan and release criteria.