AI Product Development: AI features designed for real use
We build AI into products people pay for or rely on, with the test of what a good answer is written before the build and a plan for the day the AI is wrong.
In brief
AI Product Development builds products that contain AI and keep working when the AI is wrong. AI features fail in use when they are built to impress in a demonstration, so we define good before building, design for failure and human review, build with guardrails, and give the product ways to notice and correct its own errors.
On this page
Does this product need AI in it?
AI earns its place when
- The task is judgment over loose material, such as reading, sorting, summarizing or drafting, and a person would otherwise spend the time
- A wrong answer can be caught by a reviewer or a check before it costs anyone
- You can collect real examples of the task and agree what a good answer is
Leave it out when
- A fixed rule or a lookup does the job, since the model only adds a way to fail
- A single wrong answer would harm someone and no person looks at it first
- Your own experts disagree about what a correct answer is on the same cases
A model adds a failure that ordinary software lacks: wrong output that reads as right. You give up predictability and gain capability that rules cannot cover. We say whether the task can reach the quality the product needs after the evaluation, and before the build is paid for.
A demonstration shows the examples someone chose. A product meets all the others.
What should the first AI release prove?
Whether, on real inputs, the output is good enough that a user accepts it, and what users then do with it. So the first release takes one task for one kind of user, keeps a person in the loop, and logs every input and output.
Buyers see it before it exists in full. A person produces the answers behind a prototype, using the buyer's own documents with permission, so the buyer reacts to real output quality and not to a slide. Their security lead gets the data questions answered in the design: where data goes, which provider sees it, how long anything is kept.
Who does what, and what exists at each stage
Each stage ends in an artifact a named person can inspect. The evaluation comes first because nothing after it can be judged without it.
| Stage | Who works on it | What exists at the end |
|---|---|---|
| Define good | Product owner, a domain expert, our evaluation lead | Real cases with scored answers, and a held-back set |
| Design for failure | Product owner, designer, support lead | Screens for doubt and review, the model-down behavior, a list of actions the AI never takes alone |
| Build with guardrails | Engineering lead, security lead | Input and output checks, access limits, logs that trace an answer to its inputs |
| Release to chosen users | Product owner, support | A route to report bad answers and thresholds that page a person |
| Run and change | The product team | A review routine and a run book for changing models and prompts |
A pattern we are called into
A summary feature in a legal product
A legal technology team wants the product to summarize contracts. The demonstration impressed the sales team. The lawyers who would rely on it have not been asked what a correct summary must contain.
- Define good. Lawyers mark up real agreements. A summary that omits a limit on liability fails, however fluent it reads.
- Design. Every statement links to the passage it came from. When the clause is not found, the product says so and does not guess.
- Build. A provider's model update changes how indemnity clauses are summarized. The evaluation catches it before release.
- Release. Chosen users go first. The team watches how often they open the source passage.
Where it endsA feature whose quality the team can state, with a known list of what it will not attempt.
What do you own when the build ends?
The evaluation set and its scoring rules, the prompts and configuration, the logs and alert thresholds, and the change run book are yours, in your repository. Contracts with model providers are in your company's name, so the decision to switch models stays with you. We hand over a note on how each test case was chosen and scored.
Questions buyers ask
Do you build our own model or use an existing one?
The evaluation decides. Often an existing model with good design around it is enough. A custom model is worth its cost only when the evaluation shows no existing one reaches the quality the product requires.
How do you handle the AI giving a wrong answer to a customer?
By design: the interface shows how sure the product is, a person reviews cases where an error is costly, and monitoring finds patterns of failure so they are fixed once and not repeatedly.
Can you add AI to a product we already have?
Yes. We read how the product is used today and where AI would change the work, then build the evaluation and guardrails around that one capability, so it ships without destabilizing the rest of the product.
Related
Will your AI feature still work when it is wrong?
Describe what you are weighing in a few lines. A senior practitioner replies with a suggested first conversation.