Jeremiah's notes
AI Product Development
AI features only create value when they remove real friction, not when they are added as decoration.
AI Product Development
Where AI Features Actually Make Sense in a New Product
A practical, founder-focused framework for deciding where AI belongs in a product, based on real bottlenecks, workflow impact, cost, and measurable outcomes.
Start With a Bottleneck
The strongest AI features do not start with a model, they start with friction. Specifically, a repeatable, expensive, or time-consuming bottleneck inside a workflow. That might be reviewing hundreds of support tickets, extracting data from PDFs, writing repetitive content, or stitching together information from multiple systems.
If you cannot clearly describe the bottleneck in one sentence, the AI feature is probably premature. Many teams jump straight to “we should add AI” without identifying what it is actually fixing. In those cases, the feature becomes surface-level and rarely changes user behaviour.
A good test is this: if you removed the AI feature, would the product still work almost as well? If the answer is yes, the feature is likely decorative rather than essential.
Map the Workflow Before Adding AI
Before introducing AI, map the full workflow the user is going through. Where does the data come from? What decisions are being made? Where does time get wasted? Where do errors occur?
Most valuable AI features sit inside an existing workflow, not on top of it. They reduce steps, not add new ones. When teams skip this step, they often bolt AI onto the side of the product, which creates fragmentation rather than improvement.
- Identify input → processing → decision → output stages.
- Highlight where users repeat similar actions.
- Find steps that require manual interpretation or synthesis.
- Look for delays caused by context switching between tools.
AI should compress or simplify these stages. If it introduces a new step (“click here to generate AI output”), it needs to save more time than it adds.
Prefer Assisted Workflows Over Automation
Fully autonomous AI systems are appealing in theory, but in practice they are difficult to trust, hard to debug, and expensive to maintain. Most successful AI features are assistive, not autonomous.
Assisted workflows keep the human in the loop while reducing effort. The user reviews, edits, and approves outputs instead of blindly relying on them.
- Drafting content instead of publishing automatically.
- Suggesting classifications instead of enforcing them.
- Highlighting insights instead of making final decisions.
- Extracting data with validation rather than direct insertion.
This approach reduces risk and increases adoption. Users are far more comfortable when they feel in control, especially in products that affect revenue, compliance, or customer outcomes.
Design for Trust and Failure Modes
AI systems are probabilistic. They will be wrong sometimes. Designing as if they are always correct is one of the fastest ways to lose user trust.
Instead, design for how the system behaves when it fails. This is where most AI features fall apart.
- Show source context or the data used to generate outputs.
- Allow users to easily edit, override, or reject results.
- Make uncertainty visible instead of hiding it.
- Provide fallback behaviour when AI fails or times out.
- Log and review incorrect outputs to improve prompts or models.
Trust is not created by accuracy alone. It is created by transparency, control, and predictable behaviour when things go wrong.
Choose the Right AI Pattern for the Job
A common mistake is using generative AI for problems that are better solved with simpler approaches. Not every problem requires a large language model.
- Generative AI: Best for drafting, rewriting, summarisation, and open-ended tasks.
- Retrieval-based systems: Best for answering questions grounded in known data.
- Classification models: Best for tagging, routing, and structured categorisation.
- Rules or deterministic logic: Often better for predictable, high-accuracy requirements.
The more deterministic the problem, the less suitable generative AI becomes. Matching the tool to the problem is often the difference between a reliable feature and an unpredictable one.
Understand the Real Cost of AI Features
AI is not just a feature—it is an ongoing system with cost and operational overhead. Many teams underestimate this and treat it like a one-time build.
Costs include model usage, infrastructure, retries, monitoring, prompt tuning, evaluation pipelines, and ongoing iteration. Complexity increases further when you need reliability, auditability, or compliance.
- Per-request API costs that scale with usage.
- Latency trade-offs between cost and quality.
- Edge case handling and fallback systems.
- Prompt and model version management.
- Ongoing QA for output quality.
If the feature does not produce meaningful value, these costs compound quickly without delivering return.
Define Success Before You Build
AI features should be tied to measurable outcomes. Without this, it is difficult to know whether the feature is working or worth maintaining.
Define what success looks like before implementation, not after.
- Reduction in task completion time.
- Improvement in output quality or consistency.
- Decrease in manual workload.
- Increase in conversion or engagement.
- Reduction in support or operational overhead.
If you cannot measure the outcome, you are effectively guessing whether the feature is valuable.
Where AI Consistently Works Well
- Content-heavy workflows (drafting, summarising, rewriting).
- Unstructured data processing (documents, emails, notes).
- Search and knowledge retrieval across fragmented systems.
- Support and triage systems where volume is high.
- Data extraction and normalisation tasks.
These areas benefit because they involve interpretation, repetition, and large volumes of semi-structured data exactly where AI can reduce effort without requiring perfect accuracy.
Where AI Usually Fails
- Core product functionality that requires near-perfect accuracy.
- Workflows that are not yet clearly defined or stable.
- Features added primarily for marketing or competitive pressure.
- Highly deterministic tasks that could be solved with simpler logic.
- Systems without clear feedback loops or validation mechanisms.
In these cases, AI often increases complexity without improving outcomes. The result is a feature that looks impressive in demos but is rarely used in practice.
A Simple Decision Framework
Before adding an AI feature, you can run a simple decision check:
- Is there a clear, repeatable bottleneck?
- Does the feature remove or significantly reduce that bottleneck?
- Can the user review or control the output?
- Is the problem suited to probabilistic output?
- Can we measure whether this improves the product?
- Does the value justify the ongoing cost and complexity?
If most of these answers are unclear or negative, the feature is likely premature.
AI can be one of the most powerful tools in product development, but only when applied deliberately. The best AI features are not the most advanced—they are the ones that quietly remove friction, improve workflows, and deliver measurable value without adding unnecessary complexity.