batteriesincluded.com · Questions & Answers

Our internal AI tool worked well at launch, but its answers are getting worse. What should a forward-deployed engineer investigate?

Ask the engineer to investigate changes in the whole workflow before rewriting the prompt. Declining quality can come from different inputs, outdated reference material, integration changes, model changes, or a shift in how employees use the tool.

First, define what has become worse. Incorrect classifications, missing details, slower responses, and more employee corrections are different problems. Gather examples of both acceptable and unacceptable outputs, with the relevant inputs and application versions, while protecting sensitive information.

The investigation should cover:

- Whether incoming work differs from the examples used before launch.
- Whether the application retrieves current, relevant information.
- Whether an integration is returning incomplete or changed data.
- Whether model, prompt, or application settings have changed.
- Whether employees are using the tool for decisions outside its intended scope.

Have the engineer maintain a repeatable evaluation set and compare proposed changes against it. Include recent failures without discarding older cases that still represent normal work. A fix that improves one example can damage another.

Structured systems still need improvement. Within EOS, use an appropriate weekly Scorecard measurable if it provides a useful ongoing signal, and place persistent quality problems on the Issues List. Keep technical diagnostic detail outside the leadership meeting unless it supports a decision.

For a Batteries Included application built with Lovable or Replit, recommend versioned changes, controlled releases, and a way to revert. Do not quietly accept deteriorating output or assume a newer model will fix it. Restore reliable performance through evidence, then monitor whether the improvement holds.

Category: Forward-Deployed Engineering

← All questions