How AI-built tools go wrong
| Failure mode | Example | How to catch it |
|---|---|---|
| Plausible but wrong formula | BSF computed with the outer-race formula | compare with a published value for one known case |
| Silent unit errors | g treated as m/s², peak reported as RMS | feed a sine of known amplitude in known units |
| Off-by-one and resolution errors | FFT bin frequency shifted by one bin | check a tone exactly on a bin and one between bins |
| Hard-coded demo values | sampling rate fixed at 25,600 Hz in a hidden place | run a second file with different settings |
| Invented library functions | a function that does not exist in the library version | run it; agents that can execute code catch most of these |
| Over-confident wording | "Bearing failure detected" from one weak peak | read every message the tool can show; ask for evidence and hedging |
| Data leakage in ML | random window shuffle gives 99.8% accuracy | split by recording or machine; expect a lower, honest score |
Verification checklist
Tick items as you go; your progress is saved in this browser.
Where AI-built tools fit, and where they do not
Good fit
- Analysis and calculation aids reviewed by an engineer
- Data preparation and reporting
- Prototypes that prove an idea before a proper project
- Training and teaching tools
Needs a formal software process
- Protection, interlocks and safety instrumented functions
- Control logic that changes equipment state
- Regulatory or contractual calculations
- Systems many people depend on without reviewing outputs
A tool can move from the left column to the right as it grows. When that happens, hand it to a software team with your specification and test cases. Those two documents are the most valuable part of what you built.
Key takeaways
- AI-built code usually fails quietly, with wrong numbers, not crashes.
- Known-answer tests, unit checks and edge cases catch most problems.
- Protect confidential data and state clearly what decisions the tool supports.
- Keep the specification and tests: they make the tool reviewable and transferable.