Quality and Verification
The 4 principles in this chapter define when the agent may say “done” and what evidence must back that claim. A principle is a named pstack rule kept in its own skill file (its leaf); /poteto-mode keeps an index of all of them and applies the ones a task triggers, as described in the Principles Overview. The poteto-mode index files Prove It Works, Fix Root Causes, Test Behavior, Not Implementation, and Explain the Number under its “Verification” category, which the guide summarizes as the principles that “define what counts as proof”. The fifth member of that category, Sequence Work into Verifiable Units, is covered in Execution and Operations.
Prove It Works (principle-prove-it-works)
Source: skills/principle-prove-it-works/SKILL.md
- Rule: before declaring a task done, check the real artifact directly. Run the feature, read the actual value, inspect the diff.
- What does not count: proxies such as file modification times, the agent’s own report, cached output, or the fact that the code compiles.
- Failed checks: when a check fails, the leaf tells the agent to suspect its method of observation before it suspects the system.
- Scripting: wherever possible, turn the check into a deterministic script that reruns the comparison, since a repeatable check is stronger than a single look. Keep the output where the human can see it, and commit it only if the trail must be auditable later.
- Steering: the guide’s own example covers the most common case. When the agent claims success because the build passed, type
apply prove it works. run the real import flow and show me the written records.
Fix Root Causes (principle-fix-root-causes)
Source: skills/principle-fix-root-causes/SKILL.md
- Rule: repair the fault itself, which may lie far from where the symptom appeared. The root cause is the first point at which the system did something it should not have done, or failed to handle a state it should have handled.
- Method: reproduce the problem first. Keep asking “why” until the answer is a wrong decision in the code, and when progress stalls, add instrumentation and read the actual state.
- The deciding question: was the state that set off the symptom a legitimate one? If so, deal with it at the point where the code failed. If that state should never exist, a check at the point of failure only conceals the bug, so trace whatever produced the state. The leaf lists the usual ways of concealing a bug: guards, retries, swallowed errors, clamped values, and discarded state.
- Scope: fix the pattern as well as the instance by finding every other place with the same fault.
- Follow-up: if the decisive fact required a deep investigation, propose a cheap, always-on signal that would have revealed it at first glance.
- Steering: when a diff adds a guard or a retry, ask whether the state it handles is legitimate. The description states the test plainly: “A guard is a fix only when the state it handles is legitimate.”
Test Behavior, Not Implementation (principle-test-behavior-not-implementation)
Source: skills/principle-test-behavior-not-implementation/SKILL.md
-
Rule: a test asserts something the users of the code can observe through its interface. That means return values, errors, state they can read back, and effects on external systems. Private helpers, internal call order, and intermediate data belong to the implementation.
-
2 imagined edits: the leaf judges every test against 2 hypothetical changes.
Imagined edit If the test… Verdict Break the behavior (a wrong value, an effect skipped or doubled) still passes It proves nothing Change the internals while keeping every observable result fails It is pinned to the implementation Keep the tests that catch the first edit and survive the second; rewrite or delete the others.
-
Why tests prove nothing: the leaf gives 3 common reasons. An assertion may be too loose, the expected value may come from the very code being tested, or the assertion may check data that the test built for itself.
-
Writing good tests:
- Drive the code through its public interface with a concrete input.
- Derive the expected value from the requirement.
- Count the data an external service receives as behavior, and keep calls to internal helpers out of the assertions.
- Pair every “this never happens” assertion with a case where it does happen.
- Assert the error type that callers branch on, and leave the message text unasserted.
-
Steering: if a refactor that preserves behavior breaks many tests, those tests are pinned to the implementation, and naming this principle asks the agent to rewrite them.
Explain the Number (principle-explain-the-number)
Source: skills/principle-explain-the-number/SKILL.md
- Rule: scrutinize a measured number (a speedup, a regression, a throughput, a latency, an eval result) twice. Once before measuring, and again before trusting, reporting, or acting on it. Identify what limits the number, and rule out the possibility that it measured something other than the intended work.
- The 4 steps of the leaf:
- Before running, write down the decision at stake, the result that would change it, and the noise floor. Stop if the spread between runs exceeds the effect.
- Ask the leaf’s question, “why not double?” Name the resource or code path that bounds the result, using a profile or counters captured during a run. A guess from reading the code does not qualify as a limiter.
- Consider the other things the number might reflect (errors, work that was skipped or cached, code that never executed, a side that was not tuned, noise, or a piece too small to matter), and rule each one out with evidence.
- Keep the run count, the spread, and the limiter together with the number.
- Performance numbers: the leaf hands these to the
benchmark-checklistskill, which the guide describes as turning the principle into 7 questions answered from real runs (see Verification). - Eval results: ask whether every run actually performed the task, whether the gap holds across trials and models, and whether the scenario matters at all.
- Steering: when the agent reports a bare figure, ask it to explain the number and name the limiter.