How Many Empty-Input Cases Your Regression Set Needs

The number you need is driven by branches, not by volume. Before writing a single empty string test, sit with the code and the prompt and list every distinct way the system can reject, default or degrade on empty or short input. A useful starting inventory for an LLM-backed feature looks like this:

• Input truncated to zero characters before the request is built.
• Whitespace-only input that passes a naive length check.
• Input below a minimum length enforced in application code, not by the model.
• Input below a minimum length enforced only by prompt instruction.
• Schema validation failure on the model's structured output.
• Model refusal, where the response contains a refusal field rather than content.
• Tool call that returns empty and needs a degraded data source.
• Retrieval that returns no documents above a similarity threshold.
• Timeout or provider error that should route to a static message.
• Human handoff trigger for low confidence or safety-sensitive tasks.

The distinction between static, structured, model and tool fallbacks matters because each costs different latency and money, and static and structured fallbacks belong in almost every pipeline while model fallback and human handoff should be reserved for cases where a wrong answer is worse than a slow one (MLflow). Your empty-input stratum should cover the cheap ones exhaustively and sample the expensive ones deliberately.

If your feature extracts fields from documents, the branch count explodes in a useful way. A fallback field fires when a field returns an empty array, null or undefined, but does not fire when it returns an empty string or zero, and fallbacks can chain from a layout method to a model prompt and back (Sensible's fallback field docs). Each chain link is a branch. Read the config and count them, because that document alone shows at least three distinct branch types for one logical field.

Once you have a list, mark each branch as reachable by empty input, by short input, by both, or by neither. The ones marked "by neither" need a different stratum entirely and should not inflate this count. The ones marked empty or short form your denominator.

The rule has three parts: a floor, a scaling band, and a cap that flips to branch-driven.

Floor. Plan for a minimum of 12 empty and short inputs. This is not arbitrary. Boundary value analysis, the standard technique for testing at the edges of an input domain, typically probes the minimum, one value just above it, a nominal value, one just below the maximum, the maximum, and invalid values outside the range on both sides (Virtuoso QA on boundary value analysis). For a length domain whose minimum is zero, that gives you the empty case, a one character case, a nominal case, a just-below-maximum case, the maximum case and the over-maximum case. Run that set twice, once through the application layer and once through the prompt layer, and you have 12. If you have fewer than 12 because your domains are narrow, keep the 12 anyway; they will not be wasted.

Scaling band. Above the floor, scale the stratum at roughly 2 percent of the total regression volume, up to about 500 cases. The reason is proportional rather than statistical: a suite that is 2 percent edge cases will surface a broken fallback within one or two runs of any change that touches input handling, while a suite that is 0.1 percent edge cases will hide it behind 999 happy paths that all pass. A 2 percent share survives aggressive additions of valid cases, which is exactly what happens over a suite's lifetime.

Cap. Past about 500 total cases, stop scaling by volume and hold at 20 to 30 cases plus one per distinct fallback branch. Beyond that point, extra empty strings mostly duplicate each other; extra branches do not. This is where the branch count takes over. If you have 14 fallback branches, your stratum is 34 to 44 cases regardless of whether your suite holds 600 or 6,000 cases.

The last row is the decision rule in one line: past the cap, volume stops mattering and branches take over. Teams that keep scaling edge cases linearly with volume spend most of their test budget re-proving that an empty string is still an empty string.

Coverage of a branch is not the same as confidence in it. A branch that is exercised exactly once will look healthy until that one test flakes, and then a broken fallback reads as an infrastructure problem rather than a regression. Three executions per branch gives you a signal you can act on: if two of three pass and one fails, you are looking at nondeterminism or a flaky fixture, not a code change. If all three fail, you have a real break.

Three hits also lets you vary the input meaningfully without inflating the count. For a refusal branch, for example, the three cases might be an empty string, a whitespace-only string, and a short greeting that is well formed but carries no task. That spread catches the difference between "the model refused because there was nothing to do" and "the model refused because the input was malformed", which are different bugs with different fixes.

This is where prompt regression suites overlap with software regression suites. A structured test case needs an exact input, an expected output or a set of expected properties, and tags so you can run subsets such as edge_case or high_priority, and the recommended starting point is 20 to 30 representative cases with an edge case added every time a bug is found and fixed (MLJAR's prompt regression suite prompt). Note the phrase "every time a bug is found and fixed": your branch count is not static. Every production incident on a fallback path adds a branch and therefore adds three cases.

For the cheapest branches, exact match testing against a fixed string is fine, run at temperature zero for reproducibility. For branches where wording may vary, use properties instead: the response must contain a refusal field, must not contain a URL, must be under 40 words, must name the missing field. Property checks are more robust than exact match for outputs with legitimate variability, and the same source catalogues exact match, schema validation, semantic equivalence and property tests as the four available types.

Sizing is arithmetic; finding the branches is the work. This checklist turns the count into an afternoon.

Steps 1 to 4 are discovery and they nearly always produce more branches than the team expected. A pipeline with a primary model, one fallback model and a human handoff typically has at least six infrastructure branches before any content rule is counted, and a routing function that walks a fallback chain collects an error for each provider and returns a diagnostic string when all of them fail (a multi-model fallback walkthrough). Each provider in that chain and each error path is a branch.

Step 8 deserves emphasis. Run the edge stratum on every pull request that touches prompt text, validation logic or the orchestration layer, because that is precisely the set of changes that can break a fallback silently. A static fallback that shipped last month and passes today proves nothing about the version under review.

"Short" is a range, not a point. If your minimum length is not enforced anywhere, an empty-input case and a one character case may take the same path, and your three hits per branch collapse into one test with three spellings. Choose short inputs that straddle your actual decision points:

• The empty string, which fails every length check.
• A string of spaces, which passes a naive length check and fails a trimmed one.
• One character, the first value above zero.
• One character below the declared minimum.
• The declared minimum itself, which should pass.
• The declared minimum plus one, which catches an off-by-one in the comparison.

That last pair is where off-by-one defects live. A field specified as accepting input from 18 characters might be implemented as longer than 18 rather than at least 18, functionally equivalent in the developer's mind and different in behavior for the value 18 itself (Virtuoso QA). The same class of error appears in token budgets, message length limits and pagination offsets.

Where prompt text is the only enforcement, the short-input cases are your only evidence that the instruction works. Test the instruction with both beginner and expert inputs, and test with a request that lacks essential context, because a system prompt that is silent on missing information will produce a confident guess rather than a question (Coursiv's prompt guide). A prompt that says "if the question lacks essential context, ask a clarifying question" turns one branch into a testable one.

One more trap: few-shot examples in a prompt can make short inputs look better than they are. Demonstrations condition the model, and the label space and format of the examples matter for subsequent performance (the Prompt Engineering Guide on few-shot prompting). If your short-input cases accidentally match the shape of a demonstration, they test the demonstration, not the fallback. Deliberately break the shape.

Once the cases exist, the operational questions are which model serves them, how you record results, and when a failing fallback blocks a release. For the routing question, a single API surface for text, image and embedding models removes a whole class of fallback branches around provider-specific SDKs, which is one reason teams consolidate on Quanta Inference. The stratum gets simpler to maintain when the provider is not part of the test matrix.

Recording results is where most teams lose the signal. Log which branch each case hit, not just pass or fail, so a decline in fallback health is visible as a metric rather than an inference. Monitoring fallback rate alongside task success, tool-error and latency metrics reveals degradation trends and masking issues early, and validating fallback prompts against golden sets before they reach users is the stated core rule (MLflow). A stratum of 30 cases with branch identifiers attached gives you a fallback rate per branch for free.

For teams without a test engineer on call, the work of excavating branches from prompts and code is a good candidate for delegation: an expert who reads your repository and returns a finished branch inventory and case table is a faster path than a sprint of internal archaeology, which is what Quanta Experts is built for. Whichever route you take, the deliverable is the same: a branch list, a case count derived from it, and three inputs per branch wired into CI.

The final decision rule, restated plainly: count branches, multiply by three, add a floor of 12, and let that total override any volume-based instinct once your suite passes about 500 cases. If the resulting number feels large, it is almost certainly smaller than the number of empty strings your users have already sent you.