A Prometheus configuration can load cleanly while a recording rule still produces the wrong number. That difference is easy to miss when a rule has valid YAML, valid PromQL, and a sensible name. The error often appears later, when a dashboard panel is unexpectedly high, an alert threshold is crossed early, or a dependent rule has no series to evaluate.
The useful gate is not another configuration check. It is a small rule test with known input series and an expected result. In an isolated Linux AMD64 validation with Prometheus 3.15.0-rc.1, a recording rule that summed two synthetic http_inprogress_requests series passed when the expected value was 7. Changing only the expected value to 6 made promtool test rules return exit status 1 and show the expected and actual samples. promtool check rules accepted the rule in both cases because its job is syntax and rule-file validation, not proving an intended result.
This distinction makes a practical pre-reload boundary: use check rules to reject malformed rule files, then use a deterministic rule test to reject a rule whose behavior differs from the monitoring contract. The examples use invented metric values and do not connect to a Prometheus server. Adapt the metric name, labels, evaluation interval, and expected values to a reviewed staging or production change; do not copy synthetic test data into a live rule file.
Start with the question the rule must answer
A recording rule stores the result of a PromQL expression as a new time series at each evaluation. That can make repeated dashboard queries cheaper and lets alerts depend on a named, reviewed calculation instead of repeating a long expression. It also creates an interface: once a dashboard expects job:http_inprogress_requests:sum, the recording rule’s label set and value are part of that dashboard’s contract.
For this example, the contract is deliberately narrow: at the two-minute evaluation point, the rule must add the current values from two instances carrying the same job label and produce one series with job="api". Put that expectation in words before writing YAML. It identifies whether the aggregation should retain job, whether an instance label should disappear, and what test input is needed.
Save the rule in a separate file:
# rules.yml
groups:
- name: request-count
rules:
- record: job:http_inprogress_requests:sum
expr: sum by (job) (http_inprogress_requests)
The groups list gives Prometheus an evaluation group. The record value is the name of the generated metric. sum by (job) adds input series while preserving the job label, so two input instances become one output series for each job. Do not use the example metric name as a recommendation for every workload. A counter, histogram, gauge, or rate expression needs its own semantic review before aggregation.
The expression above is small enough that it can look self-evident. It is still worth testing because a later edit might remove by (job), add an unintended matcher, retain an unwanted label, or change the metric family. PromQL can remain valid while the output expected by a dashboard changes.
Check the rule file before testing behavior
Use the promtool binary that matches the Prometheus version planned for the service. For a downloaded upstream release, verify its archive against the release’s sha256sums.txt before extracting it, then record the binary version. Never execute a downloaded monitoring binary whose archive failed verification.
Run the structural check first:
<path-to-promtool> check rules rules.yml
The 3.15.0-rc.1 validation reported one rule and exited zero:
Checking rules.yml
SUCCESS: 1 rules found
This check matters, but it has a limited claim. It confirms that promtool can parse the rule file and recognizes its rule structure. It does not know what values a live target will export, whether an aggregation is the desired aggregation, or whether a dashboard query expects the labels this rule produces. Treat a successful check as permission to run the behavioral test, not as evidence that the monitoring result is correct.
Prometheus 3.15.0-rc.1 was published on 22 September 2026. Its release notes include an Agent-mode WAL corruption fix after a failed WAL write. A release candidate is useful for testing an upgrade path, but it is not a blanket instruction to deploy a prerelease to production. Test the exact binary selected by the deployment policy and read its release notes before changing a running server.
Give the rule a small, known history
promtool test rules uses a YAML test definition containing the rule files, a synthetic series history, and expected query results. The test below evaluates every minute. At two minutes, instance a has value 2 and instance b has value 5, so the recorded result must be 7.
# passing-test.yml
rule_files:
- rules.yml
evaluation_interval: 1m
tests:
- interval: 1m
input_series:
- series: 'http_inprogress_requests{job="api",instance="a"}'
values: '1 3 2'
- series: 'http_inprogress_requests{job="api",instance="b"}'
values: '2 4 5'
promql_expr_test:
- expr: job:http_inprogress_requests:sum
eval_time: 2m
exp_samples:
- labels: 'job:http_inprogress_requests:sum{job="api"}'
value: 7
rule_files identifies the recording rules to load. evaluation_interval defines how often the synthetic engine evaluates them. Each values sequence supplies samples one minute apart because this test’s interval is one minute. promql_expr_test evaluates the recorded metric after the rule has run, rather than merely validating the original expression text.
The expected sample must include the generated metric name and every label that matters to its downstream consumers. That makes label loss and label proliferation visible in review. If the actual rule accidentally keeps instance, this expected single series will not match. If a change removes job, the test similarly fails instead of allowing a dashboard query to discover the incompatibility later.
Run the test after the structural check:
<path-to-promtool> test rules passing-test.yml
The isolated run exited zero and printed SUCCESS. That proves only the stated calculation under these synthetic samples. It does not prove that an exporter exposes http_inprogress_requests, that a scrape succeeds, or that real cardinality is safe. Those are separate checks for the target and the deployment environment.
Keep one incorrect expectation on purpose
A passing test is more credible when a nearby incorrect test demonstrates what the gate detects. Copy the test file, change the single expected value from 7 to 6, and keep all rule and input data unchanged:
exp_samples:
- labels: 'job:http_inprogress_requests:sum{job="api"}'
value: 6
Run that copy in the same way:
<path-to-promtool> test rules failing-test.yml
The 3.15.0-rc.1 validation returned exit status 1 and reported the disagreement:
FAILED:
expr: "job:http_inprogress_requests:sum", time: 2m,
exp: {__name__="job:http_inprogress_requests:sum", job="api"} 6E+00
got: {__name__="job:http_inprogress_requests:sum", job="api"} 7E+00
The negative result is the important operational finding. The same rule file passed check rules; the failure came only when promtool compared evaluated output with the declared monitoring contract. In CI, preserve the nonzero status. Do not append || true, replace a failing value with an unreviewed update, or make the test pass by dropping labels from the expected sample. A mismatch needs a decision: correct the rule, correct an intentionally changed dashboard contract, or add test coverage for the new behavior.
For a production rule change, add cases that represent the decisions the rule makes: an empty result, label variants that must remain separate, a threshold boundary for an alert-dependent value, and a value that must not trigger a false positive. Keep sample data synthetic unless the service owner has approved a redacted fixture. Metric labels can expose tenant names, internal hostnames, paths, or account identifiers just as readily as logs can.
A recording rule becomes safer when its expected output is versioned beside the expression. The validated Prometheus run showed that parser acceptance and behavioral correctness are separate gates: a valid rule passed the structural check, a correct expectation passed the rule test, and a deliberately wrong expectation failed with the actual sample. Put both checks before a reload, then verify the live target and dashboard through the change procedure that owns them.