Test runs
benchpod ci runs your hardware tests from one file committed next to them, on your laptop or in GitHub Actions. Each run is saved as a run folder, uploaded to embeddedci.com when you are signed in, and shown on the Builds page next to the build it tested, where you can compare it with the run before.
Run it
benchpod ci # run benchpod-ci.yaml in this directory benchpod ci -f nrf/benchpod-ci.yaml --pod my-pod benchpod ci validate # check the file and the pod's firmware, drive nothing benchpod ci diff <run> <run> # two run ids, or two local run folders benchpod ci upload <folder> # upload a run made without a connection
Without --pod, benchpod ci uses the pod it finds on your network or USB, or the only pod in your organization; with several, it picks a free one and records which. The run flashes, runs the steps in order, always cleans up (DAC off, DUT power off, emulated sensors and pull resistors off), writes the run folder and uploads it. The exit code is 0 when everything passed, 1 for a file or setup error, 2 for a test failure and 4 for an invalid run (the pod rebooted, lost data or lost its lease, or require was not met).
benchpod-ci.yaml
Version 1 of the file. suite and dut name what is tested; together with each step's id they are what runs are compared by, so renaming one starts a new history. One repository with several boards keeps one file per DUT and picks it with -f.
version: 1
suite: process-node-hil # with dut, the key every comparison uses
dut: process-node # what is wired to the pod; "none" for pod-only suites
wiring: process-node # a wiring profile stored on the pod
require:
pod_firmware: ">=3.10.12" # not met: the run is invalid and nothing is driven
firmware:
build: make -C firmware # optional, runs in this file's directory
file: firmware/build/process-node.elf
env:
BENCH_FAST: "1"
baseline: main # compare with the last valid run of this suite and DUT on main
timeout: 20m
keep: {traces: failed} # all | failed | none
upload: auto # auto | true | false
steps:
- id: boot
power_cycle: {until: "APP_OK", timeout: 10s}
required: true # stop the run when this step fails
expect:
boot_ms: {max: 600, drift: "20%"}
- id: idle-current
measure_power: {duration: 2s}
expect:
mean_ma: {min: 5, max: 20}
- id: hil
pytest: {path: tests, args: [-m, hardware]}
- id: go-tests
run: go test ./hwtests -json
results: gotest-jsonEach step has exactly one kind. Steps keep going after a failure unless the failing step is required.
| Step | What it does |
|---|---|
flash | Flash firmware.file with the wiring profile's flash settings (also the implicit first step when firmware is set). |
power_on, power_off | Switch the DUT's power. |
power_cycle | Power off and on, then wait for until on the UART. Metric: boot_ms. |
uart_expect | Optionally send a line, then wait for until. Metric: match_ms. |
measure_power | Measure the DUT's current for duration. Metrics: mean_ma, peak_ma, min_ma, mean_v. |
capture | An LA or ADC capture; its statistics are the metrics. |
wait | Wait a fixed time. |
pytest | Run pytest with the embeddedci plugin: every test is one case, and the measure fixture adds measurements. |
run | Any command. Pass or fail from its exit code; cases from results: (junit:<path> or gotest-json). |
expect checks a metric: min and max are limits that fail the step, and drift is the change allowed against the previous run or the baseline (as a fraction or a percentage). Every metric under expect is stored as a measurement, checked or not, so boot_ms: {} just records it. For a pytest step, name a test's measurement as <case id>/<name>.
The run folder
embeddedci-runs/20261118T140211Z-process-node-hil/ run.json the record of the run, written last junit.xml the same cases as JUnit XML trace/<case>.ndjson the pod calls of a case (kept per keep.traces) att/<n>-<what>.<ext> large payloads the traces refer to (captures) logs/<step>.log the output of a step
run.json holds the suite and DUT, the inputs (the firmware file and its hash, the tests' repository and commit, the pod's firmware and board, the wiring), the run's health, and every case with its outcome, duration, message, measurements and pod call timings. Secrets such as passwords, tokens and keys are removed from traces before they are written. The folder works without the cloud: benchpod ci diff compares two folders offline.
On embeddedci.com the results are kept for good; traces, attachments and logs are kept for 30 days.
Measurements in pytest
The measure fixture of the embeddedci pytest plugin records a value with its unit and optional limits. A value outside min or max fails the test at the end, with the list of failures; drift is checked when runs are compared.
def test_settle_time(bp, measure):
t = bp.power_cycle_and_capture(until_regex="READY").elapsed_ms
measure("settle_ms", t, unit="ms", max=1000, drift="20%")The Tests tab and compare
- A run is linked to the build its firmware came from: by the build id when it flashed an uploaded build, else by the firmware file's hash matching an uploaded artifact (see Cloud builds).
- The Builds list shows the latest run of each build ("96/98 passed" and its DUT). A build's Tests tab lists all its runs; the pod column appears only when your organization has more than one pod.
- A run's page lists failed cases first, filters by outcome, and opens a case to its message, measurements against their limits, pod call timings, and its trace and log.
- Compare with previous run diffs it with the last valid run of the same suite and DUT: the inputs that changed first, then newly failing, fixed, still failing, new, removed and newly skipped cases, measurement changes (out of limits, or beyond their drift), and cases and pod calls that got slower. Runs with a different wiring, or an invalid run, are not compared; a different pod or board is compared and flagged.
Every member of your organization can see its runs. Uploads take a benchpod login session, an API key (BENCHPOD_API_KEY) or, in GitHub Actions, the job's OIDC token. The routes are on the HTTP API page.
The pod is held for the whole run
On the network or USB, benchpod ci takes the pod's own lease (pods with firmware that has the lease capability) and passes it to the pytest and run steps it starts, so they share it. Everyone else is refused while the run holds the pod: the web app, other SDK, CLI and MCP sessions, and cloud jobs. Status and other reads keep working.
- The pod list and the pod's page show CI run in progress, with the suite, when it started and who runs it, and the pod's controls are off until it ends.
- Organization owners and admins can press Stop run. The pod drops the lease, the run stops with "lease lost", cleans up what it can, and is marked invalid.
- A lease that is not renewed ends by itself, and a pod restart clears it, so a crashed run never keeps the pod for long.