Testing a Real Product: A Thermostat Node, End to End

Most of our articles show one feature at a time: emulate a sensor, catch a boot regression, replay an analog signal. That's a good way to explain a feature and a poor way to prove a test bench. Real firmware does all of it at once, and the interesting bugs live where the features meet: the alarm that fires but never reaches the CAN bus, the control loop that drifts when a sensor read runs long, the watchdog that resets mid-update.

So we built a small product to test. It's open source, in the `process-node-stm32` example, and about 80 hardware tests run against it with a single BenchPod.

The device under test

The node is a NUCLEO-F446RE with an off-the-shelf CAN module, playing a small process controller you'd find in a building or a machine:

• an analog input for a process temperature sensor (0.5 V + 20 mV/°C)
• an analog output that drives a heater
• a thermostat: a PI controller at 50 Hz that holds a setpoint
• an environment sensor on I2C (a BMP280)
• alarms for over-temperature, a faulty process sensor and a missing environment sensor, on a GPIO pin, the console and the CAN bus
• CAN with status frames, echo and periodic messages
• a watchdog and a text console over UART

Nothing exotic, and that's the point. It looks like the firmware our customers write.

One pod plays the rest of the world

Every input the node has comes from the pod, and every output goes back to it:

The wiring is stored as the bench's wiring profile, so the tests use signal names (alarm, uart_rx) instead of channel numbers.

Something to heat

A thermostat test needs something to heat. On a desk that's a resistor glued to a thermistor and a lot of waiting. In CI it has to be fast, repeatable and adjustable.

The pod can run a control loop in its FPGA: read the ADC, look up a curve, drive the DAC, with no computer in the path. We use it as the plant. The node's heater voltage goes into the pod's ADC, the pod maps it to the temperature that heater would reach, lags toward it with a time constant of about 0.3 s, and feeds the result back to the node's sensor input:

node heater out -> pod ADC -> curve + lag (FPGA) -> pod DAC -> node sensor in -> PI -> heater out

It's fast enough for a test to settle in a second, and real enough that the controller has to do its job. A host-side simulation of the same plant (sim_thermostat.c, part of make test-host) picked the gains, so the hardware test checks them against the same model:

@pytest.mark.hardware
def test_reaches_and_holds_setpoint(node, plant):
    node.aout("200")
    plant(AMBIENT_C)
    time.sleep(1.0)
    node.ctl("on 60")
    took, peak, trace = _wait_settled(node, 60.0)
    assert peak <= 60.0 + MAX_OVERSHOOT_C, f"overshoot to {peak} degC; trace {trace}"
    pv, out, pvs = _hold(node)
    assert abs(pv - 60.0) <= HOLD_TOL_C, f"holding {pv:.2f} degC, readings {pvs}"

Other tests change the setpoint, check the plant on its own, and drop the ambient temperature by 10 °C mid-run, like a cold load going into an oven.

One fault, three outputs

The alarm tests are where end-to-end testing pays off. One fault has to show up in three places: the alarm pin, an EVT line on the console and a CAN frame. A unit test can check the logic. Only the real board can show that all three actually happen, on time:

@pytest.mark.hardware
def test_overtemp_reaches_every_output(benchpod, wiring, node, env):
    with benchpod.open_can(bitrate=BITRATE, mode="normal", term=True) as bus:
        env.set_i2c_sensor(temperature_c=60.0)              # the room overheats

        assert benchpod.wait_for_level(wiring.alarm, 1, timeout=2.0)
        node.uart.expect("EVT alarm set=overtemp active=0x01", timeout=2.0)
        frame = bus.expect(can_id=0x0A0, timeout=2.0)
        assert (frame.data[0], frame.data[1]) == (0x01, 0x01)

The same file sends a short spike the debounce should ignore, checks the 2 °C hysteresis on the way back, trips the heater and makes sure it won't restart while hot, shorts the process sensor mid-regulation, and pulls the environment sensor off the bus and puts it back.

Everything else a product needs

The rest of the suite reads like a product test plan, because it is one:

• Lifecycle. Firmware identity, uptime, and the reset cause after a power cycle, a software reset and a watchdog reset.
• Analog in and out. A 0.1-3.0 V input sweep with gain and offset per point; DC levels and 10, 50 and 200 Hz sines on the output, measured by the pod's ADC.
• Sensors. Besides the BMP280, the pod plays a BME280, an SHT4x and an MPU-6050, including the details drivers trip over: CRCs, precision modes, a device that powers up asleep.
• Timing. The logic analyzer times the node's tasks on a strobe pin: 100 Hz ADC sampling and its jitter, the 50 Hz control rate, 200 ms sensor reads, and the alarm pin rising within 0.5 ms of the reading that confirms it.
• Power. Boot time, current at idle and while regulating against a budget, and 5, 50 and 500 ms supply dropouts. The node has to ride through or reboot cleanly, never hang.
• CAN. Both directions, standard and extended IDs, request and response, periodic timing, bursts past the receive buffer, listen mode, several bitrates, a bitrate mismatch, and the termination switch: on, traffic flows; off, nothing does.
• Updates. Flash, boot and identify the image several times in a row.

--soak N runs every test N times and ends with a pass-rate table, flakiest first. A test that passes 49 times out of 50 is a bug too, in the firmware or in the test, and the table finds it.

Same tests, LAN or cloud

The suite takes a connection like any BenchPod test:

pytest process-node-stm32/tests -v --benchpod-connection=192.168.1.213 \
    --benchpod-firmware=process-node-stm32/build/process-node.elf

Point --benchpod-connection at embeddedci:<pod name> and the same tests run through embeddedci.com. The first cloud run found a real gap: an open console blocked ADC and LA captures over the cloud, where the LAN allowed both. We fixed it in the SDK and the server, and the cloud results now match the LAN.

That's exactly the kind of bug this example is for. No single-feature demo would have hit it, because it takes a console and a capture at the same time.

Run it yourself

The firmware, wiring table and tests are in the examples repo. On a digital-only BenchPod the analog tests skip on their own, and PROCESS_NODE_ANALOG=0 skips them on a bench without the analog leads. It runs in GitHub Actions the same way as in Running Hardware CI with GitHub Actions.

If your product looks like this one, a few sensors, an actuator, a bus and a control loop, the suite is a good template to copy.