Back to resources
HIL
Testing
Firmware
CI

Cutting Power in the Middle of an OTA Update, on Purpose

Every field update has a moment where the old image is half gone and the new one is half written. We cut power and pull reset at random points of that window, hundreds of times, and check over the console that the device always comes back.

Edward Viaene · November 17, 2026 · 7 min read

Every firmware update has a window where the device is fragile: the new image is half written, the bootloader's flags are about to flip, and a power cut leaves whatever happens to be in flash. In the field that window gets hit. Someone unplugs the unit, a battery dies, a breaker trips. On the bench it almost never does, because nobody sits there pulling the plug at the right millisecond.

So we let the BenchPod do it. It powers the board, watches the update over the console, cuts power (or pulls reset) at a random point, powers back on and checks what the device does next. Then it does that again, a few hundred times, in CI.

What "survives" means

A device survives a power cut during an update when the next boot ends in one of two places:

  • The old image, because the new one never got marked as good (a rollback).
  • The new image, because it was complete and verified before the switch.

Everything else is a field return. These are the outcomes we check for, from the console alone:

After the cut, the device...Verdict
boots the old versionpass, and a second update must still work
boots the new versionpass, if the image check ran before the switch
stays in the bootloader, waitingfail: bricked until someone reflashes it
reboots over and overfail: a boot loop, often a half-written image marked valid
boots the old version, refuses every later updatefail: the update state is stuck

That last row is the one teams miss. The device looks fine after the cut, so a quick test passes. A week later the next update silently does nothing.

The test

Our example node doesn't have a bootloader with A/B slots, so the code below is the shape of the test for a device that has one. It assumes the firmware takes an update <url> command on its console, prints ota: <n>% while it writes, and prints app v<N> ready once it's up. Your device will say something else; only the patterns change.

import random
import re
import time

import pytest

UPDATE_URL = "http://10.0.0.5/fw/app-v2.bin"     # served by the CI job
READY = re.compile(r"app v(\d+) ready")


@pytest.fixture
def on_v1(benchpod):
    """Start every run from a known image: flash v1 over SWD."""
    benchpod.flash(file="build/app-v1.elf")
    benchpod.power_off()
    time.sleep(0.5)
    yield benchpod
    benchpod.power_off()


@pytest.mark.hardware
@pytest.mark.parametrize("run", range(200))
def test_power_loss_during_update(on_v1, run):
    bp = on_v1
    cut_at = random.Random(run).randint(1, 99)     # same cut points every time: a failure reproduces

    with bp.open_uart() as uart:
        bp.power_on()
        uart.expect(READY, timeout=10)
        uart.write(f"update {UPDATE_URL}\n")
        uart.expect(f"ota: {cut_at}%", timeout=60)
        bp.power_off()                              # the cut

    version = boot_and_check(bp)
    assert version in (1, 2), f"run {run}: no app after a cut at {cut_at}%"

    if version == 1:
        # Rolled back. The device must still take the update.
        with bp.open_uart() as uart:
            uart.write(f"update {UPDATE_URL}\n")
            uart.expect(READY, timeout=120)
        bp.power_off()
        assert boot_and_check(bp) == 2


def boot_and_check(bp, settle=0.5, window=15.0):
    """Power on, return the version that came up, and fail on a boot loop."""
    time.sleep(settle)                              # let the board's capacitors drain
    with bp.open_uart() as uart:
        bp.power_on()
        match = uart.expect(READY, timeout=window)
        time.sleep(3.0)                             # stay a little: does it reboot again?
        boots = uart.text.count("boot:")
    assert boots <= 1, f"{boots} boots: a boot loop"
    return int(match.group(1))

A few details matter here.

The cut point is seeded by the run number. random.Random(run) gives every run its own cut point, but the same one every time the suite runs. When run 137 fails at 61%, it fails at 61% again on your desk.

`open_uart` buffers from the moment it opens. The session starts listening before power_on, so the first bytes of the boot banner, which often carry the bootloader's slot decision, are in uart.text too. That's where you'd also assert on lines like boot: slot B invalid, falling back to A.

The settle time is not optional. A board with a few hundred microfarads on its rail can keep the MCU alive for a surprisingly long time after the eFuse opens. Turn power back on too soon and you've tested a short glitch, not a power loss. Measure it once: the power monitor shows the rail current dropping to zero, and that's your minimum.

Reset is a different test

Pulling the reset line instead of the power exercises a different failure. The MCU stops mid-instruction, but RAM keeps its contents, the flash stays powered, and any retained-memory flags your bootloader uses survive. Some bugs only show up one way.

The pod drives the DUT's reset pin directly, so the reset variant is one line:

uart.expect(f"ota: {cut_at}%", timeout=60)
bp.reset_target(pulse=0.05)          # 50 ms on NRST, power stays on

reset_target times the pulse on the pod and returns once the line is released, so the DUT is already running again when the call comes back. For a longer hold (say, to keep the board in reset while you check its current), bp.set_reset(True) and bp.set_reset(False) assert and release it.

We run both variants, with the same seeds. A bootloader that survives every power cut and fails on reset usually trusts a RAM flag that's only cleared at power-on.

Where in the update to cut

"Somewhere between 1% and 99%" covers the bulk of the write, but the riskiest moments are short:

  • The erase. On many MCUs a sector erase takes tens of milliseconds and the progress counter doesn't move. Cut during ota: 0% and right after the first progress line.
  • The last write and the flag flip. The few milliseconds between "image written" and "slot marked valid" decide between a rollback and a brick. If your firmware prints a line at that point (ota: verify, ota: switch), add those as cut points.
  • The first boot of the new image. Many A/B schemes boot the new image once on trial and only commit it when the app confirms. Cut during that first boot and the device should fall back.

A short list of fixed cut points next to the random ones catches these on every run:

CUT_POINTS = ["ota: 0%", "ota: 1%", "ota: verify", "ota: switch", READY]

The pod sends the power-off command over the network, so the cut lands a few milliseconds after the console line arrives. That jitter is useful: across 200 runs it spreads the cuts around each marker rather than hitting exactly the same instruction every time.

Check the current, too

A device that "boots" can still be wrong. A bootloader stuck retrying a flash write draws a different current than an idle app. power_profile records the rail while the device comes up:

with bp.power_profile(max_duration=10.0) as prof:
    version = boot_and_check(bp)
assert prof.result.peak_current < 0.25          # amps: no stuck erase loop at full current

The power and brown-out article covers the rest of the power side: the eFuses, per-rail current and what the pod reports when a board trips its fuse.

Running it in CI

Two hundred runs at about 40 seconds each is a bit over two hours. That's too long for every push, and just right for a nightly job or a release branch. We split it:

  • Every push: the fixed cut points, both power and reset. Ten runs, about seven minutes.
  • Nightly: 200 seeded random cuts per variant.
  • Before a release: the nightly set again, on the release build.

The same test file runs everywhere: on your desk with the pod on your LAN or over its USB cable, and in GitHub Actions through embeddedci.com, where the job gets an exclusive lease on the pod so two pipelines never cut power on each other. A failure report names the run and its cut point, which is all you need to reproduce it at your desk.

The first two rows are what a quick manual test checks. The last three are what this loop is for, and it's better to find them on the bench than from a support ticket.

The full API (power, reset, UART sessions, power profiles) is in the pytest docs.