Use cases to tests: from Markdown through JSON to pytest on real hardware

The first part of this series described a protocol specification in Markdown, generated through JSON into four target languages. With that, the truth about the wire format lives in one place and every implementation follows it. What is still missing: proof that the implementations actually behave on real hardware the way the spec demands.

This article describes the second step: use cases are written in Markdown as well, transformed into the same JSON intermediate form and driven by pytest as a test bench against real hardware. The bench runs on a Linux® HIL, with the developer-laptop pre-test pattern from SocketCAN on WSL.

Use cases as Markdown

A use case describes behaviour from the user’s point of view: what is sent, what comes back, what state the device is in before and after, what errors are expected. Use cases are the bridge between domain experts and engineering, and they belong in a format both can read. Markdown is that format, with the same conventions as the protocol spec: a heading per use case, tables for pre- and post-conditions, code blocks for concrete message sequences, prose for the rationale.

A typical use case describes one workflow: “power the device, open a connection, read the configuration, change a value, verify the change is persistent”. Every step references messages from the protocol spec, so use cases can never invent something the spec does not know.

The parser, again

A second Markdown parser pulls a structured JSON representation out of the use case documents: per use case the steps, the expected responses, the pre-conditions (device configuration), the wait times, the success and failure criteria.

Same discipline as on the protocol parser: the JSON is a build artefact, not an editable format. To change a use case, change the Markdown. The parser runs in the build and fails when references to non-existing messages appear (it reads the protocol JSON alongside, so both worlds line up automatically).

pytest as the test bench runner

On the test side we use pytest because it brings three properties that matter for a test bench: parametrised tests (one test, many inputs), clear fixtures (setup and teardown cleanly separated), and a mature plugin ecosystem (reports, parallelisation, hardware allocation).

A generator turns the use case JSON into a pytest suite. Per use case there is a test_* function that iterates through the steps and for each step sends the messages (using the generated Python bindings from part 1) and compares the responses against the expected values.

Skeleton, as it typically looks:

import pytest
from generated.protocol import Connect, ReadConfig, WriteValue
from bench.fixtures import target

@pytest.mark.usecase("set_persistent_value")
def test_set_persistent_value(target):
    target.send(Connect(role="admin"))
    cfg = target.send_expect(ReadConfig(), reply_type=ReadConfig.Reply)
    target.send(WriteValue(key=42, value=1234))
    target.power_cycle()
    target.send(Connect(role="admin"))
    cfg2 = target.send_expect(ReadConfig(), reply_type=ReadConfig.Reply)
    assert cfg2.values[42] == 1234, "value not persisted across power cycle"

This function is not written by hand, it is generated from the use case. It still lives as a file in the repo, so that pytest can discover it and reviewers can see what runs.

Fixtures: the target

The target fixture is the bridge to the hardware. It takes care of allocating a free bench slot, setting up the serial or Ethernet connection, resetting the device into a clean starting state, and the teardown at the end. It is the only place that “knows” which physical hardware the test runs on. That way the same use case tests can run against a local simulator, a local USB adapter or a remote HIL, without the tests themselves noticing.

The fixture is also the lever for parallelisation. Several bench slots, several pytest workers, tests get distributed automatically. Established plugins such as pytest-xdist work with their own fixtures and handle hardware allocation atomically.

The iteration cycle

A typical change goes:

1. A domain expert extends a use case because a new workflow needs to be covered.
2. The Markdown file is edited, a new step or a new use case is added.
3. The build runs, the parser regenerates the JSON, the generator regenerates the pytest suite.
4. The pull request shows the diff of the use case Markdown, the diff of the JSON, and the diff of the generated tests. Three views of the same change.
5. pytest runs on the HIL bench slot, the result lands in the PR.

When a test fails, there are two possible truths: either the firmware does not do what the spec says, or the spec is ambiguous or wrong. Both get resolved in the Markdown spec, not in the tests. Tests are never hand-edited to make them green. The rule is the same as in the SSOT pipeline: the spec wins, always.

What the agents do here

The generator side (use case JSON to pytest code) is mechanical enough that a script could do it. We still let an AI agent do it because the same agent assists the use case reviews: it spots missing preconditions, flags redundant tests, suggests edge cases the use case does not mention. That is the place where an agent has more to offer than a generator, and exactly where it brings real value into the iteration cycle.

Important: the decision which suggestions are taken stays with the domain expert. The agent does not write into the spec. It only delivers suggestions in a comment format that flows into the pull request.

Honest pitfalls

• Flaky tests are the most common reason bench pipelines get abandoned. Discipline: every test must be reproducibly green or red. A flaky test is a bug in the test or in the hardware, not a reason to ignore it.
• Test isolation on hardware is harder than in software tests. A power cycle helps, but is slow. Clean reset protocols specified in the use case set out what holds before and after.
• Bench allocation between several workers must be atomic. Otherwise two pytest processes grab the same slot and the logs become useless.

What stays

A Markdown format for use cases, a parser to JSON, a generator to pytest, a bench fixture, and a few rules of the game. Use cases are negotiated with the domain experts, not with the tests. Tests are generated, not written. Reviewers see three consistent views of every change: use case Markdown, JSON intermediate, generated test.

Together with the protocol spec from part 1 the loop closes: the same source drives the protocol implementation in every language, the same style drives the test implementation against the real behaviour. Spec, code, tests, three views of the same truth.

Keywords: Spec Driven Development, pytest testbench embedded, use cases Markdown to test, Hardware in the Loop, HIL bench setup, embedded test automation, pytest-xdist, SocketCAN test, generate protocol tests, testbench consulting, Embedded Consulting Switzerland, swissEmbedded firmware test