System Tests
The system tests validate the behavior of a complete system with other components. In terms of embedded systems, the end product consisting of hardware and software is tested against communication interfaces or electrical signals.
The kernel influences system tests with a computational overhead that affects timing constraints. For example, interrupt handling in the kernel increases latency compared with bare-metal applications. The timing behavior is evaluated using a digital signal generator and analyzer.
Figure: Integration testing setup for timing measurements.
The test setup in the figure is built from a STMicroelectronics STM32F446 (Arm Cortex-M4F) microcontroller and a Digilent Digital Discovery logic analyzer/pattern generator. The pattern generator triggers an event on the microcontroller. In reaction the device under test outputs a signal. Depending on the test case, the logic analyzer can measure the time between the stimulus and the reaction signal (e.g. interrupt latency) or the duration the reaction signal was high (e.g. context switch).
A python script on a computer controls the testing procedure. The script sets up the measurement device, programs the microcontroller using cargo and then reads the measurements. The results are stored as raw data for statics analysis and compared against timing requirements. Thanks to pytest, the results can be easily reported to a continuous integration server.