Results

Software tests are meant to run automatically on every code change, so that breaking changes are caught as soon as possible. The testing results are often not visible to the end user. The following sections present a snapshot of the implemented tests at the time this document is published.

Unit Tests

the listing shows the test results for the kernel unit test. These tests were run on the host system.

test alloc::bump::tests::overflow ... ok
test alloc::bump::tests::alloc_and_dealloc ... ok
test alloc::const_pool::tests::drop_item ... ok
test alloc::heap::tests::only_allocations ... ok
test alloc::heap::tests::alloc_and_dealloc ... ok
test alloc::pool::tests::alloc_and_dealloc ... ok
test alloc::heap::tests::over_allocation - should panic ... ok
test alloc::pool::tests::one_partition ... ok
test alloc::pool::tests::multiple_partitions ... ok
test mem::linked_list::tests::find_and_take ... ok
test mem::linked_list::tests::insert_at_condition ... ok
test mem::linked_list::tests::iterate ... ok
test mem::linked_list::tests::iterate_mut ... ok
test mem::linked_list::tests::length ... ok
test mem::linked_list::tests::memory_overflow ... ok
test mem::linked_list::tests::one_node ... ok
test mem::linked_list::tests::pushing_and_popping ... ok
test mem::queue::mpmc_linked::tests::fifo ... ok
test mem::queue::mpmc_linked::tests::lifo ... ok
test sync::mpsc::tests::by_ref ... ok
test sync::mpsc::tests::queue_overflow ... ok
test sync::mpsc::tests::single_producer ... ok
test sync::mpsc::tests::static_channel ... ok
test sync::spsc::tests::by_ref ... ok
test sync::spsc::tests::overflow ... ok
test sync::spsc::tests::single_producer ... ok
test sync::mpsc::tests::multi_producer ... ok
test sync::spsc::tests::static_channel ... ok
test sync::spsc::tests::underflow ... ok
test sync::spsc::tests::spsc_thread ... ok

Listing: Kernel unit tests results.

The listing shows that allocators, lists and queues are tested at the unit level. These components manipulate raw memory. Because they contain many unsafe sections, the compiler cannot guarantee correctness. Safe behavior is validated using these unit tests. Additionally, these standalone components do not interact with the kernel. Therefore, they are easy to test.

Hardware Integration Tests

The following test scenarios were run against an Armv7E-M based microcontroller (STM32F446RE).

  1. Thread: \begin{enumerate}[nosep]

  2. Creating a thread.

  3. Capturing resources from the environment.

  4. Putting a thread into sleep mode.

    \item Memory Protection:

  5. Accessing data in the parent process.

  6. Accessing peripherals.

  7. Preventing stack overflow.

  8. Preventing access to memory outside the parent process.

    \item Mutex:

  9. Manipulating data stored in a mutex.

  10. Awaiting a mutex.

  11. Behavior on priority inversion.

    \item Semaphore:

  12. Manipulating a semaphore.

  13. Awaiting a semaphore.

\end{enumerate}

These tests focus on the correct behavior of the kernel in case that a thread violates a memory boundary. Correct scheduling of the threads was tested implicitly. As mentioned in the unit test section, testing the interacting components requires more effort. Therefore, the event system used for semaphores and mutexes is tested at the hardware integration level instead.

System

The timing results can be either checked against fixed timing requirements or returned as continuous values. The statistics for three test cases are shown in the table. For those tests, the microcontroller was running at 168 MHz.

timing-stats

Two tests are interrupt driven. The first one uses an interrupt service routine without kernel interaction. The second one uses the kernel interrupt handler. As expected is the direct ISR call was faster than the kernel handler. Though, in release mode the difference is only three fold instead of 10 fold in debug mode. Overall, the release mode improves performance by a factor of 10 to 20. Additionally, jitter is very low, which might be due to the low system load.

The last test case evaluates the time it takes to switch from a low priority thread releasing a semaphore to a high priority thread awaiting the semaphore. In [20] the same test was carried out on an STM32F407 running at 168 MHz with RTOSs written in C. The results are comparable as the microcontrollers are almost identical and they compiled their application with optimization level O3. Bern RTOS (3.8 \textmu s) performs well compared to FreeRTOSv10 (4.13 \textmu s), μC/OS-III (5.56 \textmu s) and RTX (3.26 \textmu s). For a final verdict the tests with the RTOS written in C should be repeated to ensure equivalent implementation. Especially, as Bern RTOS should be slightly slower than the other RTOS because it must change the memory protection setting in the context switch.