An inclusive communication bridge for children: a child raises an open palm, a cartoon face on screen greets them, and the two hold a short conversation, a game, a story, a goodbye — using only hand shapes. No network stack, no camera data ever leaves the device, so the conversation stays private to the child using it.
The conversation, the rock-paper-scissors game, and the story generation, all running live on the EK-RA8P1 board.
The exchange is meant to mimic conversational reciprocity — the back-and-forth turn-taking of a real conversation — rather than a single command triggering a single response, with the animated face appearing to understand and talk back through its reactions.
The TRON × AI theme is about combining RTOS and AI rather than running one alongside the other. On µT-Kernel 3.0, this demo does exactly that: two independent AI workloads, computer vision and language generation, share a single Renesas RA8P1 chip through the kernel's own scheduling and memory primitives.
On a single microcontroller, combining two AI workloads is not automatic. AI workloads are large relative to an MCU's compute and memory budget, and running more than one at once — for example a vision pipeline that must keep up with a live camera while a language model computes in the background — puts them in direct competition for CPU time and for the limited fast memory an AI accelerator needs. Without guaranteed task order, a slower AI workload can stall a time-critical one.
The vision half feeds an MIPI camera into a palm-detection model and a hand-landmark model, both running on the on-chip Ethos-U55 NPU; the resulting 21 joint positions become one of five named gestures through geometry ratio tests on the Cortex-M85. The language half runs a "Tiny" Llama2 model at 8-bit quantization that writes a short story one token at a time, reading its weights directly out of external Octo-SPI flash instead of copying them into RAM, with its int8 matrix multiplication calling the Cortex-M85's Helium vector instructions directly — sixteen int8 lanes per instruction instead of one.
µT-Kernel's fixed task priorities let these two workloads share one chip safely, as four application tasks. Story generation only uses CPU time left over after the other three, so it cannot delay the camera or the display: the camera picture keeps updating and story text keeps appearing together, with no dropped frames on the panel.
The same benchmark other microcontrollers run alone —
llama2.c's stories15M, 8-bit quantized —
runs here at a comparable rate while also driving a live camera
and two NPU vision models at once.
For comparison, published microcontroller runs of the same
stories15M language model alone — with no
camera or vision workload alongside it — report about
2.5 tokens/s on a Cortex-M7 and about 2.96 tokens/s on an
ESP32-S3.
The same four tasks coordinate through a single µT-Kernel event flag plus shared, single-writer memory — a signaling path small enough that its worst-case timing is easy to reason about.
Only ready and done bits pass through it. An interrupt sets its bits directly, with nothing in between that could overflow.
Processed images, joint positions, gesture results, and story text each live in a structure written by exactly one task, so no two tasks race over the same memory.
21 joint positions become one of five named gestures through geometry ratio tests on the Cortex-M85 — no embedding model, no network call, just distances and ratios computed on-device.
This demo is deliberately small — one board, one camera, one child-sized conversation. But the same properties that make it work reach well beyond this one demo. A gesture-only interface reaches people voice and touch assistants don't: non-verbal children, hospital patients, anyone across a language barrier — the back-and-forth reciprocity here is what makes an interface feel like it's listening, which matters most for the users existing assistants serve worst.
Keeping every byte on-device means that reach doesn't cost privacy: nothing about a child's face or voice has anywhere to leak, by construction, not by policy.
And underneath both is the discipline we build every engagement on: hardware-aware intelligence. We bridge the gap between heavy algorithms and constrained silicon, engineering code that strictly respects memory, thermal, and energy limits rather than asking the hardware to stretch to fit the software. This demo puts that discipline under real pressure — two AI workloads sharing one RA8P1 chip, scheduled by µT-Kernel's fixed priorities, without ever missing a frame. AI on a microcontroller doesn't have to mean flaky or cloud-dependent — not when the algorithm is engineered to fit the silicon it runs on, instead of the other way around.
This is one build from our Renesas RA8P1 R&D work. Share your hardware targets and we'll walk you through what it takes to run vision and language workloads together on a single microcontroller.
services@gnomons.com