What happened to Alamatic?

Back in 2013 I started designing and implementing a new programming language aimed at systems programming for "small" systems, like microcontrollers. I continued that work on and off for a couple of years afterwards, but I had to take some time away from it for other priorities and then had trouble getting the motivation back.

I've been thinking about several of my older projects again this year, and have concluded that the conditions that motivated me to start working on Alamatic no longer apply. I don't expect to be doing any more work on Alamatic, and this article is a sort of wrap up and retrospective to see the project off.

What was Alamatic anyway?

I started working on Alamatic because I felt that folks writing code for microcontrollers and other small systems were being underserved by C as the primary programming language. C's very light abstractions over the underlying system and its ability to run freestanding (with little or no runtime library) serve small systems programming very well, but the concepts offered by the language for system decomposition and reusability are rather lacking.

Firstly, it lacks a first-class module system, instead relying on the C preprocessor to concatenate source files together and perform simple macro substitutions. In the context of small systems, this tends to discourage using abstractions at all, and a codebase for an embedded system will tend to be very tightly coupled to the exact hardware it was built for, making a port to different hardware far harder than I'd like.

Secondly, the total lack of memory safety has plagued embedded applications with obscure bugs and security vulnerabilities. The ability to access memory directly is important when writing drivers, but it's unfortunate that with C this is an "all or nothing" decision: if you use C, you'll need to take care of memory safety yourself (and you'll probably get it wrong!).

My initial experiments in this area were in using C++ instead of C and using its additional language features to create low-cost or zero-cost abstractions, leaning heavily on C++ template metaprogramming and the then-rather-new C++11 features like constexpr. The general methodology here was to define standard interfaces for common system buses like SPI, I²C, UART, etc and then have implementations of those interfaces for each target microcontroller/board. Device drivers would then consume these to allow reusing them across many different target systems.

I used this early C++ library in a few different projects at the time, composing device drivers I wrote with implementations of the interfaces targetting both Atmel AVR and later ARM Cortex M microcontollers from NXP. Here's an example of some "wiring" code (that is, hard-wired peripheral configuration code) from a project that was rendering Conway's Game of Life on a large pixel display built from zig-zagging LED strips with LPD8806 chips driving them:

#include <alambre/system/avr.h>
#include <alambre/bus/spi.h>
#include <alambre/capability/1dgraphics.h>
#include <alambre/capability/2dgraphics.h>
#include <alambre/device/lpd8806.h>
#include "gameoflife.h"
#include "gameofliferenderer.h"
#include "ledmatrix.h"

SoftwareSpiLeaderOutputBus <typeof(*avr_system.B3), typeof(*avr_system.B5)> spi_bus(avr_system.B3, avr_system.B5);
Lpd8806Device<typeof(spi_bus)> strip_device(&spi_bus);
typedef typeof(strip_device) strip_device_type;

Lpd8806Color palette[] = {
    Lpd8806Color::get_closest(0, 0, 48),
    Lpd8806Color::get_closest(0, 0, 48),
    Lpd8806Color::get_closest(0, 0, 48),
    Lpd8806Color::get_closest(0, 0, 48),
    Lpd8806Color::get_closest(0, 0, 48),
    Lpd8806Color::get_closest(0, 0, 48),
    Lpd8806Color::get_closest(0, 0, 48),
    Lpd8806Color::get_closest(0, 0, 56),
    Lpd8806Color::get_closest(0, 0, 64),
    Lpd8806Color::get_closest(0, 0, 72),
    Lpd8806Color::get_closest(0, 0, 80),
    Lpd8806Color::get_closest(0, 0, 88),
    Lpd8806Color::get_closest(0, 0, 96),
    Lpd8806Color::get_closest(0, 0, 104),
    Lpd8806Color::get_closest(0, 0, 112),
    Lpd8806Color::get_closest(0, 0, 120),
    Lpd8806Color::get_closest(0, 0, 128),
    Lpd8806Color::get_closest(0, 0, 136),
    Lpd8806Color::get_closest(0, 0, 144),
    Lpd8806Color::get_closest(0, 0, 152),
    Lpd8806Color::get_closest(0, 0, 160),
    Lpd8806Color::get_closest(0, 0, 168),
    Lpd8806Color::get_closest(0, 0, 176),
    Lpd8806Color::get_closest(0, 0, 184),
    Lpd8806Color::get_closest(0, 0, 192),
    Lpd8806Color::get_closest(0, 0, 200),
    Lpd8806Color::get_closest(0, 0, 208),
    Lpd8806Color::get_closest(0, 0, 216),
    Lpd8806Color::get_closest(0, 0, 224),
    Lpd8806Color::get_closest(0, 0, 232),
    Lpd8806Color::get_closest(0, 0, 240),
    Lpd8806Color::get_closest(0, 0, 254)
};
PaletteBitmap1d<unsigned int, strip_device_type::raw_color, 161, sizeof(palette) / sizeof(palette[0])> bitmap1d(palette);
Lpd8806Bitmap1dDisplay<typeof(bitmap1d.render_bitmap), strip_device_type, 161> display(&strip_device);
RadiantMuralZigZagMutableBitmap1dAsBitmap2dAdapter<typeof(bitmap1d)> bitmap(&bitmap1d);
GameOfLife<24, 9> game_of_life;
GameOfLifeRenderer<typeof(game_of_life), typeof(bitmap)> renderer(&game_of_life, &bitmap);
auto white = strip_device.get_closest_color(255, 255, 255);
auto blue = strip_device.get_closest_color(0, 0, 127);

void main() {
    game_of_life.next_frame();
    renderer.update();
    display.update(&bitmap1d.render_bitmap);
}

In this particular example, we are passing te LPD886 driver a software ("bigbang") SPI implementation using the GPIO interface for an AVR microcontroller, then wrapping that driver in a "1D graphics" interface. Then we have an application-specific implementation of the 2D graphics interface that understands the weird/special way I sliced up and arranged the 1D LED strip to create a 2D grid, which finally allows for generic Game of Life and Game of Life Renderer implementations that are unaware of the physical layout of the pixels and the communication protocols needed to update them.

From a technical standpoint, this approach worked very well. Through the use of template metaprogramming the above all reduced down into small machine code directly accessing the AVR's GPIO registers.

From a usability standpoint though, I was unhappy. Although C++11 type inference via the auto keyword allowed hiding some of the ugly generic types, some of them still ended up visible. Furthermore, any mistakes in this sort of wiring code would often produce multiple screenfuls of confusing and un-actionable error messages from the C++ compiler, because C++ template metaprogramming tends to fail deep in the implementation, rather than at the surface.

From this experience, I started designing a new programming language that would be both designed to make the above sort of automatic wiring code more readable and intuitive, and also to include a built-in lightweight cooperative scheduler to help with the event-driven processing that is typical of this sort of system. I did several iterations of the design, so there isn't any single language I can show an example of, but I can illustrate the general idea with a hypothetical transliteration of the above C++ example:

import avr
import spi
import graphics_1d
import lpd8806
import led_matrix
import game_of_life

const spi_bus = spi.new_software_leader_bus(mosi=avr.B3, sclk=avr.B5)
const led_strip_dev = lpd8806.new_spi(spi_bus)
const palette = []lpd8806.Color{
    # (long color palette definition omitted for brevity)
}
const bitmap = graphics_1d.new_palette_bitmap(palette, length=161)
const disp_1d = led_strip_dev.new_1d_display(bitmap)
const disp_2d = led_matrix.new_zigzag_adapter(disp_1d)
const game = game_of_life.new(bitmap.size)
const renderer = game_of_life.new_renderer(bitmap)

proc main():
    loop:
        game.next_frame()
        renderer.update()
        disp_2d.update(renderer.bitmap)
        yield

The first obvious difference compared to the C++ example is that Alamatic was to have a Python-like syntax with blocks indicated by whitespace. That cosmetic decision aside, the two initial motivations here were to automatically determine the types of constants and variables through inference (to hide the generics) and to put the main application code in so-called "processes" (the proc keyword above) that can potentially run concurrently using a co-operative scheduler.

There were some other ideas in the mix here too, such as a programmable compiler (allowing the platform-specific build and deploy process to be encoded as part of the main program), a language-syntax-aware macro system built into the language, and automatic memory management via a mixture of lifetime inference and reference counting, but those parts did not actually make it from the drawing board.

At the time I was initially working on this, C and C++ were the only portable languages in widespread use for microcontrollers and other small systems. My decision to explicitly end work on Alamatic now comes from a number of changes in the ecosystem in the intervening years, which either already address or are showing the potential to address many of the problems I was aiming to solve. I'll explore these in the remaining sections.

(I would be remiss not to mention that some other languages were attacking similar problems but that were arguably seeing more niche use. The most visible one was D, which remains somewhat popular in a small community but I would not consider it to be in mainstream use.)

Rust

The programming language Rust already had development well underway by the time I was starting work on Alamatic, but the Rust language was still in its early design iterations, with several features not yet well baked and things changing rapidly.

As a consequence, although I was aware of and paying attention to Rust during my early Alamatic work, at that time it didn't feel to me like it was addressing the same set of problems I was. Part of that reaction was honestly misunderstanding on my part: with things moving so quickly, I can see that in retrospect there were details I didn't see clearly.

Rust 1.0 was released in 2015, about two years after I'd started work on Alamatic. By that time, the language had been refined quite a bit and it had shed many concepts that had turned out to be superflous or better represented as library features.

Although a typical Rust "crate" assumes access to an operating system via functionality in Rust's std libraries, it's straightforward to write a crate that uses only the low-level runtime library features, by annotating the crate as no_std. The program can then provide its own system bootstrapping code, memory allocators, etc in a way that's appropriate to target "bare metal" programming on typical embedded systems.

Rust's concept of traits (originally inherited from Self), when combined with generics, end up achieving a very similar set of capabilities to Alamatic interfaces. The Rust embedded programming ecosystem includes embedded-hal, which defines a set of traits abstracting over common buses, allowing drivers to be written against these traits and thus run on any platform that offers implementations of those traits.

Rust also has a well-integrated macro system that works with the grammar constructs of the Rust language itself, and which can encapsulate code generation tasks that can allow for more complex abstractions while retaining readability and safety.

The main Alamatic idea that has been lacking in Rust is some analog to Alamatic's cooperative scheduler. However, in recent releases Rust has introduced the long-awaited async/await features that provide language-level building blocks to enable both OS-based and bare metal cooperative concurrency. There is still some remaining work to make this usable in a no_std context for embedded, systems, but that work does seem to be underway. I'm excited to see where this leads. In the meantime, I had some good success with the less-general model offered by the rtfm framework, which offers event dispatch and task priorities using interrupts on ARM Cortex M microcontrollers.

The rust-embedded community has already achieved good coverage of a number of embedded platforms and drivers for various devices. I've written some Rust libraries for embedded systems myself, and have broadly enjoyed the experience:

lpc81x_hal: HAL implementations for NXP's LPC81x series of microcontrollers, which I'd previously targetted in Alamatic and its predecessor C++ library.
ssd1322: A portable device driver for the SSD1322 OLED display driver. (Not yet released in isolation; included as part of a larger project in progress.)
buspirate: HAL implementations that are intended to run on a full operating system and communicate with devices using a Bus Pirate. I've used this to do initial development of device drivers directly on my full PC, prior to integrating them into a microcontroller-oriented codebase.
spidriver: Similar to buspirate, but using a SPIDriver interface instead.

In particular, the Embedded Rust community uses typestate programming to get a similar effect to the Alamatic system wiring model, where the compiler can help ensure that the system hardware is configured in a correct and safe manner with little or no runtime overhead. Using my buspirate library as an example, switching the Bus Pirate into a different mode selects a different API at compile time, so the compiler can help ensure that the intended mode is selected before issuing commands.

Espruino, MicroPython and CircuitPython

Rust covers or will cover many of the features I had in mind for Alamatic, but I think it's fair to say that Rust is quite a complex language that is hard to learn.

Usability was also a big motivator for Alamatic, and part of that was selecting a Python-like syntax and aiming to make tradeoffs that would keep the more complex features like generics and metaprogramming confined to board-support and driver code, with "normal" user code focusing on composition and logic.

When I started Alamatic, I had considered it mandatory to have a static type system and ahead-of-time compilation. I was pleasantly surprised to see that in subsequent years the resources on typical low-cost microcontrollers got big enough to accommodate cut-down ports of common dynamic programming languages, including JavaScript via Espruino and Python via MicroPython. Popular electronics hobbiest vendor Adafruit later embraced MicroPython (via their fork, CircuitPython) as a primary development environment for many of their microcontroller development boards, popular with hobbiests and education.

While I would still love to see a blend of these into a single language, I think MicroPython and similar have done well to fill a gap of providing an accessible, memory-safe programming environment that enables composition and abstractions, and have done so using dynamic language VM rather than via increasingly-complex static type systems.

import time
import board
import busio
import displayio
import adafruit_ssd1322

spi = busio.SPI(board.SCL, board.SDA)
tft_cs = board.D6
tft_dc = board.D9
tft_reset = board.D5

display_bus = displayio.FourWire(
    spi,
    command=tft_dc,
    chip_select=tft_cs,
    reset=tft_reset,
    baudrate=1000000,
)
time.sleep(1)

display = adafruit_ssd1322.SSD1322(
    display_bus,
    width=256,
    height=64,
    colstart=28,
)

I personally prefer working with Rust because I enjoy the static type system, but these VM implementations seem to work really well for those who prefer dynamic programming languages, and they seem to have been really successful as successors to the C++-based Arduino programming environment for hobbiests.

WebAssembly

In an article that has been primarily talking about microcontroller programming, a section on WebAssembly might feel out of place. However, I was careful while introducing Alamatic above to use the more general term "small systems".

My reason for this is that in my initial design phase for Alamatic I'd also observed that code running in web browsers as part of web applications was increasingly taking on characteristics similar to embedded systems, albeit with a generalized rendering engine alongside rather than specialized hardware.

For early Alamatic work, the likely target would've been Emscripten, which in those days was compiling to compact, efficient JavaScript code but has since shifted its focus to asm.js and then WebAssembly.

Modern WebAssembly provides a programming environment that has constraints quite similar to embedded systems: we want to keep the code size relatively small so it can be delivered efficiently over the network, we want to detect events from the user interface and respond with as little latency as possible, and we interact with the outside world (e.g. the host browser) via a low-level, constrained interface.

I'm excited to watch the development of the WebAssembly ecosystem, because the web development space has a history of investing in excellent, productive development tools. I'm expecting we'll see similar investment in tools for productive use of WebAssembly.

A promising project already in progress is AssemblyScript, which is a statically-typed language derived from TypeScript that focuses on efficient generation of WebAssembly code. AssemblyScript programmers can mix use of the thin AssemblyScript runtime library and direct interactions with the WebAssembly runtime, using a familiar JavaScript-like syntax.

Although WebAssembly's initial focus is delivery of code to run in web browsers as part of web applications, there are already efforts to define a set of conventions for using WebAssembly for applications running on traditional operating systems, via WASI.

I expect we'll see WebAssembly continue to broaden its application space, including ultimately taking on use-cases similar to Espruino and MicroPython with code running on Microcontrollers. This could either take the form of running a WebAssembly VM on a microcontroller itself, or we might see ahead-of-time compilation of WebAssembly into ARM or RISC-V native code that could run with similar performance characteristics as code written in C and Rust.

I'm particularly excited to see if this could be the missing piece to bring together the use-cases served by MicroPython and the use-cases served by Rust today. AssemblyScript, or something like it, could occupy a space similar to where I'd targeted Alamatic. WebAssembly could serve as a new cross-language ABI allowing application code written in AssemblyScript to call into driver code written in Rust with little or no additional overhead.

Onward!

The most important difference between all of the projects I described above and my own Alamatic project is their momentum: there are thousands of people working on these projects, and they are improving fast.

I do of course have some sadness around declaring my own project as dead, but it's outweighed by my optimism about what is coming.