Back in October 2020, I built an on-screen keyboard for Squeak
Smalltalk. It ended up being about 230 lines
of code (!) in total.
I’ve been using Smalltalk as the primary UI for my experimental
cellphone software stack, and I needed a way to type input.
Using VNC to develop on the phone works fine, but
to work on the device itself - both for day-to-day phone tasks as well
as developing the system - I need a proper on-device keyboard.
This video shows the keyboard in action. As you can see,
it’s not perfect! At one point I tapped ctrl while typing, leading
to an unexpected pop-up menu appearing.
But the basic idea is sound. (Aside: why aren’t there any Android
keyboards that are laid out like a PC keyboard, with punctuation in
the right place, etc.?)
The usual shortcuts like Alt-P for “print it”, Alt-D for “do it”
and Alt-I for “inspect it” all work for evaluating snippets of
The keyboard integrates with my previously-written touchscreen support
code, with the red/blue/yellow modifier buttons affecting touches,
making them into simulated left-/middle-/right-clicks, and with
keyboard focus changes auto-popping-up and -down the keyboard.
Simulating mouse clicks is a temporary state of affairs that lets me
use the touchscreen reasonably naturally, making use of context menus
and so on, without having to make invasive changes to the input
pipeline of Squeak’s
How it works
Class OnScreenKeyboardMorph synthesizes keyboard events and injects
them into the Morphic world.
You can have arbitrarily many keyboards
instantiated, but there’s a global one living in a
flap at the bottom of the
screen. That’s the blue tab at the bottom left of the screen you can
see in the video.
I had already written code to read from /dev/input/eventN,
tracking each multitouch contact separately. A subclass of HandMorph
overrides newKeyboardFocus: to pop the keyboard up and down as the
keyboard focus comes and goes:
Each key is represented by an OnScreenKeyMorph. It tracks keyboard
shift state by registering as a dependent of the
The important part, of course, is the mouse event handler for
The only other interesting part is the keyboard initialization
routine, which builds the layout:
The rest is book-keeping and simple delegation methods, like this:
and so on.
I haven’t released the whole package yet, because it’s all very much
in flux, but if you’re interested, you can take a look at the
changeset for the Keyboard-related code here:
UPDATE: Full code available at
(also embedded below).
Say you have the following
TypeScript interface I that you
want to invoke remotely by passing messages of type M; or that you
receive messages of type M and want to handle them using an object
of type I:
Keeping things type-safe looks really tedious! There’s obviously a
connection between I and M. Can we avoid writing them by hand?
TL;DR TypeScript lets us define generic types Messages and
Methods such that M = Messages<I> and I = Methods<M>. Read on
for the details.
Interface ⟶ Messages
Let’s start with what, for me, has been the common case: given an
interface type, automatically produce the type of messages that can be
sent to implementors of the interface.
First, how do we want to represent messages?
I’ve taken a leaf out of Smalltalk’s book, and made a message include
a selector, the name of the method the message intends to invoke,
and some args, the provided arguments to the method. The Args
extends never check is to help type inference deduce the empty
argument tuple: without it, the type system won’t complain about
I’ve also added a callback to Message. The technique I describe
here can be further extended to “asynchronous” or callbackless
settings with minor modifications.
The next definition, of type Messages<I>, is where the magic
happens. It expands to a union of Messages representing the methods
defined in I:
And that’s it! Here’s how it works:
MessagesProduct is a mapped type that describes a modified
interface, where all (and only) the method properties of interface
I are rewritten to have a Message as their type, but keeping
the same key;
then, the ...[keyof I] part in the definition of Messages uses
index types to set up a union type built from all
the value types (“indexed access operator”) associated with all the
keys in I (“index type query operator”).
Messages ⟶ Interface
Going in the other direction is simpler:
It’s a mapped type, again, that maps union members that have
Message type to an appropriate function signature. It takes
advantage of TypeScript’s automatic distributivity: a union of
products gets rewritten to be a product of unions. Then, in the
conditional typeM extends Message<...> ? ..., it projects out
just exactly the member of interest again.2
This time we use the mapped type as-is instead of re-projecting it
into a union using indexed access like we did with MessagesProduct
Type-safe interpretation of messages
Now we have types for our interfaces, and types for the messages that
match them, can we write a type-safe generic perform function? Yes,
Given the above definition for I, actually using Messages<I>
produces the following type definition3 (according
to the IDE that I use):
Conversely, given the M from the top of the file, we get the
following for Methods<M>:
Roundtripping works too: both Methods<Messages<I>> and
Messages<Methods<M>> give what you expect.
Then, building on the language itself, emacs with
company makes for a very pleasant
Congratulations to Sam, whose ideas really have worked out amazingly
well, and to the TypeScript team for producing such a polished and
Appendix: Full code implementing this idea
This module implements the idea described in this article, extended
with the notion of EventMessages, which don’t have a callback.
Well, at least in TypeScript v4.x, anyway. I
don’t know about earlier versions. ↩
Actually I’ll admit to not being quite sure that
this is what’s really going on here. TypeScript’s unions feel a
bit murky: there’s been more than one occasion I’ve been surprised
at what a union-of-products has been automatically “simplified”
(?) into. ↩
Hey, what’s going on with those named tuple
slots? I would have expected a tuple type like [number, string]
not to be able to have names attached to the slots, but it turns
out I’m wrong and the compiler at least internally propagates
names in some circumstances! It even re-uses them if you convert a
Messages<I> back into an interface, Methods<Messages<I>>… ↩
Here’s my .emacs TypeScript setup, based on the examples in the tide manual:
Recently on HN,
rbanffy brought up a form of the old chestnut about crashing the
image by simply executing true become: false.
It turns out it’s no longer true!
In the latest Squeak,
doesn’t work – the compiler complains that you can’t assign into a
read-only variable. So let’s try this:
But now the metaprogramming system complains you’re trying to modify a
read-only binding! So we view source on ClassBinding»value:, and see
that a resumable exception is being used to guard the modification, so
let’s explicitly signal that we really want to modify that
Finally! Now, evaluating True yields False.
But the image keeps running! Use of the literal class True seems to
be rare enough that things are OK for at least several minutes after
Doing this, however, definitely should immediately torpedo things:
… huh. It didn’t work. It used to! Again, on this current Squeak
version, we see a different behaviour. This time, it says Cannot
execute #elementsExchangeIdentityWith: on read-only object #(false).
So we’ll have to try harder:
That doesn’t work either! Same error as for #become:.
Welp, I’m actually all out of ways to crash this modern image in
analogous ways to the easy pitfalls of images of yesteryear…
P.S. Running the following command,
analogous to True := False, definitely does have an analogous
effect on a similarly-“alive” Unix image :-) . I just tried it on a
scratch VM; the results are pretty intimidating!
One lovely thing about working in Smalltalk is the effortlessness of
I started off developing the code on my desktop machine, and
occasionally testing it on the phone itself. I just rsync the image
and changes files back and forth. This lets me pick up exactly where I
left off on the other device each time I move over.
However, developing on the phone was challenging because of the lack
of a keyboard (though I’ll post soon about an on-screen keyboard I’ve
written). So I installed RFB (from
here) into my image on the desktop,
and tested it. Then I saved the image and rsynced it to the phone as
usual, and presto, I can develop and test interactively on the phone
Using VNC to develop on the phone itself
There were a couple of things I had to do to get this to work:
As I’ve been working on a mobile Smalltalk system, I’ve found myself needing to decode
and encode a number of complex telephony packet
formats1 such as the following, an incoming SMS
delivery message containing an SMS-DELIVER TPDU in
GSM 03.40 format,
containing seven-bit (!)
GSM 03.38-encoded text:
It turns out there are a plethora of such binary formats needed to
get a working cellphone.
I started off hand-rolling them, but it quickly became too much, so
I borrowed liberallystole from Erlang, and implemented
BitSyntax for Smalltalk. (After all, I am
Erlang-influenced actors for
the Smalltalk system daemons!)
The BitSyntax package includes a BitSyntaxCompiler class which
interprets BitSyntaxSpecification objects, producing reasonably
efficient Smalltalk for decoding and encoding binary structures,
mapping from bytes to instance variables and back again.
The interface to the compiled code is simple. After compiling a
BitSyntaxSpecification for the data format above, we can analyze the
example message straightforwardly:
and, if we wish, serialize it again:
How does it work?
Syntax specifications are built using an embedded domain-specific
For example, for the above data format, we would supply the following
spec for class SmsIncoming:
along with appropriate specs for SmsAddress and SmsPdu (omitted
for space reasons here) and the following for the SmsPdu subclass
These are non-trivial examples; the simple cases are simple, and the
complex cases are usually possible to express without having to write
code by hand. The EDSL is extensible, so more combinators and parser
types can be easily added as the need arises.
The package BitSyntax-Help contains an extensive manual written for
Squeak’s built-in documentation system.
Telephony packet formats are particularly
squirrelly in places. Seven-bit text encoding? Really? Multiple
ways to encode phone numbers. Lengths sometimes in octets,
sometimes in half-octets, sometimes in septets (!) with padding
implicit. Occasional eight-bit data shoehorned into a septet-based
section of a message. Bit fields everywhere. Everything is an
acronym, cross-referenced to yet another document. Looking at the
3GPP and GSM specs gave me flashbacks to the last time I worked in
telephony, nearly 20 years ago… ↩
I’ve been running PostmarketOS on my Samsung Galaxy S7
recently. As far as I can tell, none of the available modem/telephony
stacks supports the Galaxy S7 modem.
I was expecting to be able to just open /dev/ttyACM0 or similar and
speak AT commands
to it, like I used to be able to on Openmoko.
However, Samsung phones have a proprietary,
undocumented1, unstandardized binary interface
to their modems. Operating a modem is the same across members of the
Samsung family, but each different handset seems to have a different
procedure for booting the thing.
So I decided to reverse engineer (a fancy name for “running strace
on cbd and rild and reading a lot of kernel source code”) the
The result is a couple of
quick-and-dirty python scripts which, together, boot
the modem and print out the messages it sends us. It’d be
straightforward to extend it to, for example, send SMS, or to manage
incoming and outgoing calls.
How Android (LineageOS) does it
LineageOS uses Samsung’s proprietary cbd and rild programs,
extracted as binary blobs from the stock firmware.
there’s an undocumented flag, -P (cf. the documented -p) which lets you supply a partition path fragment rather than a partition number
That last is crucial for running the same binary on PostmarketOS,
which has a different layout of the files in /dev.
How libsamsung-ipc does it
The open-source libsamsung-ipc from the
Replicant project handles the boot and
communication processes for a number of (older?) Samsung handsets, but
not the Galaxy S7. Once a modem is booted, libsamsung-ipc passes
higher-level protocol messages back and forth between the modem and
libsamsung-ril, which layers Android telephony support atop the
device-independent abstraction that libsamsung-ipc provides.
Generally, modems are booted by a combination of ioctl calls and
uploads of firmware blobs, with specific start addresses and blob
layouts varying per modem type.
At first, I was concerned that I wouldn’t have enough information to
figure out the necessary constants for the S7 modem. However, luckily
just strace combined with dmesg output and kernel source code was
enough to get it working.
The overall sequence is similar to, but not quite the same as, other
Samsung models already supported by libsamsung-ipc.
Running strace on Samsung’s cbd yields the following steps.
Extract the firmware blobs. The firmware partition has a
table-of-contents that has a
From here, we can read out the chunks of data we will need to upload
to the modem.2
Acquire a wake-lock. I don’t understand the Android wake-lock
system, but during modem boot, cbd acquires the ss310 wakelock.
Open /dev/umts_boot0. This character device is the focus of most
of the subsequent activity. I’ll call the resulting file descriptor
Issue a modem reset ioctl. Send IOCTL_MODEM_RESET (0x6f21) to
Issue a “security request”. Whatever that is! Send
IOCTL_SECURITY_REQ (0x6f53) with mode=2, size_boot=0 and size_main=0
According to cbd’s diagnostics3, this is asking
for “insecure” mode; the same ioctl will be used later to enter a
One interesting thing about this particular call to
IOCTL_SECURITY_REQ is that it answers error status 11 if you run it
as root. The Samsung cbd does a prctl(PR_SET_KEEPCAPS,
1)/setuid(1001)just before IOCTL_SECURITY_REQ, which appears
to give a happier result of 0 from the ioctl. However, if you ask
cbd to stay as root by supplying the command-line flag -or to it,
then it too gets error status 11 from the ioctl. Fortunately, the
error code seems to be ignorable and the modem seems to boot
successfully despite it! This makes me think that perhaps running
IOCTL_SECURITY_REQ at this point, in this way, is optional.
(UPDATE. Here’s what I think is going on. Setting mode=2
apparently asks for “insecure” mode, which allows uploading of
firmware chunks. If we omit the mode=2 call to IOCTL_SECURITY_REQ,
the phone reboots if it has previously successfully booted the modem.
Later, mode=0 requests “secure” mode, which is what causes the phone
to reboot unless mode=2 is selected. So ultimately
IOCTL_SECURITY_REQ with mode=2 is mandatory, because otherwise
you’ll end up crashing hard each time you restart the modem daemon.)
Upload three binary blob chunks. In order, send the BOOT and
MAIN blobs from the firmware table-of-contents, followed by the
contents of the nv_data.bin file on the EFS partition.
Sending a blob is a slightly involved process. Repeat the following
steps until you run out of blob to upload:
Read the next chunk into memory. The stock cbd confines itself
to chunks of 62k (yes, 62k, not 64k) or smaller. The kernel
doesn’t appear to care, but why mess with success?
The binary field should be the address in RAM of the start of
the chunk you just read.
The size field is the total size of the blob being uploaded.
The m_offset field is an offset into the chunk of RAM
reserved on the kernel side for uploaded blobs. The BOOT blob
goes in at offset 0, so its load_addr field from the firmware
partition’s table-of-contents, which for me was 0x40000000,
corresponds to m_offset 0.
As another example, for the first 62k chunk of the MAIN blob,
which has load_addr 0x40010000, m_offset should be 0x10000,
since MAIN’s load_addr is 0x10000 greater than BOOT’s
load_addr. For the second 62k chunk, m_offset should be
0x1f800, and so on.
The b_offset field isn’t currently used by the kernel, but I
suspect that cbd fills it in anyway, so I do too: I set it to
the offset within the firmware partition of the beginning of
the chunk being uploaded.
The mode field is interpreted by the kernel simply on a
zero/nonzero basis, despite some hints elsewhere that valid
values are 0, 1 and 2. Set mode=0 for all the uploaded
chunks, since this is what cbd does.
Finally, the len field is the length of the chunk to be
Issue an ioctl IOCTL_MODEM_XMIT_BOOT to boot0_fd, with argument
a pointer to the modem_firmware descriptor you just filled in.
Note well that the chunks are to be read out of the firmware
partition, following the appropriate TOC entries, for the BOOT and
MAIN blobs, but chunks are to be read from nv_data.bin on the
EFS partition, and not anywhere in the RADIO partition, for the
Issue a second “security request”. This time, send
IOCTL_SECURITY_REQ (0x6f53) with mode=0, size_boot set to the size
of the BOOT blob from the TOC, and size_main set to the size of the
MAIN blob. For me, those values were 9572 and 40027244,
respectively. According to cbd’s diagnostics, this is asking for
Tell the modem to power on. This interacts with power management
code on the kernel side somehow. Issue ioctl IOCTL_MODEM_ON (0x6f19)
Tell the modem to start its boot sequence. Issue
IOCTL_MODEM_BOOT_ON (0x6f22) to boot0_fd.
Tell the kernel to forward the firmware blobs to the modem. Issue
IOCTL_MODEM_DL_START (0x6f28) to boot0_fd.
At this point, cbd engages in a little dance with the newly-booted
modem, apparently to verify that it is running as expected. I don’t
know the sources of these magic numbers, I just faithfully reproduce
Write 0D 90 00 00 to boot0_fd.
Read back four bytes. Expect them to be 0D A0 00 00.
Write 00 9F 00 00 to boot0_fd.
Read back four bytes. Expect them to be 00 AF 00 00.
Tell the modem the boot sequence is complete. Issue
IOCTL_MODEM_BOOT_OFF (0x6f23) to boot0_fd.
Close boot0_fd, and release the wake-lock. At this point the modem
is booted! Congratulations!
After finishing the boot procedure, cbd goes into a loop apparently
waiting for administrative messages from the modem. It does this by
opening /dev/umts_boot0 again and reading from it. My script
does the same.
I don’t know what cbd does with the results: I’ve yet to see any
information come out of the modem this way.
To interact with the modem, open /dev/umts_ipc0 and
/dev/umts_rfs0. IPC stands for the usual Inter-Process
Communication, but RFS apparently (?) stands for Remote File System.
Most modem interaction happens over the IPC channel. I don’t really
know what the RFS channel is for yet.
Each packet sent to or from /dev/umts_ipc0 is formatted as a
that includes its own length, making parsing easy. Simply read and
write a series of sipc_fmt_hdrs (with appropriate body bytes tacked
on after each) from and to /dev/umts_ipc0.
Decoding them is another matter entirely! The libsamsung-ril
library does this well. However, a little bit more information can be
gleaned by matching bytes sent and received by rild to the
diagnostic outputs it produces. Here’s a lightly-reformatted snippet
of straced output of Samsung’s rild:
Analysing the first seven bytes (the sipc_fmt_hdr) send to fd 18, we
see len=11, msg_seq=0xd3, ack_seq=0, main_cmd=2, sub_cmd=2,
From the log message it prints, we can deduce that CALL_CMD=2,
CALL_INCOMING=2, and NOTI=3. This lines up well with the definitions
in libsamsung-ril, and it turns out that by observing the modem in
operation you can learn a few more definitions not included in
Perhaps a good next step would be to translate this knowledge into
support for the Galaxy S7 in libsamsung-ipc; for now, I don’t need
that for myself, but it’s probably the Right Thing To Do. I’ll see if
they’re interested in taking such a contribution.
Actually the information about “insecure” and
“secure” IOCTL_SECURITY_REQ comes from a dmesg trace uploaded by
someone anonymous to a pastebin I cannot find again. My own cbd
doesn’t seem to produce these diagnostics. Sorry. ↩
Here’s a quick demo video of the status of my quixotic project to get
Squeak running as a kind of standalone userland on a modern cellphone:
(The sound is bad! I recorded it on my other cellphone…)
Currently, I develop by coding in Squeak on my Linux desktop, using my
graphics tablet as a kind of proxy for a touchscreen. I use the FFI
and AIO/OSProcess support in Squeak to read events from
/dev/input/event.... For event sources that present absolute axes, I
create instances of HandMorph in the World and animate them
according to the incoming events.
Every now and then, to test on the real hardware, I use rsync to
copy the changes and image files up to the cellphone, and then log in
to the phone over ssh to restart the Cog VM.
At the moment, the image has enough smarts to figure out how to read
the touchscreen and offer basic touchscreen click support. This lets
me do simple things like open, move and close windows, and lets me
save and/or quit the image.
Next steps are to make it harder to misclick – perhaps by increasing
the size of some of the touch targets – and to think about coding up
a simple onscreen keyboard.
Alternatively, on a parallel path, I’ve been reverse-engineering
(really nothing more sophisticated than strace of cbd and rild)
the Samsung protocols for booting and operating the cellular modem.
The code is short and simple. Perhaps instead of an onscreen keyboard
I’ll code up a quick dialer Morph and get Squeak making phone calls.
Back in 2007, when Openmoko
was first a thing, I
wrote an Erlang-based userland
that got to the point of being able to take and make calls and receive
and send SMS. The project stalled: the Openmoko GTA01 was too slow,
its power-management too primitive, and Erlang’s GUI facilities too
rudimentary to make further work worthwhile.
Modern cellphone hardware is much more capable. Is it time to have
another run at the idea of a mobile personal computer?
PostmarketOS is awesome
Last week, I installed PostmarketOS on my
previous cellphone, a Samsung Galaxy S7 (using PostmarketOS’s
PostmarketOS turns out to be a beautifully engineered system that’s
easy to understand and modify. The basics of kernel and Alpine Linux
userland installed cleanly and easily on the phone, and it’s running
well as a development platform. I’m looking forward to getting into
PostmarketOS more deeply.
Running htop on the phone shows what an amazing little machine it
is! So much power. Loads of cores, lots of RAM. Plenty of space to
explore alternative visions of mobile personal computing.
However, the built-in demos, such as the
Weston demo (shown above
at right), currently leave quite a bit to be desired. Perhaps some of
user interface options
included with PostmarketOS could get me closer to a day-to-day usable
cellphone - but I’m interested in running my own software! Let’s get
Running my own programs
PostmarketOS is a plain, clean Alpine Linux distribution. You can SSH
into it initially
via USB networking.
From there, you can
configure wifi using nmcli,
set up SSH keys, and then access it directly using SSH over wifi.
Building software is just as simple:
To experiment with drawing to the framebuffer and reading touchscreen
input via /dev/input, I compiled and ran an old
quick and dirty framebuffer hack
I wrote years ago. The results (shown at left) were encouraging: the
program effortlessly animates tens of thousands of points at 30 frames
per second, responding to touch inputs. Display is via brute-force
pixel output to the mmap‘d frame buffer. It doesn’t even use a full
PostmarketOS turns a phone into a fully capable Linux machine, with
total control over the attached hardware, and with everything
accessible to the developer in the usual places using the usual tools.
But Unix tools are inappropriate for a mobile personal computing
platform. We’ll need something else.
A Smalltalk phone
Smalltalk could make an ideal basis for a mobile personal computing
From here, I can experiment with new ideas using the full power of a
modern Smalltalk environment.
My previous Openmoko experiments foundered, in part, on the GUI aspect
of the system; GTK+ via Erlang was fine for quick prototyping but
wasn’t really up to the task for a day-to-day usable machine.
I recall getting Squeak running on my GTA01, in order to see if it
could provide a viable UI. However, I remember being stymied by the
mismatch between the expectations of the Smalltalk environment and the
realities of the phone.
Squeak wants a mouse and keyboard. It assumes a monitor-sized display,
in everything from widget and font sizes to window management. To work
well on a phone, it needs a touchscreen-based, high-DPI UI in addition
to its existing toolset.