eighty-twenty

Part of a series: #squeak-phone

Back in October 2020, I built an on-screen keyboard for Squeak Smalltalk. It ended up being about 230 lines of code (!) in total.

I’ve been using Smalltalk as the primary UI for my experimental cellphone software stack, and I needed a way to type input. Using VNC to develop on the phone works fine, but to work on the device itself - both for day-to-day phone tasks as well as developing the system - I need a proper on-device keyboard.

This video shows the keyboard in action. As you can see, it’s not perfect! At one point I tapped ctrl while typing, leading to an unexpected pop-up menu appearing.

But the basic idea is sound. (Aside: why aren’t there any Android keyboards that are laid out like a PC keyboard, with punctuation in the right place, etc.?)

The usual shortcuts like Alt-P for “print it”, Alt-D for “do it” and Alt-I for “inspect it” all work for evaluating snippets of Smalltalk interactively.

The keyboard integrates with my previously-written touchscreen support code, with the red/blue/yellow modifier buttons affecting touches, making them into simulated left-/middle-/right-clicks, and with keyboard focus changes auto-popping-up and -down the keyboard.

Simulating mouse clicks is a temporary state of affairs that lets me use the touchscreen reasonably naturally, making use of context menus and so on, without having to make invasive changes to the input pipeline of Squeak’s Morphic.

How it works

Class OnScreenKeyboardMorph synthesizes keyboard events and injects them into the Morphic world.

OnScreenKeyboardMorph >> emitKeystroke: aCharacter
    | evt |
    evt := KeyboardEvent new
        setType: #keystroke
        buttons: modifiers
        position: 0@0
        keyValue: aCharacter asciiValue
        hand: ActiveHand
        stamp: Time millisecondClockValue.
    self resetModifiers.
    ActiveHand handleEvent: evt.

You can have arbitrarily many keyboards instantiated, but there’s a global one living in a flap at the bottom of the screen. That’s the blue tab at the bottom left of the screen you can see in the video.

OnScreenKeyboardMorph class >> rebuildFlap
    | f k |
    self flap ifNotNil: [:old |
        Flaps removeFlapTab: old keepInList: false.
        ActiveWorld reformulateUpdatingMenus].

    k := self new.
    f := FlapTab new referent: k beSticky.
    f setName: self flapId edge: #bottom color: Color blue lighter.
    k beFlap: true.

    Flaps addGlobalFlap: f.
    ActiveWorld addGlobalFlaps.
    ActiveWorld reformulateUpdatingMenus.

I had already written code to read from /dev/input/eventN, tracking each multitouch contact separately. A subclass of HandMorph overrides newKeyboardFocus: to pop the keyboard up and down as the keyboard focus comes and goes:

LinuxInputHandMorph >> newKeyboardFocus: aMorphOrNil
    aMorphOrNil
        ifNil: [OnScreenKeyboardMorph hideFlap]
        ifNotNil: [(OnScreenKeyboardMorph future: 200) raiseFlap].
    ^ super newKeyboardFocus: aMorphOrNil.

Each key is represented by an OnScreenKeyMorph. It tracks keyboard shift state by registering as a dependent of the OnScreenKeyboardMorph:

OnScreenKeyMorph >> keyboard: anOnScreenKeyboardMorph
    keyboard := anOnScreenKeyboardMorph.
    keyboard addDependent: self.

OnScreenKeyMorph >> update: aParameter
    aParameter = #keyboardShiftState ifTrue: [^ self recomputeLabel].
    super update: aParameter.

The important part, of course, is the mouse event handler for OnScreenKeyMorph:

OnScreenKeyMorph >> mouseDown: evt
    selector
        ifNil: [keyboard emitKeystroke: self activeLabel first]
        ifNotNil: [keyboard perform: selector]

The only other interesting part is the keyboard initialization routine, which builds the layout:

OnScreenKeyboardMorph >> initialize
    super initialize.
    self borderWidth: 0.
    self extent: 180 points @ 120 points.
    self layoutPolicy: TableLayout new.
    self wantsHaloFromClick: false.

    font := (TextStyle named: #Roboto) fontOfPointSize: 18.
    modifiers := 0.
    stickyModifiers := 0.
    buttons := Dictionary new.

    {
        '`~ 1! 2@ 3# 4$ 5% 6^ 7& 8* 9( 0) -_ =+' substrings, {'<-' -> #backspace}.
        {'->' -> #tab}, 'qQ wW eE rR tT yY uU iI oO pP [{ ]}' substrings.
        {'esc' -> #escape}, 'aA sS dD fF gG hH jJ kK lL ;: ''" \|' substrings, {#cr}.
        {#shift}, 'zZ xX cC vV bB nN mM ,< .> /?' substrings.
        {#ctrl. #space. #alt}.
        {'|<-' -> #home. '<' -> #arrowLeft. 'v' -> #arrowDown. '^' -> #arrowUp. '>' -> #arrowRight. '->|' -> #end.}.
    } do: [:r | self addMorphBack: (self makeRow: (r collect: [:item | self makeButton: item]))].
    self addMorphBack:
        (self makeRow:
            {
                OnScreenMouseButtonMorph new bit: 4.
                OnScreenMouseButtonMorph new bit: 2.
                OnScreenMouseButtonMorph new bit: 1.
            }).

The rest is book-keeping and simple delegation methods, like this:

OnScreenKeyboardMorph >> shift
    self toggleButton: 8

OnScreenKeyboardMorph >> space
    self emitKeystroke: Character space

and so on.

I haven’t released the whole package yet, because it’s all very much in flux, but if you’re interested, you can take a look at the changeset for the Keyboard-related code here: LinuxIO-Morphic-KeyboardMorph.st

UPDATE: Full code available at this gist (also embedded below).

Say you have the following TypeScript interface I that you want to invoke remotely by passing messages of type M; or that you receive messages of type M and want to handle them using an object of type I:

interface I {
    m1(a: string, b: number): boolean;
    m2(): void;
    m3(n: number): void;
    m4(x: [string, string]): { k: string, j: string };
    m5(a: string, b: string[]): number;
}

type M = { selector: "m1", args: [string, number], callback: (result: boolean) => void }
       | { selector: "m2", args: [], callback: (result: void) => void }
       | { selector: "m3", args: [number], callback: (result: void) => void }
       | { selector: "m4", args: [[string, string]], callback: (result: { k: string; j: string }) => void }
       | { selector: "m5", args: [string, string[]], callback: (result: number) => void }

Keeping things type-safe looks really tedious! There’s obviously a connection between I and M. Can we avoid writing them by hand?

Can we derive M from I? Can we derive I from M?

The answer to all of these questions is yes!¹

TL;DR TypeScript lets us define generic types Messages and Methods such that M = Messages<I> and I = Methods<M>. Read on for the details.

Interface ⟶ Messages

Let’s start with what, for me, has been the common case: given an interface type, automatically produce the type of messages that can be sent to implementors of the interface.

First, how do we want to represent messages?

type Message<Selector extends ValidSelector, Args extends any[], Result> =
    Args extends never[]
    ? { selector: Selector, args: [], callback: (result: Result) => void }
    : { selector: Selector, args: Args, callback: (result: Result) => void }

type ValidSelector = string | number | symbol

I’ve taken a leaf out of Smalltalk’s book, and made a message include a selector, the name of the method the message intends to invoke, and some args, the provided arguments to the method. The Args extends never[] check is to help type inference deduce the empty argument tuple: without it, the type system won’t complain about missing arguments.

I’ve also added a callback to Message. The technique I describe here can be further extended to “asynchronous” or callbackless settings with minor modifications.

The next definition, of type Messages<I>, is where the magic happens. It expands to a union of Messages representing the methods defined in I:

type Messages<I> = MessagesProduct<I>[keyof I]
type MessagesProduct<I> = {
    [K in keyof I]: (I[K] extends (...args: infer P) => infer Q ? Message<K, P, Q> : never);
}

And that’s it! Here’s how it works:

• MessagesProduct is a mapped type that describes a modified interface, where all (and only) the method properties of interface I are rewritten to have a Message as their type, but keeping the same key;

• then, the ...[keyof I] part in the definition of Messages uses index types to set up a union type built from all the value types (“indexed access operator”) associated with all the keys in I (“index type query operator”).

Messages ⟶ Interface

Going in the other direction is simpler:

type Methods<M extends { selector: ValidSelector }> = {
    [S in M['selector']]: (
        M extends Message<S, infer P, infer R> ? (...args: P) => R :
        never);
}

It’s a mapped type, again, that maps union members that have Message type to an appropriate function signature. It takes advantage of TypeScript’s automatic distributivity: a union of products gets rewritten to be a product of unions. Then, in the conditional type M extends Message<...> ? ..., it projects out just exactly the member of interest again.²

This time we use the mapped type as-is instead of re-projecting it into a union using indexed access like we did with MessagesProduct above.

Type-safe interpretation of messages

Now we have types for our interfaces, and types for the messages that match them, can we write a type-safe generic perform function? Yes, we can!

function perform<I extends Methods<M>,
                 S extends ValidSelector,
                 M extends Message<S, any, any>>(i: I, m: M): void
{
    m.callback(i[m.selector](... m.args));
}

An example

Given the above definition for I, actually using Messages<I> produces the following type definition³ (according to the IDE that I use):

type M = Message<"m1", [a: string, b: number], boolean>
       | Message<"m2", [], void>
       | Message<"m3", [n: number], void>
       | Message<"m4", [x: [string, string]], { k: string; j: string }>
       | Message<"m5", [a: string, b: string[]], number>

Conversely, given the M from the top of the file, we get the following for Methods<M>:

type I = {
    m1: (a: string, b: number) => boolean;
    m2: () => void;
    m3: (n: number) => void;
    m4: (x: [string, string]) => { k: string; j: string };
    m5: (a: string, b: string[]) => number;
}

Roundtripping works too: both Methods<Messages<I>> and Messages<Methods<M>> give what you expect.

TypeScript is really cool

It’s a fully-fledged, ergonomic realization of the research started by Sam Tobin-Hochstadt, who invented Occurrence Typing, the technology at the heart of TypeScript.

Then, building on the language itself, emacs with tide, flycheck, and company makes for a very pleasant IDE.⁴

Congratulations to Sam, whose ideas really have worked out amazingly well, and to the TypeScript team for producing such a polished and pleasant language.

Appendix: Full code implementing this idea

This module implements the idea described in this article, extended with the notion of EventMessages, which don’t have a callback.

// This Tuple type (and tuple() function) is a hack to induce
// TypeScript to infer tuple types rather than array types. (Source:
// https://github.com/microsoft/TypeScript/issues/27179#issuecomment-422606990)
//
// Without it, [123, 'hi', true] will often get the type (string |
// number | boolean)[] instead of [number, string, boolean].
//
export type Tuple = any[] | [any];
export const tuple = <A extends Tuple>(... args: A) => args;

// Type ValidSelector captures TypeScript's notion of a valid object
// property name.
//
export type ValidSelector = string | number | symbol;

export type EventMessage<Selector extends ValidSelector, Args extends any[]> =
    { selector: Selector, args: Args };

export type RequestMessage<Selector extends ValidSelector, Args extends any[], Result extends Exclude<any, void>> =
    { selector: Selector, args: Args, callback: (result: Result) => void };

export type Message<Selector extends ValidSelector, Args extends any[], Result> =
    void extends Result ? EventMessage<Selector, Args> : RequestMessage<Selector, Args, Result>;

// Function message() is needed for similar reasons to tuple() above:
// to help TypeScript infer the correct literal type for the selector
// (as well as the arguments).
//
export const message = <S extends ValidSelector, A extends Tuple, R>(m: Message<S, A, R>) => m;

type MessagesProduct<I, ContextArgs extends any[]> = {
    [K in keyof I]: (I[K] extends (...args: [...ContextArgs, ...infer P]) => infer Q
        ? Message<K, P, Q>
        : never);
};

export type Messages<I, ContextArgs extends any[] = []> = MessagesProduct<I, ContextArgs>[keyof I];

export type Methods<M extends { selector: ValidSelector }, ContextArgs extends any[] = []> = {
    [S in M['selector']]: (
        M extends RequestMessage<S, infer P, infer R>
            ? (void extends R ? never : (...args: [...ContextArgs, ...P]) => R)
            : (M extends EventMessage<S, infer P>
                ? (...args: [...ContextArgs, ...P]) => void
               : never));
};

export function perform<I extends Methods<M, ContextArgs>,
                        S extends ValidSelector,
                        M extends RequestMessage<S, Tuple, any>,
                        ContextArgs extends any[] = []>
    (i: I, m: M, ...ctxt: ContextArgs): (M extends RequestMessage<S, Tuple, infer R> ? R : never);
export function perform<I extends Methods<M, ContextArgs>,
                        S extends ValidSelector,
                        M extends EventMessage<S, Tuple>,
                        ContextArgs extends any[] = []>
    (i: I, m: M, ...ctxt: ContextArgs): void;
export function perform<I extends Methods<M, ContextArgs>,
                        S extends ValidSelector,
                        M extends RequestMessage<S, Tuple, any>,
                        ContextArgs extends any[] = []>
    (i: I, m: M, ...ctxt: ContextArgs): any
{
    const r = i[m.selector](...ctxt, ... m.args);
    m.callback?.(r);
    return r;
}

Recently on HN, rbanffy brought up a form of the old chestnut about crashing the image by simply executing true become: false.

It turns out it’s no longer true!

In the latest Squeak,

True := False.

doesn’t work – the compiler complains that you can’t assign into a read-only variable. So let’s try this:

Smalltalk at: #True put: False.

But now the metaprogramming system complains you’re trying to modify a read-only binding! So we view source on ClassBinding»value:, and see that a resumable exception is being used to guard the modification, so let’s explicitly signal that we really want to modify that binding:

[Smalltalk at: #True put: False]
  on: AttemptToWriteReadOnlyGlobal
  do: [:ex | ex resume: true].

Finally! Now, evaluating True yields False.

But the image keeps running! Use of the literal class True seems to be rare enough that things are OK for at least several minutes after the change.

Doing this, however, definitely should immediately torpedo things:

true become: false.

… huh. It didn’t work. It used to! Again, on this current Squeak version, we see a different behaviour. This time, it says Cannot execute #elementsExchangeIdentityWith: on read-only object #(false).

So we’ll have to try harder:

true becomeForward: false.

That doesn’t work either! Same error as for #become:.

Welp, I’m actually all out of ways to crash this modern image in analogous ways to the easy pitfalls of images of yesteryear…

P.S. Running the following command,

ln -sf /dev/zero /lib/x86_64-linux-gnu/libc.so.6

analogous to True := False, definitely does have an analogous effect on a similarly-“alive” Unix image :-) . I just tried it on a scratch VM; the results are pretty intimidating!

Part of a series: #squeak-phone

One lovely thing about working in Smalltalk is the effortlessness of development.

I started off developing the code on my desktop machine, and occasionally testing it on the phone itself. I just rsync the image and changes files back and forth. This lets me pick up exactly where I left off on the other device each time I move over.

However, developing on the phone was challenging because of the lack of a keyboard (though I’ll post soon about an on-screen keyboard I’ve written). So I installed RFB (from here) into my image on the desktop, and tested it. Then I saved the image and rsynced it to the phone as usual, and presto, I can develop and test interactively on the phone itself:

Using VNC to develop on the phone itself

There were a couple of things I had to do to get this to work:

• Use this version of RFBServer: http://source.squeak.org/ss/

• Change AllowTcpForwarding no to yes in /etc/ssh/sshd_config on the phone and then service sshd restart

• Use ssh -L 5900:localhost:5900 pm to log into the phone (that’s the green-screen transcript in the background in the picture above)

Part of a series: #squeak-phone

Hand-written binary parsing/unparsing sucks

As I’ve been working on a mobile Smalltalk system, I’ve found myself needing to decode and encode a number of complex telephony packet formats¹ such as the following, an incoming SMS delivery message containing an SMS-DELIVER TPDU in GSM 03.40 format, containing seven-bit (!) GSM 03.38-encoded text:

02 01 ffff
01 28 07911356131313f3
04 0b911316325476f8 000002909021044480 0ec67219644e83cc6f90b9de0e01

It turns out there are a plethora of such binary formats needed to get a working cellphone.

I started off hand-rolling them, but it quickly became too much, so I borrowed liberally stole from Erlang, and implemented BitSyntax for Smalltalk. (After all, I am already using Erlang-influenced actors for the Smalltalk system daemons!)

I’ve done this before, for Racket, and there are plenty of other similar projects for e.g. JavaScript and OCaml.

Every language needs a BitSyntax, it seems!

What does BitSyntax do?

The BitSyntax package includes a BitSyntaxCompiler class which interprets BitSyntaxSpecification objects, producing reasonably efficient Smalltalk for decoding and encoding binary structures, mapping from bytes to instance variables and back again.

The interface to the compiled code is simple. After compiling a BitSyntaxSpecification for the data format above, we can analyze the example message straightforwardly:

parsedMessage := SmsIncoming loadFrom: (ByteArray fromHex:
    '02 01 ffff
     01 28 07911356131313f3
     04 0b911316325476f8 000002909021044480 0ec67219644e83cc6f90b9de0e01')

and, if we wish, serialize it again:

serializedBytes := ByteArray streamContents: [:w | parsedMessage saveTo: w]

How does it work?

Syntax specifications are built using an embedded domain-specific language (EDSL).

For example, for the above data format, we would supply the following spec for class SmsIncoming:

        (1 byte >> #msgType),
        (1 byte >> #type),
        (2 bytesLE >> #simIndex),
        (1 byte >> #id),
        ((1 byte storeTemp: #payloadLength expr: 'payload size'), 'payloadLength' bytes)
            >>> #payload <<<
            (
                (SmsAddress codecCountingOctets >> #smscAddress),
                (SmsPdu codecIncoming >> #tpdu))

along with appropriate specs for SmsAddress and SmsPdu (omitted for space reasons here) and the following for the SmsPdu subclass SmsPduDeliver:

        (1 bit boolean >> #replyPath),
        (1 bit boolean >> #userDataHeaderIndicator),
        (1 bit boolean >> #statusReportIndication),
        (2 bits),
        (1 bit boolean >> #moreMessagesToSend),
        (2 bits = 0),

        (SmsAddress codecCountingSemiOctets >> #originatingAddress),
        (1 byte >> #protocolIdentifier),
        (1 byte >> #dataCodingScheme),
        ((7 bytes
                transformLoad: [:v | 'self class decodeSmscTimestamp: ', v]
                save: [:v | 'self class encodeSmscTimestamp: ', v])
            >> #serviceCentreTimeStamp),
        (((1 byte >> #itemCount)
            transformLoad: [:v | 'self userDataOctetsFor: ', v]
            save: [:v | 'itemCount'])
                storeTemp: #userDataLength expr: 'userData size'),
        (#userDataLength bytes >> #userData)

These are non-trivial examples; the simple cases are simple, and the complex cases are usually possible to express without having to write code by hand. The EDSL is extensible, so more combinators and parser types can be easily added as the need arises.

How do I get it?

Load it into an up-to-date trunk Squeak image:

(Installer squeaksource project: 'BitSyntax')
    install: 'BitSyntax-Core';      "the compiler and EDSL"
    install: 'BitSyntax-Examples';  "non-trivial examples"
    install: 'BitSyntax-Help'.      "user guide and reference"

You can also visit the project page directly.

The package BitSyntax-Help contains an extensive manual written for Squeak’s built-in documentation system.

Enjoy!

Today I noticed that the keyboard on my 2013-vintage Acer C720 chromebook was visibly curved, being pushed up from below by the battery, which had swollen enormously.

This can be really dangerous, so I took the back off the machine and removed the battery. Here’s what it looks like:

Yikes!

Changing the battery is extremely quick and easy, and replacement batteries can be had for about USD40/EUR40 delivered. I’ve already ordered a replacement.

Part of a series: #squeak-phone

I’ve been running PostmarketOS on my Samsung Galaxy S7 recently. As far as I can tell, none of the available modem/telephony stacks supports the Galaxy S7 modem.

I was expecting to be able to just open /dev/ttyACM0 or similar and speak AT commands to it, like I used to be able to on Openmoko.

However, Samsung phones have a proprietary, undocumented¹, unstandardized binary interface to their modems. Operating a modem is the same across members of the Samsung family, but each different handset seems to have a different procedure for booting the thing.

So I decided to reverse engineer (a fancy name for “running strace on cbd and rild and reading a lot of kernel source code”) the protocol.

The result is a couple of quick-and-dirty python scripts which, together, boot the modem and print out the messages it sends us. It’d be straightforward to extend it to, for example, send SMS, or to manage incoming and outgoing calls.

How Android (LineageOS) does it

LineageOS uses Samsung’s proprietary cbd and rild programs, extracted as binary blobs from the stock firmware.

The rild program implements the Android Radio Interface Layer (RIL), talking directly to the modem over /dev/umts_ipc0.

The cbd program performs the modem boot and reset sequences, and is started after rild is running by Android’s init. It talks to the modem over /dev/umts_boot0.

The cbd program relies on the availability of the Android RADIO partition, which contains the modem firmware, as well as on the existence of a file nv_data.bin stored on the phone’s EFS partition.

My S7, running LineageOS, started its cbd with the following command-line:

/sbin/cbd -d -tss310 -bm -mm -P platform/155a0000.ufs/by-name/RADIO

Here’s the “help” output of cbd, that we can use to decode this:

CP Boot Daemon
Usage: ./cbd [OPTION]
 -h Usage
 -d Daemon mode
 -t modem Type [cmc22x, xmm626x, etc.]
 -b Boot link [d, h, s, u, p, c, m, l]
 	 d(DPRAM) h(HSIC) s(SPI) u(UART) p(PLD) c(C2C) m(shared Memory) l(LLI)
 -m Main link [d, h, s, p, c, m, l]
 	 d(DPRAM) h(HSIC) s(SPI) p(PLD) c(C2C) m(shared Memory) l(LLI)
 -o Options [u, t, r]
 	 u(Upload test) t(Tegra EHCI) r(run with root)
 -p Partition# of CP binary

The command line, then, tells us a few interesting things:

• the modem is an “SS310”.
• it uses shared memory (corresponding to drivers/misc/modem_v1/link_device_shmem.c in the kernel source code) to communicate
• the modem firmware partition name is RADIO
• there’s an undocumented flag, -P (cf. the documented -p) which lets you supply a partition path fragment rather than a partition number

That last is crucial for running the same binary on PostmarketOS, which has a different layout of the files in /dev.

How libsamsung-ipc does it

The open-source libsamsung-ipc from the Replicant project handles the boot and communication processes for a number of (older?) Samsung handsets, but not the Galaxy S7. Once a modem is booted, libsamsung-ipc passes higher-level protocol messages back and forth between the modem and libsamsung-ril, which layers Android telephony support atop the device-independent abstraction that libsamsung-ipc provides.

Generally, modems are booted by a combination of ioctl calls and uploads of firmware blobs, with specific start addresses and blob layouts varying per modem type.

At first, I was concerned that I wouldn’t have enough information to figure out the necessary constants for the S7 modem. However, luckily just strace combined with dmesg output and kernel source code was enough to get it working.

Booting the modem - qnd_cbd.py

The overall sequence is similar to, but not quite the same as, other Samsung models already supported by libsamsung-ipc.

Running strace on Samsung’s cbd yields the following steps.

Extract the firmware blobs. The firmware partition has a table-of-contents that has a known structure. From here, we can read out the chunks of data we will need to upload to the modem.²

Acquire a wake-lock. I don’t understand the Android wake-lock system, but during modem boot, cbd acquires the ss310 wakelock.

Open /dev/umts_boot0. This character device is the focus of most of the subsequent activity. I’ll call the resulting file descriptor boot0_fd below.

Issue a modem reset ioctl. Send IOCTL_MODEM_RESET (0x6f21) to boot0_fd.

Issue a “security request”. Whatever that is! Send IOCTL_SECURITY_REQ (0x6f53) with mode=2, size_boot=0 and size_main=0 (like this). According to cbd’s diagnostics³, this is asking for “insecure” mode; the same ioctl will be used later to enter a “secure” mode.

One interesting thing about this particular call to IOCTL_SECURITY_REQ is that it answers error status 11 if you run it as root. The Samsung cbd does a prctl(PR_SET_KEEPCAPS, 1)/setuid(1001) just before IOCTL_SECURITY_REQ, which appears to give a happier result of 0 from the ioctl. However, if you ask cbd to stay as root by supplying the command-line flag -or to it, then it too gets error status 11 from the ioctl. Fortunately, the error code seems to be ignorable and the modem seems to boot successfully despite it! This makes me think that perhaps running IOCTL_SECURITY_REQ at this point, in this way, is optional. (UPDATE. Here’s what I think is going on. Setting mode=2 apparently asks for “insecure” mode, which allows uploading of firmware chunks. If we omit the mode=2 call to IOCTL_SECURITY_REQ, the phone reboots if it has previously successfully booted the modem. Later, mode=0 requests “secure” mode, which is what causes the phone to reboot unless mode=2 is selected. So ultimately IOCTL_SECURITY_REQ with mode=2 is mandatory, because otherwise you’ll end up crashing hard each time you restart the modem daemon.)

Upload three binary blob chunks. In order, send the BOOT and MAIN blobs from the firmware table-of-contents, followed by the contents of the nv_data.bin file on the EFS partition.

Sending a blob is a slightly involved process. Repeat the following steps until you run out of blob to upload:

1. Read the next chunk into memory. The stock cbd confines itself to chunks of 62k (yes, 62k, not 64k) or smaller. The kernel doesn’t appear to care, but why mess with success?

2. Prepare a struct modem_firmware descriptor, pointing to the chunk.

   1. The binary field should be the address in RAM of the start of the chunk you just read.

   2. The size field is the total size of the blob being uploaded.

   3. The m_offset field is an offset into the chunk of RAM reserved on the kernel side for uploaded blobs. The BOOT blob goes in at offset 0, so its load_addr field from the firmware partition’s table-of-contents, which for me was 0x40000000, corresponds to m_offset 0.

      As another example, for the first 62k chunk of the MAIN blob, which has load_addr 0x40010000, m_offset should be 0x10000, since MAIN’s load_addr is 0x10000 greater than BOOT’s load_addr. For the second 62k chunk, m_offset should be 0x1f800, and so on.

   4. The b_offset field isn’t currently used by the kernel, but I suspect that cbd fills it in anyway, so I do too: I set it to the offset within the firmware partition of the beginning of the chunk being uploaded.

   5. The mode field is interpreted by the kernel simply on a zero/nonzero basis, despite some hints elsewhere that valid values are 0, 1 and 2. Set mode=0 for all the uploaded chunks, since this is what cbd does.

   6. Finally, the len field is the length of the chunk to be uploaded.

3. Issue an ioctl IOCTL_MODEM_XMIT_BOOT to boot0_fd, with argument a pointer to the modem_firmware descriptor you just filled in.

Note well that the chunks are to be read out of the firmware partition, following the appropriate TOC entries, for the BOOT and MAIN blobs, but chunks are to be read from nv_data.bin on the EFS partition, and not anywhere in the RADIO partition, for the NV blob.

Issue a second “security request”. This time, send IOCTL_SECURITY_REQ (0x6f53) with mode=0, size_boot set to the size of the BOOT blob from the TOC, and size_main set to the size of the MAIN blob. For me, those values were 9572 and 40027244, respectively. According to cbd’s diagnostics, this is asking for “secure” mode.

Tell the modem to power on. This interacts with power management code on the kernel side somehow. Issue ioctl IOCTL_MODEM_ON (0x6f19) to boot0_fd.

Tell the modem to start its boot sequence. Issue IOCTL_MODEM_BOOT_ON (0x6f22) to boot0_fd.

Tell the kernel to forward the firmware blobs to the modem. Issue IOCTL_MODEM_DL_START (0x6f28) to boot0_fd.

At this point, cbd engages in a little dance with the newly-booted modem, apparently to verify that it is running as expected. I don’t know the sources of these magic numbers, I just faithfully reproduce them:

1. Write 0D 90 00 00 to boot0_fd.
2. Read back four bytes. Expect them to be 0D A0 00 00.
3. Write 00 9F 00 00 to boot0_fd.
4. Read back four bytes. Expect them to be 00 AF 00 00.

Tell the modem the boot sequence is complete. Issue IOCTL_MODEM_BOOT_OFF (0x6f23) to boot0_fd.

Close boot0_fd, and release the wake-lock. At this point the modem is booted! Congratulations!

Monitoring the modem - qnd_cbd.py

After finishing the boot procedure, cbd goes into a loop apparently waiting for administrative messages from the modem. It does this by opening /dev/umts_boot0 again and reading from it. My script does the same. I don’t know what cbd does with the results: I’ve yet to see any information come out of the modem this way.

Interacting with the modem - qnd_ril.py

To interact with the modem, open /dev/umts_ipc0 and /dev/umts_rfs0. IPC stands for the usual Inter-Process Communication, but RFS apparently (?) stands for Remote File System.

Most modem interaction happens over the IPC channel. I don’t really know what the RFS channel is for yet.

Each packet sent to or from /dev/umts_ipc0 is formatted as a struct sipc_fmt_hdr that includes its own length, making parsing easy. Simply read and write a series of sipc_fmt_hdrs (with appropriate body bytes tacked on after each) from and to /dev/umts_ipc0.

Decoding them is another matter entirely! The libsamsung-ril library does this well. However, a little bit more information can be gleaned by matching bytes sent and received by rild to the diagnostic outputs it produces. Here’s a lightly-reformatted snippet of straced output of Samsung’s rild:

[pid  3969] 13:25:51.716591
  read(18</dev/umts_ipc0>,
       "\x0b\x00\xd3\x00\x02\x02\x03\x00\x01\x00\x01",
       264192) = 11 <0.000448>

[pid  3969] 13:25:51.718454
  writev(3<socket:[2555]>,
         [
           {"\x01\x81\x0f?jW_\x12\xc0\xc7*", 11},
           {"\x06", 1},
           {"RILD\x00", 5},
           {"[G] RX: (M)CALL_CMD (S)CALL_INCOMING (T)NOTI l:b m:d3 a:00 [ 00 01 00 01 ]\x00", 75}
         ],
         4) = 92 <0.000528>

Analysing the first seven bytes (the sipc_fmt_hdr) send to fd 18, we see len=11, msg_seq=0xd3, ack_seq=0, main_cmd=2, sub_cmd=2, cmd_type=3.

From the log message it prints, we can deduce that CALL_CMD=2, CALL_INCOMING=2, and NOTI=3. This lines up well with the definitions in libsamsung-ril, and it turns out that by observing the modem in operation you can learn a few more definitions not included in libsamsung-ril.

Next steps

Perhaps a good next step would be to translate this knowledge into support for the Galaxy S7 in libsamsung-ipc; for now, I don’t need that for myself, but it’s probably the Right Thing To Do. I’ll see if they’re interested in taking such a contribution.

Part of a series: #squeak-phone

Here’s a quick demo video of the status of my quixotic project to get Squeak running as a kind of standalone userland on a modern cellphone:

(The sound is bad! I recorded it on my other cellphone…)

Currently, I develop by coding in Squeak on my Linux desktop, using my graphics tablet as a kind of proxy for a touchscreen. I use the FFI and AIO/OSProcess support in Squeak to read events from /dev/input/event.... For event sources that present absolute axes, I create instances of HandMorph in the World and animate them according to the incoming events.

Every now and then, to test on the real hardware, I use rsync to copy the changes and image files up to the cellphone, and then log in to the phone over ssh to restart the Cog VM.

At the moment, the image has enough smarts to figure out how to read the touchscreen and offer basic touchscreen click support. This lets me do simple things like open, move and close windows, and lets me save and/or quit the image.

Next steps are to make it harder to misclick – perhaps by increasing the size of some of the touch targets – and to think about coding up a simple onscreen keyboard.

Alternatively, on a parallel path, I’ve been reverse-engineering (really nothing more sophisticated than strace of cbd and rild) the Samsung protocols for booting and operating the cellular modem. The code is short and simple. Perhaps instead of an onscreen keyboard I’ll code up a quick dialer Morph and get Squeak making phone calls.

Part of a series: #squeak-phone

I’ve made some minor image changes to adjust cached glyphs in TrueType fonts in Squeak when the DPI changes. Here are the results:

On the left, a stock, fresh-from-squeak.org unconfigured image, that wrongly believes the screen to be 96 DPI.

On the right, my dev image as I left it on my desktop PC, simply scped up to the phone and run, set to the correct 535 DPI resolution for the phone. Much better!

Part of a series: #squeak-phone

Back in 2007, when Openmoko was first a thing, I wrote an Erlang-based userland that got to the point of being able to take and make calls and receive and send SMS. The project stalled: the Openmoko GTA01 was too slow, its power-management too primitive, and Erlang’s GUI facilities too rudimentary to make further work worthwhile.

Modern cellphone hardware is much more capable. Is it time to have another run at the idea of a mobile personal computer?

PostmarketOS is awesome

Last week, I installed PostmarketOS on my previous cellphone, a Samsung Galaxy S7 (using PostmarketOS’s samsung-herolte configuration).

PostmarketOS turns out to be a beautifully engineered system that’s easy to understand and modify. The basics of kernel and Alpine Linux userland installed cleanly and easily on the phone, and it’s running well as a development platform. I’m looking forward to getting into PostmarketOS more deeply.

Running htop on the phone shows what an amazing little machine it is! So much power. Loads of cores, lots of RAM. Plenty of space to explore alternative visions of mobile personal computing.

However, the built-in demos, such as the Weston demo (shown above at right), currently leave quite a bit to be desired. Perhaps some of the other user interface options included with PostmarketOS could get me closer to a day-to-day usable cellphone - but I’m interested in running my own software! Let’s get hacking.

Running my own programs

PostmarketOS is a plain, clean Alpine Linux distribution. You can SSH into it initially via USB networking. From there, you can configure wifi using nmcli, set up SSH keys, and then access it directly using SSH over wifi.

Building software is just as simple:

apk add alpine-sdk

To experiment with drawing to the framebuffer and reading touchscreen input via /dev/input, I compiled and ran an old quick and dirty framebuffer hack I wrote years ago. The results (shown at left) were encouraging: the program effortlessly animates tens of thousands of points at 30 frames per second, responding to touch inputs. Display is via brute-force pixel output to the mmap‘d frame buffer. It doesn’t even use a full core.

PostmarketOS turns a phone into a fully capable Linux machine, with total control over the attached hardware, and with everything accessible to the developer in the usual places using the usual tools.

But Unix tools are inappropriate for a mobile personal computing platform. We’ll need something else.

A Smalltalk phone

Smalltalk could make an ideal basis for a mobile personal computing platform.

I’ve enjoyed using, developing with, and contributing to the Squeak Smalltalk implementation since the mid ’00s.

So I compiled the Cog Smalltalk VM on the phone itself, making use of the 64-bit ARM support code that landed extremely recently.

And lo and behold, it runs! Shown to the right is a bleeding-edge, fully up-to-date Squeak 6.0-alpha image running on the phone itself. (Click here or on the image to embiggen.)

From here, I can experiment with new ideas using the full power of a modern Smalltalk environment.

What next?

My previous Openmoko experiments foundered, in part, on the GUI aspect of the system; GTK+ via Erlang was fine for quick prototyping but wasn’t really up to the task for a day-to-day usable machine.

I recall getting Squeak running on my GTA01, in order to see if it could provide a viable UI. However, I remember being stymied by the mismatch between the expectations of the Smalltalk environment and the realities of the phone.

Squeak wants a mouse and keyboard. It assumes a monitor-sized display, in everything from widget and font sizes to window management. To work well on a phone, it needs a touchscreen-based, high-DPI UI in addition to its existing toolset.

Smalltalk, in both its language aspect and its system design aspect, also suffers from some weaknesses in areas where Erlang shines.

However, in the years since the GTA01:

• the hardware is much better,

• the Squeak VM and image are better,

• I’ve learned a heck of a lot about some good ways to design interactive systems, and

• I’ve recently built some tools that help bring Erlang- and Syndicate-style architectural patterns for concurrency to Smalltalk.

So I think using Erlang/Syndicate-style Actors to structure a Smalltalk-based phone userland, perhaps with cgroups-based sub-virtual-machines and images, could work well.

My initial experiments have concentrated on

• fixing the tiny fonts (the DPI-change support code in the image needs work, and the support in the VM seems to be absent (?)),

• reading from the touchscreen (probably like this),

• thinking about how to structure Actor supervision hierarchies and Dataspaces for a mobile phone (probably borrowing some design elements from my earlier Openmoko Erlang-based userland), and

• thinking about how to layer a touchscreen (panel-based?) GUI atop Squeak’s Morphic UI.

I’ll write more on this blog under the tag #squeak-phone as things develop.