eighty-twenty

Back in 2002/2003, Ian Piumarta wrote an “essentially complete” VT102 terminal emulator for Squeak Smalltalk.¹

Here are his unmodified original changesets, encoded using the MacRoman character set; these do not load directly into current Squeak images as-is:

• PseudoTTY-3.2-4.st (MacRoman, not loadable)
• telnet.301.cs (MacRoman, not loadable)

The problem is not just that they’re encoded using MacRoman—Squeak still has support for that—but that there are punning uses of strings to represent bytewise mappings, rather than characterwise mappings.

The first step to getting them loadable is to convert them to UTF-8. I did this using emacs² because both Squeak itself and iconv(1) choked on some of the tricky encodings going on in the files. A subsequent step will be to repair the tricky parts, re-writing them to hopefully use in-image Unicode support.

• PseudoTTY-3.2-4.st (UTF-8, not loadable)
• telnet.301.cs (UTF-8, not loadable)

The Squeak Smalltalk language has also changed a little since 2003: assignment is no longer ← (written using an underscore, _), but is instead the digraph :=; it is no longer permitted to store into method or block arguments; and so on. Fixing these issues yields the following:

• PseudoTTY-3.2-4.st (UTF-8, loadable)
• telnet.301.cs (UTF-8, loadable)

Now, filing in telnet.301.cs yields an error in TeletypeMorph class » initializeCharacterClasses. This is the main (?) place involving 8-bit character set assumptions that will have to be revisited. Changing that method temporarily to delete its actual body, replacing it with the commented-out table taken directly from xterm, allows the fileIn to complete.

Filing in PseudoTTY-3.2-4.st appears to succeed without problems.

Next steps will be seeing if all this code actually runs!

Smalltalk programs strictly alternate between data and behaviour. Messages (the only kind of data in Smalltalk) are implicit, constructed fresh at each point in the program method call syntax is used, and are shallow, meaning that the slots in each message object always contain references to objects, never other messages.¹

This is in contrast to most other languages with true data, where (for example) lists may contain other data, and are not constrained to containing only object references / function values.

So, what would a Smalltalk be like without this strict alternation, where messages could appear as values? Suddenly the universe of Smalltalk values grows larger: previously, everything was an object, but now some things are data!

Usually, method lookup results in a single body to execute. This is analogous to the way pattern matching usually tries patterns in order, selecting just one continuation to execute.

What if, instead, we allowed all matching patterns from a bunch of alternatives to execute? (Or, in object-oriented terms: executed all methods potentially matching a given method call.)

Pattern alternatives do not become a kind of superposition, because there’s no notion of mutual exclusion; instead, they become a way of creating multiple concurrent branches of execution, somehow. (Not to say there’s any particular kind of interleaving or parallel execution of these branches that makes any sense! One could well limit consideration to sequential execution of each matching method, to start with.)

Can we recover true alternatives from this kind of every-match construct (it needs a name!)? One approach is to follow the idea from Alex’s, Mahdi’s and my PEG paper,¹ treating alternation (/) as a kind of parallel match construct and using negative lookahead to cause a later branch to fail if some earlier branch succeeds.

A fantasy abstract is a short piece of writing in the style of the abstract of an academic paper presenting the outline of a piece of research that has not (yet) been done. It acts as inspiration, as a means of communicating an idea, and as a place to nucleate further thinking on the topic, perhaps eventually kicking off the desired research.

It’s something I started doing during my PhD studies to help record and structure the relationships among ideas. I record a little bit of metadata about each abstract: not much more than the names of one or more broad research “themes” or threads that the idea might fit into.

For example, here’s one from January 2011, shortly after I started experimenting with the form. Fourteen years later it remains fantastic and unexplored (by me at least!):

Contracts for Protocols

Created: 2011-01-10

Thread

Network Languages

Abstract

Existing messaging middleware systems provide very low-level facilities to application developers, ranging from simple point-to-point datagram transfer up through simple stereotypical interaction patterns such as (optionally transacted) request-reply or publish-subscribe. These low-level facilities are then composed by the application developer into higher-level interactions, but without the benefit of any formal way of describing the higher-level interactions. This paper introduces contracts for messaging protocols implemented using messaging middleware, describes a prototype implementation, and discusses lessons learned.

Some things to note:

• Citations are useful if you have them, but the main point is to capture the idea, not do an exhaustive background literature survey. In the example above there’s the ludicrous omission of any mention of session types, for example; what I had in mind was something akin to what, these days, are called “dynamic monitors” in the session types literature. Perhaps if I’d expanded this abstract into an actual paper at the time it’d have been a timely contribution :-) It’s a bit stale now…

• It can be an absolute fantasy. Feel free to refer to nonexistent (but plausible?) research results. If you ever pick up the idea or gift it to someone else, it’ll be made rigorous and realistic then. Use the fantasy abstract to get the feeling of your idea.

I always forget how to do this, so I’m writing it here in part as a reminder for myself next time I need to do this.

These instructions are for using daemontools and daemontools-run packages on Debian.

1. Create a directory /etc/service/tonyg-services

2. Create a file /etc/service/tonyg-services/run containing

   #!/bin/sh
   exec setuidgid tonyg svscan /home/tonyg/services
   

   Replace setuidgid with sudo -u if you want to preserve supplemental groups.

3. Run chmod a+x /etc/service/tonyg-services/run
4. Create /home/tonyg/services
5. Create services ad libitum therein.

Small experiments in the use of libliftoff to try out the modern Linux graphics stack drove home quite how slow DRM “dumb buffers” can be, but also that it’s reading that’s slow, not writing.

Reading from a “dumb buffer” on my AMD GPU is orders of magnitude slower than reading from RAM. It can take seconds to read out a full 4k frame. It’s roughly a thousand times slower than reading RAM.^{1 2}

Writing, by contrast, is quick.

While it is folklore that “dumb buffers are slow”, I found it challenging to find any authoritative source on the matter. However, I did find something. In /usr/include/drm/drm.h, we see the following comment, which sort of hints at the wider situation:

/**
 * DRM_CAP_DUMB_PREFER_SHADOW
 *
 * If set to 1, the driver prefers userspace to render to a shadow buffer
 * instead of directly rendering to a dumb buffer. For best speed, userspace
 * should do streaming ordered memory copies into the dumb buffer and never
 * read from it.
 *
 * Note that this preference only applies to dumb buffers, it's irrelevant for
 * other types of buffers.
 */
#define DRM_CAP_DUMB_PREFER_SHADOW	0x4

Indeed, “for best speed […] never read from it.”

Update: Subsequent experimentation using gbm to allocate buffer objects shows that it doesn’t help if you need to read or write pixel data to them (as opposed to, presumably, using the GPU to render into them). Setting the GBM_BO_USE_WRITE flag when allocating a buffer object, to allow subsequent writing of pixel data, causes the dri backend of gbm to simply allocate a “dumb buffer”!

“Erlang supports change of code in a running system.”

However, the details are a bit fiddly. Here’s a cheat-sheet I used recently for a simple TCP service written using Erlang.

My program was a single module, running outside of any OTP application context. The instructions here need minor emendation to either explicitly list modules to purge and reload or to discover all modules within a single application; see the places in server-reload below mentioning the atom my_server.

I did not use the -on_load() directive, because I wanted to be able to use multiple nodes rather than controlling reloads from a single node’s shell repl, and I couldn’t figure out how to make the two play nicely together.

The Erlang

I exported a code_change/0 from my module, to be called after loading a new version of the module into a node. It sends a message code_change to each “global” actor in my program (in this case, there was only one).

-export([code_change/0]).

code_change() ->
    io:format("+ code_change~n"),
    %% name registered previously with `global:register_name/2`:
    global:send(name_of_my_global_actor, code_change),
    ok.

That actor distributes the notification on to any inferior actors it is managing, and then does an “MFA” self-call to upgrade its own codebase.

index(Connected) ->
    receive
        code_change ->
            [P ! code_change || {_Peer, P} <- Connected],
            ?MODULE:index(Connected);
        ...
    end.

Similarly, all other notified actors perform “MFA” self-calls.

connection(Sock, Username, IndexPid) ->
    receive
        code_change ->
            ?MODULE:connection(Sock, Username, IndexPid);
        ...
    end.

Actors need to take care to manage upgrades of their state at the same time as they do the “MFA” self-calls.

Starting the program

I wanted it to be run by daemontools, so created the following shell script called run, which daemontools will pick up to start a service:

#!/bin/sh
set -e
erlc -o ebin my_server.erl
exec erl \
     -noshell \
     -pa ebin \
     -sname mainnode \
     -setcookie f98b3a1e-80ec-11ef-b752-0b638e4de31c \
     -s my_server

Pick a fresh random cookie for the -setcookie argument. I used uuid(1).

Then, I created this script, server-reload:

#!/bin/sh
set -e
erlc -o ebin my_server.erl
exec erl \
     -noshell \
     -pa ebin \
     -setcookie f98b3a1e-80ec-11ef-b752-0b638e4de31c \
     -sname undefined \
     -eval "
           ServerNode = mainnode@$(hostname -s),
           io:format(\"ServerNode: ~p~n\", [ServerNode]),
           true = net_kernel:connect_node(ServerNode),
           spawn(ServerNode, fun () ->
               code:purge(my_server),
               code:load_file(my_server),
               ok = my_server:code_change()
           end),
           init:stop()"

Running server-reload causes the source code to be compiled and hot-loaded into the running server.

Grace notes

Then, I used a git post-receive hook to automatically recompile and reload the code on push to live:

#!/bin/sh
set -e
unset GIT_DIR
cd $HOME/location-of-checkout-of-server-repository
git pull --ff-only
./server-reload

That’s it

That’s all. The end result worked well: I used it to run a hotfix to my TCP service with many tens of live, active connections, and not one of them noticed a thing.

Back in June, I made a quick-and-dirty attempt to get the big-bang model of functional UI running in Processing 4.

Unfortunately Processing uses a dialect of Java predating introduction of Java Records (JEP395), so I, er, creatively broke out m4 as a preprocessor.

The resulting macros turn this:

_record(Rect extends Pict, {{float x, float y, float w, float h}}, {{
  public void render() {
    rectMode(CORNER);
    rect(this.x, this.y, this.w, this.h);
  }
}});

into this:

class Rect extends Pict {
  public final float x;
  public final float y;
  public final float w;
  public final float h;
  public Rect(float x, float y, float w, float h) {
    this.x = x;
    this.y = y;
    this.w = w;
    this.h = h;
  }

  public void render() {
    rectMode(CORNER);
    rect(this.x, this.y, this.w, this.h);
  }
};

The macros

Not yet properly factored out into a utility library or anything, just pasted straight at the top of the file. Shield your eyes!

/* -*- mode: java; c-basic-offset: 2 -*- */
changecom(`//')dnl
changequote(`{{',`}}')dnl
dnl);
define({{_record}}, {{class $1 {_record_fields($2,)
  public _record_classname($1)($2) {_record_inits($2,)
  }
$3dnl;
}{{}}}})dnl;
define({{_record_fields}}, {{ifelse({{$#}}, {{1}},, {{
  public final $1;$0(shift($@))}})}})dnl;
define({{_record_inits}}, {{ifelse({{$#}}, {{1}},, {{
    this._record_fieldname({{$1}}) = _record_fieldname({{$1}});$0(shift($@))}})}})dnl;
define({{_record_classname}}, {{regexp({{$1}}, {{^\(\w+\).*$}}, {{\1}})}})dnl;
define({{_record_fieldname}}, {{regexp({{$1}}, {{^.+\s\(\w+\)$}}, {{\1}})}})dnl;
dnl;//---------------------------------------------------------------------------

I had a small insight yesterday while building a component for a small web app: the user interface for editing an incomplete value of sum type A+B needs to remember a product of input 2×A×B from the user:

A + B ⟿ 2 × A × B

This allows the user to ergonomically change their mind about whether they’re building an A or a B without losing partially constructed values.

More precisely, the UI for a value of type A+B needs in general to be able to remember and manipulate 2×(A+1)×(B+1):

A + B ⟿ 2 × (A+1) × (B+1)

The extra 1s allow for nulls, for temporarily missing but required values. You could similarly generalise to allow for temporarily invalid or unparseable values.

Example

Consider UI for creating a new project in an IDE, with two available options: create a new local project, by simply creating a new directory, or clone an existing git repository.

data NewProject =
   Local { projectName :: String }
 | Clone { gitUrl :: String,
           credential :: String,
           projectName :: String }

Abstractly, this is roughly Str + Str×Str×Str.

The user interface for this will look something like

Here we see that while a value of type NewProject is being built, we need to remember four strings (abstractly, Str×Str×Str×Str), plus a boolean indicating whether we ultimately want a “local” or “clone” project type (abstractly, 2).

All told, that’s

Str + Str×Str×Str ⟿ 2 × Str×Str×Str×Str

which exactly fits the pattern of

A + B ⟿ 2 × A × B

Generalization to bigger sums

The translation can be applied recursively, but it (harmlessly) remembers slightly too much transient UI state,

A+(B+C) ⟿ 2 × A × (2 × B × C)

so perhaps it’s better to think about it applying directly to n-ary sums:

A+B+C ⟿ 3 × A × B × C
A+B+C+D ⟿ 4 × A × B × C × D

and so on.

I noticed a bug in Guile 3.0.9’s aarch64 atomics handling, and found a couple of apparent solutions (1, 2), but one of them is weird enough for me to write this post.

(ETA: Nonstory. The problem was that the mov instruction isn’t idempotent! Hat tip to Andy Wingo for figuring out what the issue was. I’ve updated the rest of the article, and I’ll leave it here for posterity.)

Long story short, the problem was with the equivalent of C’s atomic_exchange. Here’s the code that Guile’s JIT was generating:

1:
    mov     x16, x0
    ldaxr   x0, [x1]
    stlxr   w17, x16, [x1]
    cbnz    w17, 1b

This code appears to occasionally lose writes (!). ETA: This code definitely loses writes when interference means it has to go around the loop.

The first patch I wrote boringly replaced the lot with a single

    swpal   x0, x0, [x1]

which is fine, if you have an ARM v8.1 device to hand, but not if you don’t have a machine with Large System Extensions. So I tried, on a hunch, the second patch, which just changed the target of the cbnz, producing code like this:

    mov     x16, x0
1:
    ldaxr   x0, [x1]
    stlxr   w17, x16, [x1]
    cbnz    w17, 1b

… and the issue disappeared! What! This shouldn’t have made a difference! Should it? ETA: And fair enough, too! If the branch targets the mov instruction, the value of x0 that ldaxr set is used, meaning that the whole operation simply becomes a no-op assignment.

Are aarch64 atomics really this sensitive? Is there only One True Instruction Sequence that should be used to implement atomic_exchange? Why does making this seemingly-insignificant change produce such a noticeable effect? ETA: Nothing to see here :-)