eighty-twenty

Small experiments in the use of libliftoff to try out the modern Linux graphics stack drove home quite how slow DRM “dumb buffers” can be, but also that it’s reading that’s slow, not writing.

Reading from a “dumb buffer” on my AMD GPU is orders of magnitude slower than reading from RAM. It can take seconds to read out a full 4k frame. It’s roughly a thousand times slower than reading RAM.^{1 2}

Writing, by contrast, is quick.

While it is folklore that “dumb buffers are slow”, I found it challenging to find any authoritative source on the matter. However, I did find something. In /usr/include/drm/drm.h, we see the following comment, which sort of hints at the wider situation:

/**
 * DRM_CAP_DUMB_PREFER_SHADOW
 *
 * If set to 1, the driver prefers userspace to render to a shadow buffer
 * instead of directly rendering to a dumb buffer. For best speed, userspace
 * should do streaming ordered memory copies into the dumb buffer and never
 * read from it.
 *
 * Note that this preference only applies to dumb buffers, it's irrelevant for
 * other types of buffers.
 */
#define DRM_CAP_DUMB_PREFER_SHADOW	0x4

Indeed, “for best speed […] never read from it.”

Update: Subsequent experimentation using gbm to allocate buffer objects shows that it doesn’t help if you need to read or write pixel data to them (as opposed to, presumably, using the GPU to render into them). Setting the GBM_BO_USE_WRITE flag when allocating a buffer object, to allow subsequent writing of pixel data, causes the dri backend of gbm to simply allocate a “dumb buffer”!

“Erlang supports change of code in a running system.”

However, the details are a bit fiddly. Here’s a cheat-sheet I used recently for a simple TCP service written using Erlang.

My program was a single module, running outside of any OTP application context. The instructions here need minor emendation to either explicitly list modules to purge and reload or to discover all modules within a single application; see the places in server-reload below mentioning the atom my_server.

I did not use the -on_load() directive, because I wanted to be able to use multiple nodes rather than controlling reloads from a single node’s shell repl, and I couldn’t figure out how to make the two play nicely together.

The Erlang

I exported a code_change/0 from my module, to be called after loading a new version of the module into a node. It sends a message code_change to each “global” actor in my program (in this case, there was only one).

-export([code_change/0]).

code_change() ->
    io:format("+ code_change~n"),
    %% name registered previously with `global:register_name/2`:
    global:send(name_of_my_global_actor, code_change),
    ok.

That actor distributes the notification on to any inferior actors it is managing, and then does an “MFA” self-call to upgrade its own codebase.

index(Connected) ->
    receive
        code_change ->
            [P ! code_change || {_Peer, P} <- Connected],
            ?MODULE:index(Connected);
        ...
    end.

Similarly, all other notified actors perform “MFA” self-calls.

connection(Sock, Username, IndexPid) ->
    receive
        code_change ->
            ?MODULE:connection(Sock, Username, IndexPid);
        ...
    end.

Actors need to take care to manage upgrades of their state at the same time as they do the “MFA” self-calls.

Starting the program

I wanted it to be run by daemontools, so created the following shell script called run, which daemontools will pick up to start a service:

#!/bin/sh
set -e
erlc -o ebin my_server.erl
exec erl \
     -noshell \
     -pa ebin \
     -sname mainnode \
     -setcookie f98b3a1e-80ec-11ef-b752-0b638e4de31c \
     -s my_server

Pick a fresh random cookie for the -setcookie argument. I used uuid(1).

Then, I created this script, server-reload:

#!/bin/sh
set -e
erlc -o ebin my_server.erl
exec erl \
     -noshell \
     -pa ebin \
     -setcookie f98b3a1e-80ec-11ef-b752-0b638e4de31c \
     -sname undefined \
     -eval "
           ServerNode = mainnode@$(hostname -s),
           io:format(\"ServerNode: ~p~n\", [ServerNode]),
           true = net_kernel:connect_node(ServerNode),
           spawn(ServerNode, fun () ->
               code:purge(my_server),
               code:load_file(my_server),
               ok = my_server:code_change()
           end),
           init:stop()"

Running server-reload causes the source code to be compiled and hot-loaded into the running server.

Grace notes

Then, I used a git post-receive hook to automatically recompile and reload the code on push to live:

#!/bin/sh
set -e
unset GIT_DIR
cd $HOME/location-of-checkout-of-server-repository
git pull --ff-only
./server-reload

That’s it

That’s all. The end result worked well: I used it to run a hotfix to my TCP service with many tens of live, active connections, and not one of them noticed a thing.

Back in June, I made a quick-and-dirty attempt to get the big-bang model of functional UI running in Processing 4.

Unfortunately Processing uses a dialect of Java predating introduction of Java Records (JEP395), so I, er, creatively broke out m4 as a preprocessor.

The resulting macros turn this:

_record(Rect extends Pict, {{float x, float y, float w, float h}}, {{
  public void render() {
    rectMode(CORNER);
    rect(this.x, this.y, this.w, this.h);
  }
}});

into this:

class Rect extends Pict {
  public final float x;
  public final float y;
  public final float w;
  public final float h;
  public Rect(float x, float y, float w, float h) {
    this.x = x;
    this.y = y;
    this.w = w;
    this.h = h;
  }

  public void render() {
    rectMode(CORNER);
    rect(this.x, this.y, this.w, this.h);
  }
};

The macros

Not yet properly factored out into a utility library or anything, just pasted straight at the top of the file. Shield your eyes!

/* -*- mode: java; c-basic-offset: 2 -*- */
changecom(`//')dnl
changequote(`{{',`}}')dnl
dnl);
define({{_record}}, {{class $1 {_record_fields($2,)
  public _record_classname($1)($2) {_record_inits($2,)
  }
$3dnl;
}{{}}}})dnl;
define({{_record_fields}}, {{ifelse({{$#}}, {{1}},, {{
  public final $1;$0(shift($@))}})}})dnl;
define({{_record_inits}}, {{ifelse({{$#}}, {{1}},, {{
    this._record_fieldname({{$1}}) = _record_fieldname({{$1}});$0(shift($@))}})}})dnl;
define({{_record_classname}}, {{regexp({{$1}}, {{^\(\w+\).*$}}, {{\1}})}})dnl;
define({{_record_fieldname}}, {{regexp({{$1}}, {{^.+\s\(\w+\)$}}, {{\1}})}})dnl;
dnl;//---------------------------------------------------------------------------

I had a small insight yesterday while building a component for a small web app: the user interface for editing an incomplete value of sum type A+B needs to remember a product of input 2×A×B from the user:

A + B ⟿ 2 × A × B

This allows the user to ergonomically change their mind about whether they’re building an A or a B without losing partially constructed values.

More precisely, the UI for a value of type A+B needs in general to be able to remember and manipulate 2×(A+1)×(B+1):

A + B ⟿ 2 × (A+1) × (B+1)

The extra 1s allow for nulls, for temporarily missing but required values. You could similarly generalise to allow for temporarily invalid or unparseable values.

Example

Consider UI for creating a new project in an IDE, with two available options: create a new local project, by simply creating a new directory, or clone an existing git repository.

data NewProject =
   Local { projectName :: String }
 | Clone { gitUrl :: String,
           credential :: String,
           projectName :: String }

Abstractly, this is roughly Str + Str×Str×Str.

The user interface for this will look something like

Here we see that while a value of type NewProject is being built, we need to remember four strings (abstractly, Str×Str×Str×Str), plus a boolean indicating whether we ultimately want a “local” or “clone” project type (abstractly, 2).

All told, that’s

Str + Str×Str×Str ⟿ 2 × Str×Str×Str×Str

which exactly fits the pattern of

A + B ⟿ 2 × A × B

Generalization to bigger sums

The translation can be applied recursively, but it (harmlessly) remembers slightly too much transient UI state,

A+(B+C) ⟿ 2 × A × (2 × B × C)

so perhaps it’s better to think about it applying directly to n-ary sums:

A+B+C ⟿ 3 × A × B × C
A+B+C+D ⟿ 4 × A × B × C × D

and so on.

I noticed a bug in Guile 3.0.9’s aarch64 atomics handling, and found a couple of apparent solutions (1, 2), but one of them is weird enough for me to write this post.

(ETA: Nonstory. The problem was that the mov instruction isn’t idempotent! Hat tip to Andy Wingo for figuring out what the issue was. I’ve updated the rest of the article, and I’ll leave it here for posterity.)

Long story short, the problem was with the equivalent of C’s atomic_exchange. Here’s the code that Guile’s JIT was generating:

1:
    mov     x16, x0
    ldaxr   x0, [x1]
    stlxr   w17, x16, [x1]
    cbnz    w17, 1b

This code appears to occasionally lose writes (!). ETA: This code definitely loses writes when interference means it has to go around the loop.

The first patch I wrote boringly replaced the lot with a single

    swpal   x0, x0, [x1]

which is fine, if you have an ARM v8.1 device to hand, but not if you don’t have a machine with Large System Extensions. So I tried, on a hunch, the second patch, which just changed the target of the cbnz, producing code like this:

    mov     x16, x0
1:
    ldaxr   x0, [x1]
    stlxr   w17, x16, [x1]
    cbnz    w17, 1b

… and the issue disappeared! What! This shouldn’t have made a difference! Should it? ETA: And fair enough, too! If the branch targets the mov instruction, the value of x0 that ldaxr set is used, meaning that the whole operation simply becomes a no-op assignment.

Are aarch64 atomics really this sensitive? Is there only One True Instruction Sequence that should be used to implement atomic_exchange? Why does making this seemingly-insignificant change produce such a noticeable effect? ETA: Nothing to see here :-)

As is well-known, JavaScript’s Promise is not a monad. It will happily treat Promise<Promise<T>> as if it was Promise<T>:

> [123, await Promise.resolve(123), await Promise.resolve(Promise.resolve(123))]
[ 123, 123, 123 ]

This can bite you in unexpected ways. Imagine you have a CSP-like Channel<T> class for sending Ts back and forth. Channel<T> might have a method like this:

async pop(): Promise<T | undefined> { ... }

There’s an obvious problem here: what if undefined ∈ T? So you make sure to note, in the comment attached to Channel<T>, that T is not allowed to include undefined.

But the less obvious problem is that T is not allowed to contain Promise<undefined> either, even though in other contexts a promise of undefined cannot be confused with undefined:

> typeof undefined
'undefined'
> typeof Promise.resolve(undefined)
'object'

To see why this is a problem, instantiate T with Promise<undefined>, and look at the type of pop():

Promise<Promise<undefined> | undefined>

Because JavaScript collapses promises-of-promises to just promises, this is equivalent to just

Promise<undefined>

and you’ve lost the ability to tell whether pop() yielded a T or an undefined.

TypeScript does not warn you about this, incidentally. (Ask me how I know.)

Workaround

Instead of accepting this loss of structure and adding another caveat to Channel<T> to work around JavaScript’s broken design—“T must not include either undefined or Promise<undefined> or Promise<Promise<undefined>> etc.”—I decided to change the signature of pop():

async pop(): Promise<Maybe<T>> { ... }

type Maybe<T> = Just<T> | undefined;
type Just<T> = { item: T };

Now both Channel<undefined> and Channel<Promise<undefined>> are sensible and work as expected. No more exceptions regarding what Ts a Channel may carry.

When T is Promise<undefined>, in particular, we see that the type of pop() is

Promise<{ item: Promise<undefined> } | undefined>

Because the Promises aren’t immediately nested, JavaScript won’t erase our structure.

(Ironically, we’ve introduced a monad (Maybe<T>) to fix the bad behaviour of something that should have been a monad…)

Python 3.13 will, for inscrutable reasons, remove the crypt module from the standard library. The excuses given in PEP 0594 boil down to “here are some good reasons why new code shouldn’t use this module.” What about existing code? Ah well.

So anyway, for those of us who need some way of generating $6$ SHAcrypt SHA-512 shadow-password-database entries from Python, stick the following module into your codebase (you can also download it, shacrypt512.py) and replace code like

crypt.crypt(password, salt=crypt.METHOD_SHA512)
crypt.crypt(password, '$6$salt$...')
crypt.crypt(password, '$6$salt$...') == '$6$salt$...'

with

shacrypt512.shacrypt(password.encode('utf-8'))
shacrypt512.shacrypt(password.encode('utf-8'), b'salt')
shacrypt512.password_ok(password.encode('utf-8'), '$6$salt$...')

respectively.

Without further ado, here’s shacrypt512.py:

# SHAcrypt using SHA-512, after https://akkadia.org/drepper/SHA-crypt.txt.
#
# Copyright © 2024 Tony Garnock-Jones.
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

import hashlib
import secrets

alphabet = \
    [ord(c) for c in './0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz']
permutation = [
    [0, 21, 42], [22, 43, 1], [44, 2, 23], [3, 24, 45],
    [25, 46, 4], [47, 5, 26], [6, 27, 48], [28, 49, 7],
    [50, 8, 29], [9, 30, 51], [31, 52, 10], [53, 11, 32],
    [12, 33, 54], [34, 55, 13], [56, 14, 35], [15, 36, 57],
    [37, 58, 16], [59, 17, 38], [18, 39, 60], [40, 61, 19],
    [62, 20, 41], [-1, -1, 63],
]
def encode(bs64):
    result = bytearray(4 * len(permutation))
    i = 0
    for group in permutation:
        g = lambda j: bs64[j] if j != -1 else 0
        bits = g(group[0]) << 16 | g(group[1]) << 8 | g(group[2])
        result[i] = alphabet[bits & 63]
        result[i+1] = alphabet[(bits >> 6) & 63]
        result[i+2] = alphabet[(bits >> 12) & 63]
        result[i+3] = alphabet[(bits >> 18) & 63]
        i = i + 4
    return bytes(result).decode('ascii')[:-2]

def repeats_of(n, bs): return bs * int(n / len(bs)) + bs[:n % len(bs)]
def digest(bs): return hashlib.sha512(bs).digest()

def shacrypt(password, salt = None, rounds = 5000):
    if salt is None: salt = encode(secrets.token_bytes(64))[:16].encode('ascii')
    salt = salt[:16]

    B = digest(password + salt + password)
    Ainput = password + salt + repeats_of(len(password), B)
    v = len(password)
    while v > 0:
        Ainput = Ainput + (B if v & 1 else password)
        v = v >> 1
    A = digest(Ainput)

    DP = digest(password * len(password))
    P = repeats_of(len(password), DP)
    DS = digest(salt * (16+A[0]))
    S = repeats_of(len(salt), DS)

    C = A
    for round in range(rounds):
        Cinput = b''
        Cinput = Cinput + (P if round & 1 else C)
        if round % 3: Cinput = Cinput + S
        if round % 7: Cinput = Cinput + P
        Cinput = Cinput + (C if round & 1 else P)
        C = digest(Cinput)

    if rounds == 5000:
        return '$6$' + salt.decode('ascii') + '$' + encode(C)
    else:
        return '$6$rounds=' + str(rounds) + '$' + salt.decode('ascii') + '$' + encode(C)

#---------------------------------------------------------------------------

def extract_salt_and_rounds(i): # i must be '$6$...'
    pieces = i.split('$')
    if pieces[1] != '6': raise TypeError('shacrypt512 only supports $6$ hashes')
    if pieces[2].startswith('rounds='):
        rounds = int(pieces[2][7:])
        if rounds < 1000: rounds = 1000
        if rounds > 999999999: rounds = 999999999
        return (pieces[3].encode('ascii'), rounds)
    else:
        return (pieces[2].encode('ascii'), 5000)

def password_ok(input_password, existing_crypted_password):
    (salt, rounds) = extract_salt_and_rounds(existing_crypted_password)
    return existing_crypted_password == shacrypt(input_password, salt, rounds)

if __name__ == '__main__':
    _test_password = 'Hello world!'.encode('ascii')
    _test_salt = 'saltstring'.encode('ascii')
    _test_rounds = 5000
    _test_crypted_password = '$6$saltstring$svn8UoSVapNtMuq1ukKS4tPQd8iKwSMHWjl/O817G3uBnIFNjnQJuesI68u4OTLiBFdcbYEdFCoEOfaS35inz1'
    assert shacrypt(_test_password, _test_salt, _test_rounds) == _test_crypted_password
    assert password_ok(_test_password, _test_crypted_password)

    _test_password = 'Hello world!'.encode('ascii')
    _test_salt = 'saltstringsaltstring'.encode('ascii')
    _test_rounds = 10000
    _test_crypted_password = '$6$rounds=10000$saltstringsaltst$OW1/O6BYHV6BcXZu8QVeXbDWra3Oeqh0sbHbbMCVNSnCM/UrjmM0Dp8vOuZeHBy/YTBmSK6H9qs/y3RnOaw5v.'
    assert shacrypt(_test_password, _test_salt, _test_rounds) == _test_crypted_password
    assert password_ok(_test_password, _test_crypted_password)

    import sys
    salt = None if len(sys.argv) < 2 else sys.argv[1].encode('ascii')
    print(shacrypt(sys.stdin.readline().strip().encode('utf-8'), salt))

direnv works fine on Windows 11, but if an .envrc tries to set the PATH, the result will be a path in Windows format, not Unix format.¹

Instead of adding eval $(direnv hook bash) to your .bashrc, try the following snippet:

export _unmangle_direnv_names='PATH'
_unmangle_direnv_paths() {
    for k in $_unmangle_direnv_names; do
        eval "$k=\"\$(/usr/bin/cygpath -p \"\$$k\")\""
    done
}
eval "$(direnv hook bash | sed -e 's@export bash)@export bash)\
_unmangle_direnv_paths@')"

This modifies the output of direnv hook bash slightly, adding code to fix path-like variables after direnv sets the environment up.²

The variable names to unmangle are drawn from a new variable, _unmangle_direnv_names, initially set to PATH, which should contain a space-separated list of variable names.

If, in a particular .envrc, you need path-unmangling for an additional variable, you can add that variable’s name to _unmangle_direnv_names. For example,

_unmangle_direnv_names="$_unmangle_direnv_names XPATH"
export PATH="$PATH:some_addition"
export XPATH="$PATH:some_addition"

will unmangle both PATH and XPATH.

Trusting Signal and/or WhatsApp and/or Android (google) and/or iOS (apple) with your precious photos, videos, and chats is a huge mistake.

All the photos, videos and chat history on my mother’s phone are now completely inaccessible to us, despite having recent backups and all the necessary keys and passphrases.

Signal

Can I transfer Signal backups directly from android to iphone? No.

Can I make a backup file and transfer that wirelessly without uploading it to the cloud? No.

Can I transfer it using a USB cable? Also no.

If I transfer it using the cloud, can I then restore from it on the iphone? … No.

It is not possible to transfer Signal chat history and media from an android phone to an iphone.

WhatsApp

What about Whatsapp?

Can I transfer Whatsapp backups directly from android to iphone? No.

Can I make a backup file and transfer that? No.

Can I back up to google drive and use that? No.

Even if I could get the backup file, would I be able to import it on the iphone? Also no.

It is not possible to transfer WhatsApp chat history and media from an android phone to an iphone.

The entire ecosystem is sick

Signal is to blame: they do not make it possible to import backup data on iphone. They do not offer tools for working with backup data.

WhatsApp is to blame: they do not make it possible to retrieve or work with android backup data. They do not make it possible to migrate data from android to ios without fully wiping and resetting the iphone (and it didn’t work even when we did fully wipe it).

Apple is to blame: there is no way to transfer files from android without going through the cloud. Even using a usb cable doesn’t work.

Google is to blame: there is no way to transfer files to iphone without going through the cloud. There is no way to access a whatsapp backup blob in google drive containing my own data.

We all are to blame: we have accepted and continue to make excuses for an industry that acts in such a user-hostile way.

I’ve just updated the Preserves spec to version 0.990.0. I feel like a 1.0-rc is approaching!

The main change since spec version 0.7.1 has been to simplify the binary syntax for Preserves Values:

• Both “short” and “medium” SignedInteger representations (starting with tags 0x9x/0xAx) were removed. They weren’t pulling their weight. Every SignedInteger now has tag 0xB0.

• Float and Double are now encoded with tag 0x87 and a length, rather than with fixed tags 0x82 and 0x83, opening the door to other IEEE754 formats in future.

Here’s the 0.990.0 syntax in “reference card” format, where we write «V» for the binary encoding of some value V:

                      «#f» = [0x80]
                      «#t» = [0x81]

                    «@W V» = [0x85] ++ «W» ++ «V»
                     «#!V» = [0x86] ++ «V»

  «V» if V ∈ Float         = [0x87, 0x04] ++ binary32(V)
  «V» if V ∈ Double        = [0x87, 0x08] ++ binary64(V)

  «V» if V ∈ SignedInteger = [0xB0] ++ varint(|intbytes(V)|) ++ intbytes(V)
  «V» if V ∈ String        = [0xB1] ++ varint(|utf8(V)|) ++ utf8(V)
  «V» if V ∈ ByteString    = [0xB2] ++ varint(|V|) ++ V
  «V» if V ∈ Symbol        = [0xB3] ++ varint(|utf8(V)|) ++ utf8(V)

           «<L F_1...F_m>» = [0xB4] ++ «L» ++ «F_1» ++...++ «F_m» ++ [0x84]
             «[X_1...X_m]» = [0xB5] ++ «X_1» ++...++ «X_m» ++ [0x84]
            «#{E_1...E_m}» = [0xB6] ++ «E_1» ++...++ «E_m» ++ [0x84]
     «{K_1:V_1...K_m:V_m}» = [0xB7] ++ «K_1» ++ «V_1» ++...++ «K_m» ++ «V_m» ++ [0x84]

                 varint(v) = [v] if v < 128
                             [(v & 0x7F) + 128] ++ varint(v >> 7) if v ≥ 128