eighty-twenty news feed for tag "syntax"2024-01-24T14:37:46+00:00http://eighty-twenty.org/tag/syntax/index.atomtonygmikeb#lang something: An alternative syntax for Racket2019-07-21T16:03:42+00:00http://eighty-twenty.org/2019/07/21/indentation-syntax-for-rackettonyg<p>Recent discussions (e.g.
<a href="https://groups.google.com/forum/#!topic/racket-users/HiC7z3A5O-k">1</a>,
<a href="https://groups.google.com/forum/#!topic/racket-users/3aIPOGbGgmc">2</a>)
about potentially revising Racket syntax for Racket2 have reminded me
I never properly announced <a href="https://github.com/tonyg/racket-something"><code class="language-plaintext highlighter-rouge">#lang something</code></a>,
an experiment from back in 2016.</p>
<p>The main idea is S-expressions, but with usually-implicit parentheses
and support for prefix/infix/postfix operators. Indentation for
grouping is explicitly represented in the S-expression returned from
the reader.</p>
<ul>
<li>(+) keeps a semi-structured input format: reader yields ordinary syntax</li>
<li>(+) macros Just Work</li>
<li>(+) you can do “if … then … else …”: <a href="https://github.com/tonyg/racket-something/blob/4a00a9a6d37777f5aab4f28b03c53555b87e21d7/examples/if-then-else.rkt">an example</a></li>
<li>(+) you can do “… where …”: <a href="https://github.com/tonyg/racket-something/blob/4a00a9a6d37777f5aab4f28b03c53555b87e21d7/examples/where.rkt">an example</a></li>
<li>(-) uses indentation (though it doesn’t have to; see for example <a href="https://github.com/tonyg/racket-something/blob/4a00a9a6d37777f5aab4f28b03c53555b87e21d7/src/something/shell2.rkt">this module</a>)</li>
<li>(-) the function syntax isn’t <code class="language-plaintext highlighter-rouge">function(arg, ...)</code></li>
</ul>
<p>(More links at the bottom of this post.)</p>
<p>In addition to the reader, <code class="language-plaintext highlighter-rouge">#lang something</code> provides a small
selection of special forms that take advantage of the new syntax, and
<code class="language-plaintext highlighter-rouge">#lang something/shell</code> adds Unix-shell-like behaviour and a few
associated utilities.</p>
<p>This program:</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="o">#</span><span class="nv">lang</span> <span class="nv">something</span>
<span class="nv">for</span> <span class="p">{</span> <span class="nv">x:</span> <span class="mi">1</span> <span class="o">..</span> <span class="mi">10</span> <span class="p">}</span>
<span class="nv">def</span> <span class="nv">y:</span> <span class="nv">x</span> <span class="nv">+</span> <span class="mi">1</span>
<span class="nv">printf</span> <span class="s">"x ~a y ~a\n"</span> <span class="nv">x</span> <span class="nv">y</span></code></pre></figure>
<p>… reads as this S-expression:</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="k">module</span> <span class="nv">something-module</span> <span class="nv">something/base</span>
<span class="p">(</span><span class="nf">#%rewrite-body</span>
<span class="p">(</span><span class="nf">for</span> <span class="p">(</span><span class="nf">block</span> <span class="p">(</span><span class="nf">x</span> <span class="p">(</span><span class="nf">block</span> <span class="p">(</span><span class="nf">1</span> <span class="o">..</span> <span class="mi">10</span><span class="p">))))</span>
<span class="p">(</span><span class="nf">block</span> <span class="p">(</span><span class="nf">def</span> <span class="nv">y</span> <span class="p">(</span><span class="nf">block</span> <span class="p">(</span><span class="nf">x</span> <span class="nv">+</span> <span class="mi">1</span><span class="p">)))</span>
<span class="p">(</span><span class="nb">printf</span> <span class="s">"x ~a y ~a\n"</span> <span class="nv">x</span> <span class="nv">y</span><span class="p">)))))</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">#%rewrite-body</code> macro, together with its companion
<code class="language-plaintext highlighter-rouge">#%rewrite-infix</code>, consults an operator table, extendable via the
<code class="language-plaintext highlighter-rouge">def-operator</code> macro, to rewrite infix syntax into standard prefix
S-expressions using a Pratt parser.</p>
<p>The <code class="language-plaintext highlighter-rouge">block</code> syntax has many different interpretations. It has a macro
binding that turns it into a Racket <code class="language-plaintext highlighter-rouge">match-lambda*</code>, and it is used as
literal syntax as input to other macro definitions.</p>
<p>For example, here’s one possible implementation of that <code class="language-plaintext highlighter-rouge">for</code> syntax:</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="o">#</span><span class="nv">lang</span> <span class="nv">something</span>
<span class="nv">provide</span>
<span class="nv">for</span>
<span class="nv">require</span>
<span class="nv">for-syntax</span> <span class="nv">something/base</span>
<span class="nv">prefix-in</span> <span class="nv">base_</span> <span class="nv">racket/base</span>
<span class="nv">def-syntax</span> <span class="nv">for</span> <span class="nv">stx</span>
<span class="nv">syntax-case</span> <span class="nv">stx</span> <span class="p">(</span><span class="nf">block</span><span class="p">)</span>
<span class="nv">_</span> <span class="p">(</span><span class="nf">block</span> <span class="p">(</span><span class="nf">v</span> <span class="p">(</span><span class="nf">block</span> <span class="nv">exp</span><span class="p">))</span> <span class="o">...</span><span class="p">)</span> <span class="p">(</span><span class="nf">block</span> <span class="nv">body</span> <span class="o">...</span><span class="p">)</span>
<span class="p">(</span><span class="k">syntax</span> <span class="p">(</span><span class="nf">base_for</span> <span class="p">((</span><span class="nf">v</span> <span class="nv">exp</span><span class="p">)</span> <span class="o">...</span><span class="p">)</span> <span class="nv">body</span> <span class="o">...</span><span class="p">))</span>
<span class="nv">def-operator</span> <span class="o">..</span> <span class="mi">10</span> <span class="nv">nonassoc</span> <span class="nv">in-range</span></code></pre></figure>
<p>Notice how the <code class="language-plaintext highlighter-rouge">block</code> S-expressions are rewritten into a normal
S-expression compatible with the underlying <code class="language-plaintext highlighter-rouge">for</code> from <code class="language-plaintext highlighter-rouge">racket/base</code>.</p>
<p>Generally, all of these forms are equivalent</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">x y z x y z: x y z { a; b }
a a
b b</code></pre></figure>
<p>and they are read as</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="nf">x</span> <span class="nv">y</span> <span class="nv">z</span> <span class="p">(</span><span class="nf">block</span> <span class="nv">a</span> <span class="nv">b</span><span class="p">))</span></code></pre></figure>
<p>and are then made available to the normal macro-expansion process
(which involves a new infix-rewriting semi-phase).</p>
<p>Colons are optional to indicate a following suite at the end of an
indentation-sensitive line. Indentation-sensitivity is disabled inside
parentheses. If inside a parenthesised expression,
indentation-sensitivity can be reenabled with a colon at the end of a
line:</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="nv">a</span> <span class="nv">b</span> <span class="p">(</span><span class="nf">c</span> <span class="nv">d:</span>
<span class="nv">e</span>
<span class="nv">f</span><span class="p">)</span>
<span class="nv">=</span> <span class="p">(</span><span class="nf">a</span> <span class="nv">b</span> <span class="p">(</span><span class="nf">c</span> <span class="nv">d</span> <span class="p">(</span><span class="nf">block</span> <span class="nv">e</span> <span class="nv">f</span><span class="p">)))</span>
<span class="nv">a</span> <span class="nv">b</span> <span class="p">(</span><span class="nf">c</span> <span class="nv">d</span>
<span class="nv">e</span>
<span class="nv">f</span><span class="p">)</span>
<span class="nv">=</span> <span class="p">(</span><span class="nf">a</span> <span class="nv">b</span> <span class="p">(</span><span class="nf">c</span> <span class="nv">d</span> <span class="nv">e</span> <span class="nv">f</span><span class="p">))</span></code></pre></figure>
<p>Conversely, long lines may be split up and logically continued over
subsequent physical lines with a trailing <code class="language-plaintext highlighter-rouge">\</code>:</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="nv">a</span> <span class="nv">b</span> <span class="nv">c</span> <span class="nv">\</span>
<span class="nv">d</span> <span class="nv">\</span>
<span class="nv">e</span>
<span class="nv">=</span> <span class="p">(</span><span class="nf">a</span> <span class="nv">b</span> <span class="nv">c</span> <span class="nv">d</span> <span class="nv">e</span><span class="p">)</span></code></pre></figure>
<p>Semicolons may also appear in vertically-laid-out suites; these two
are equivalent:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">x y z
a
b; c
d
x y z { a; b; c; d }</code></pre></figure>
<p>Suites may begin on the same line as their colon. Any indented
subsequent lines become children of the portion after the colon,
rather than the portion before.</p>
<p>This example:</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="nv">x</span> <span class="nv">y</span> <span class="nv">z:</span> <span class="nv">a</span> <span class="nv">b</span>
<span class="nv">c</span> <span class="nv">d</span>
<span class="nv">e</span></code></pre></figure>
<p>reads as</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="nf">x</span> <span class="nv">y</span> <span class="nv">z</span> <span class="p">(</span><span class="nf">block</span> <span class="p">(</span><span class="nf">a</span> <span class="nv">b</span> <span class="p">(</span><span class="nf">block</span> <span class="p">(</span><span class="nf">c</span> <span class="nv">d</span><span class="p">)</span> <span class="nv">e</span><span class="p">))))</span></code></pre></figure>
<p>Square brackets are syntactic sugar for a <code class="language-plaintext highlighter-rouge">#%seq</code> macro:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">[a; b; c; d e f] → (#%seq a b c (d e f))
[ → (#%seq a (b (block c)) (d e f))
a
b
c
d e f
]</code></pre></figure>
<p>Forms starting with <code class="language-plaintext highlighter-rouge">block</code> in expression context expand into
<code class="language-plaintext highlighter-rouge">match-lambda*</code> like this:</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">{</span>
<span class="nv">pat1a</span> <span class="nv">pat1b</span>
<span class="nv">exp1a</span>
<span class="nv">exp1b</span>
<span class="nv">pat2a</span>
<span class="nv">exp2</span>
<span class="p">}</span>
<span class="nv">→</span> <span class="p">(</span><span class="nf">match-lambda*</span>
<span class="p">[(</span><span class="nb">list</span> <span class="nv">pat1a</span> <span class="nv">pat1b</span><span class="p">)</span> <span class="nv">exp1a</span> <span class="nv">exp1b</span><span class="p">]</span>
<span class="p">[(</span><span class="nb">list</span> <span class="nv">pat2a</span><span class="p">)</span> <span class="nv">exp2</span><span class="p">])</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">map*</code> function exported from <code class="language-plaintext highlighter-rouge">something/base</code> differs from <code class="language-plaintext highlighter-rouge">map</code>
in <code class="language-plaintext highlighter-rouge">racket/base</code> in that it takes its arguments in the opposite order,
permitting maps to be written</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">map* [1; 2; 3; 4]
item:
item + 1
map* [1; 2; 3; 4]
item: item + 1
map* [1; 2; 3; 4]: item: item + 1
map* [1; 2; 3; 4] { item: item + 1 }</code></pre></figure>
<p>A nice consequence of all of the above is that curried functions have
an interesting appearance:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">def curried x:: y:: z:
[x; y; z]
require rackunit
check-equal? (((curried 1) 2) 3) [1; 2; 3]</code></pre></figure>
<p>A few more links:</p>
<ul>
<li>
<p><a href="https://github.com/tonyg/racket-something">The repository</a></p>
</li>
<li><code class="language-plaintext highlighter-rouge">#lang something/shell</code> in action: <a href="https://asciinema.org/a/83450">https://asciinema.org/a/83450</a></li>
<li><a href="https://github.com/tonyg/racket-something/blob/4a00a9a6d37777f5aab4f28b03c53555b87e21d7/src/something/shell.rkt">Implementation of <code class="language-plaintext highlighter-rouge">#lang something/shell</code></a></li>
<li><a href="https://github.com/tonyg/racket-something/blob/4a00a9a6d37777f5aab4f28b03c53555b87e21d7/examples/sh.rkt">An example “shell script”</a></li>
</ul>
<p>Another small example “shell script”, which I used while working on the module today:</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="o">#</span><span class="nv">lang</span> <span class="nv">something/shell</span>
<span class="nv">def</span> <span class="nv">in-hashlang?</span> <span class="nv">re</span> <span class="nv">::</span> <span class="nv">source-filename</span>
<span class="nv">zero?</span> <span class="p">(</span><span class="nf">cat</span> <span class="nv">source-filename</span> <span class="nv">|</span> <span class="nv">grep</span> <span class="nv">-Eq</span> <span class="p">(</span><span class="nb">format</span> <span class="s">"^#lang.*(~a).*"</span> <span class="nv">re</span><span class="p">))</span>
<span class="p">(</span><span class="nf">find</span> <span class="s">"."</span> <span class="nv">-iname</span> <span class="s">"*.rkt"</span>
<span class="nv">|</span> <span class="nv">port->lines</span>
<span class="nv">|></span> <span class="nv">filter</span> <span class="p">(</span><span class="nf">in-hashlang?</span> <span class="s">"something"</span><span class="p">)</span>
<span class="nv">|></span> <span class="nv">ag</span> <span class="nv">-F</span> <span class="s">"parameterize"</span><span class="p">)</span></code></pre></figure>
Extensible Double Dispatch for Racket2016-07-24T22:59:58+00:00http://eighty-twenty.org/2016/07/24/extensible-double-dispatch-for-rackettonyg<p>Both Racket’s <a href="http://docs.racket-lang.org/reference/mzlib_class.html">object
system</a>
and its (separate!) <a href="http://docs.racket-lang.org/reference/struct-generics.html">generic interface
system</a>
offer <em>single-dispatch</em> object-oriented programming: the choice of
method body to execute depends on the type of just one of the
arguments given to the method, usually the first one.</p>
<p>In some cases, the first thing that a method will do is to decide
what to do next based on the type of a second argument. This is
called <em>double dispatch</em>, and it has a long history in
object-oriented programming languages—at least as far back as the
original Smalltalk.</p>
<p>As an example, consider implementing addition for classes
representing numbers. A different method body would be needed
for each pair of representations of numbers.</p>
<p>I stumbled across the need for something like this when
<a href="https://github.com/tonyg/racket-operational-transformation/#readme">implementing Operational Transformation (OT) for
Racket</a>.
The macro
<a href="https://github.com/tonyg/racket-operational-transformation/blob/988d411/operational-transformation-lib/operational-transformation/operation.rkt#L67-L76"><code class="language-plaintext highlighter-rouge">operation-transformer</code></a>
in that code base is <em>almost</em> the <code class="language-plaintext highlighter-rouge">double-dispatch</code> macro from
this post; the difference is that for operational transformation,
the method concerned yields <em>two</em> results, and if the arguments
are switched on the way in, they must be switched on the way out.</p>
<h3 id="basic-double-dispatch">Basic Double Dispatch</h3>
<p>Here’s a basic double-dispatch macro:</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="k">define-syntax-rule</span> <span class="p">(</span><span class="nf">double-dispatch</span> <span class="nv">op</span> <span class="p">(</span><span class="nf">arg1</span> <span class="nv">arg2</span><span class="p">)</span> <span class="p">[</span><span class="nf">pred?</span> <span class="nv">body</span> <span class="o">...</span><span class="p">]</span> <span class="o">...</span><span class="p">)</span>
<span class="p">(</span><span class="k">cond</span>
<span class="p">[(</span><span class="nf">pred?</span> <span class="nv">arg2</span><span class="p">)</span> <span class="nv">body</span> <span class="o">...</span><span class="p">]</span> <span class="o">...</span>
<span class="p">[</span><span class="nf">else</span> <span class="p">(</span><span class="nb">error</span> <span class="ss">'op</span> <span class="s">"Unimplemented for ~v and ~v"</span> <span class="nv">arg1</span> <span class="nv">arg2</span><span class="p">)]))</span></code></pre></figure>
<p>It assumes that it will be used in a method where dispatch has
already been done on <code class="language-plaintext highlighter-rouge">arg1</code>, and that the next step is to inspect
<code class="language-plaintext highlighter-rouge">arg2</code>. It applies the <code class="language-plaintext highlighter-rouge">pred?</code>s in sequence until one of them
answers true, and then evaluates the corresponding body. If none
of the <code class="language-plaintext highlighter-rouge">pred?</code>s hold, it signals an error.</p>
<p>It’s often convenient to use it inside a class definition or
generic interface implementation with the following macros, which
simply define <code class="language-plaintext highlighter-rouge">op</code> to delegate immediately to <code class="language-plaintext highlighter-rouge">double-dispatch</code>.
The first is to be used with Racket’s object system, where the
first argument is bound implicitly to <code class="language-plaintext highlighter-rouge">this</code> and where predicates
should use Racket’s
<a href="file:///Users/tonyg/src/racket/racket/doc/reference/objectutils.html#%28def._%28%28lib._racket%2Fprivate%2Fclass-internal..rkt%29._is-a~3f%29%29"><code class="language-plaintext highlighter-rouge">is-a?</code></a>
function. The second is to be used with Racket’s generic interface
system, where both arguments are explicitly specified and
predicates are more general.</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="k">define-syntax-rule</span> <span class="p">(</span><span class="nf">define/public/double-dispatch</span> <span class="p">(</span><span class="nf">op</span> <span class="nv">arg2</span><span class="p">)</span> <span class="p">[</span><span class="nf">class</span> <span class="nv">body</span> <span class="o">...</span><span class="p">]</span> <span class="o">...</span><span class="p">)</span>
<span class="p">(</span><span class="nf">define/public</span> <span class="p">(</span><span class="nf">op</span> <span class="nv">arg2</span><span class="p">)</span>
<span class="p">(</span><span class="nf">double-dispatch</span> <span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">a</span> <span class="nv">b</span><span class="p">)</span> <span class="p">(</span><span class="nf">send</span> <span class="nv">a</span> <span class="nv">op</span> <span class="nv">b</span><span class="p">))</span> <span class="p">(</span><span class="nf">this</span> <span class="nv">arg2</span><span class="p">)</span>
<span class="p">[(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">v</span><span class="p">)</span> <span class="p">(</span><span class="nf">is-a?</span> <span class="nv">v</span> <span class="nv">class</span><span class="p">))</span> <span class="nv">body</span> <span class="o">...</span><span class="p">]</span> <span class="o">...</span><span class="p">)))</span>
<span class="p">(</span><span class="k">define-syntax-rule</span> <span class="p">(</span><span class="nf">define/double-dispatch</span> <span class="p">(</span><span class="nf">op</span> <span class="nv">arg1</span> <span class="nv">arg2</span><span class="p">)</span> <span class="p">[</span><span class="nf">pred?</span> <span class="nv">body</span> <span class="o">...</span><span class="p">]</span> <span class="o">...</span><span class="p">)</span>
<span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">op</span> <span class="nv">arg1</span> <span class="nv">arg2</span><span class="p">)</span>
<span class="p">(</span><span class="nf">double-dispatch</span> <span class="nv">op</span> <span class="p">(</span><span class="nf">arg1</span> <span class="nv">arg2</span><span class="p">)</span> <span class="p">[</span><span class="nf">pred?</span> <span class="nv">body</span> <span class="o">...</span><span class="p">]</span> <span class="o">...</span><span class="p">)))</span></code></pre></figure>
<h3 id="commutative-double-dispatch">Commutative Double Dispatch</h3>
<p>For commutative operations like addition, it’s common to see the
same code appear for adding an A to a B as for adding a B to an A.</p>
<p>The next macro automatically flips its arguments and tries again
to see if B’s method has support for A, if it can’t find support
for B within A’s method. That way, code for combining B with A
need only be supplied in one place. It uses a
<a href="http://docs.racket-lang.org/guide/parameterize.html">parameter</a>
to keep track of whether it’s currently trying out a flipped pair
of arguments.</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="k">define</span> <span class="nv">trying-flipped?</span> <span class="p">(</span><span class="nb">make-parameter</span> <span class="no">#f</span><span class="p">))</span>
<span class="p">(</span><span class="k">define-syntax-rule</span> <span class="p">(</span><span class="nf">commutative-double-dispatch</span> <span class="nv">op</span> <span class="p">(</span><span class="nf">arg1</span> <span class="nv">arg2</span><span class="p">)</span> <span class="p">[</span><span class="nf">pred?</span> <span class="nv">body</span> <span class="o">...</span><span class="p">]</span> <span class="o">...</span><span class="p">)</span>
<span class="p">(</span><span class="k">cond</span>
<span class="p">[(</span><span class="nf">pred?</span> <span class="nv">arg2</span><span class="p">)</span> <span class="p">(</span><span class="k">parameterize</span> <span class="p">((</span><span class="nf">trying-flipped?</span> <span class="no">#f</span><span class="p">))</span> <span class="nv">body</span> <span class="o">...</span><span class="p">)]</span> <span class="o">...</span>
<span class="p">[(</span><span class="nf">trying-flipped?</span><span class="p">)</span> <span class="p">(</span><span class="nb">error</span> <span class="ss">'op</span> <span class="s">"Unimplemented for ~v and ~v"</span> <span class="nv">arg2</span> <span class="nv">arg1</span><span class="p">)]</span>
<span class="p">[</span><span class="nf">else</span> <span class="p">(</span><span class="k">parameterize</span> <span class="p">((</span><span class="nf">trying-flipped?</span> <span class="no">#t</span><span class="p">))</span> <span class="p">(</span><span class="nf">op</span> <span class="nv">arg2</span> <span class="nv">arg1</span><span class="p">))]))</span></code></pre></figure>
<p>Writing a simple wrapper works well for using
<code class="language-plaintext highlighter-rouge">commutative-double-dispatch</code> in a class definition:</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="k">define-syntax-rule</span> <span class="p">(</span><span class="nf">define/public/commutative-double-dispatch</span> <span class="p">(</span><span class="nf">op</span> <span class="nv">arg2</span><span class="p">)</span> <span class="p">[</span><span class="nf">class</span> <span class="nv">body</span> <span class="o">...</span><span class="p">]</span> <span class="o">...</span><span class="p">)</span>
<span class="p">(</span><span class="nf">define/public</span> <span class="p">(</span><span class="nf">op</span> <span class="nv">arg2</span><span class="p">)</span>
<span class="p">(</span><span class="nf">commutative-double-dispatch</span> <span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">a</span> <span class="nv">b</span><span class="p">)</span> <span class="p">(</span><span class="nf">send</span> <span class="nv">a</span> <span class="nv">op</span> <span class="nv">b</span><span class="p">))</span> <span class="p">(</span><span class="nf">this</span> <span class="nv">arg2</span><span class="p">)</span>
<span class="p">[(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">v</span><span class="p">)</span> <span class="p">(</span><span class="nf">is-a?</span> <span class="nv">v</span> <span class="nv">class</span><span class="p">))</span> <span class="nv">body</span> <span class="o">...</span><span class="p">]</span> <span class="o">...</span><span class="p">)))</span></code></pre></figure>
<p>but a wrapper for use with the generic interface system needs to
take care not to accidentally shadow the outer dispatch mechanism.
This macro uses
<a href="http://docs.racket-lang.org/reference/struct-generics.html#%28form._%28%28lib._racket%2Fgeneric..rkt%29._define%2Fgeneric%29%29"><code class="language-plaintext highlighter-rouge">define/generic</code></a>
to make <code class="language-plaintext highlighter-rouge">op*</code> an alias of <code class="language-plaintext highlighter-rouge">op</code> that always does a full dispatch on
its arguments:</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="k">define-syntax-rule</span> <span class="p">(</span><span class="nf">define/commutative-double-dispatch</span> <span class="p">(</span><span class="nf">op</span> <span class="nv">arg1</span> <span class="nv">arg2</span><span class="p">)</span> <span class="p">[</span><span class="nf">pred?</span> <span class="nv">body</span> <span class="o">...</span><span class="p">]</span> <span class="o">...</span><span class="p">)</span>
<span class="p">(</span><span class="k">begin</span> <span class="p">(</span><span class="nf">define/generic</span> <span class="nv">op*</span> <span class="nv">op</span><span class="p">)</span>
<span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">op</span> <span class="nv">arg1</span> <span class="nv">arg2</span><span class="p">)</span>
<span class="p">(</span><span class="nf">commutative-double-dispatch</span> <span class="nv">op*</span> <span class="p">(</span><span class="nf">arg1</span> <span class="nv">arg2</span><span class="p">)</span> <span class="p">[</span><span class="nf">pred?</span> <span class="nv">body</span> <span class="o">...</span><span class="p">]</span> <span class="o">...</span><span class="p">))))</span></code></pre></figure>
<h2 id="examples">Examples</h2>
<p>Let’s see the system in operation! First, using Racket’s object
system, and then using Racket’s generic interfaces.</p>
<h3 id="example-scenario">Example Scenario</h3>
<p>We will first define two types of value <em>foo</em> and <em>bar</em>, each
responding to a single doubly-dispatched method, <code class="language-plaintext highlighter-rouge">operator</code> which
produces results according to the following table:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> | foo | bar |
-----|-----|-----|
foo | foo | bar |
bar | bar | foo |
-----|-----|-----|
</code></pre></div></div>
<p>Then, we’ll extend the system to include a third type, <em>zot</em>,
which yields a <em>zot</em> when combined with any of the three types.</p>
<h3 id="double-dispatch-with-classes">Double Dispatch with Classes</h3>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="k">define</span> <span class="nv">foo%</span>
<span class="p">(</span><span class="nf">class</span> <span class="nv">object%</span>
<span class="p">(</span><span class="nf">super-new</span><span class="p">)</span>
<span class="p">(</span><span class="nf">define/public/commutative-double-dispatch</span> <span class="p">(</span><span class="nf">operator</span> <span class="nv">other</span><span class="p">)</span>
<span class="p">[</span><span class="nf">foo%</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">foo%</span><span class="p">)]</span>
<span class="p">[</span><span class="nf">bar%</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">bar%</span><span class="p">)])))</span>
<span class="p">(</span><span class="k">define</span> <span class="nv">bar%</span>
<span class="p">(</span><span class="nf">class</span> <span class="nv">object%</span>
<span class="p">(</span><span class="nf">super-new</span><span class="p">)</span>
<span class="p">(</span><span class="nf">define/public/commutative-double-dispatch</span> <span class="p">(</span><span class="nf">operator</span> <span class="nv">other</span><span class="p">)</span>
<span class="p">[</span><span class="nf">bar%</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">foo%</span><span class="p">)])))</span></code></pre></figure>
<p>Some tests show that this is doing what we expect. Notice that we
get the right result when the first operand is a <code class="language-plaintext highlighter-rouge">bar%</code> and the
second a <code class="language-plaintext highlighter-rouge">foo%</code>, even though <code class="language-plaintext highlighter-rouge">bar%</code> only explicitly specified the
case for when the second operand is also a <code class="language-plaintext highlighter-rouge">bar%</code>. This shows the
automatic argument-flipping in operation.</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="nf">module+</span> <span class="nv">test</span>
<span class="p">(</span><span class="k">require</span> <span class="nv">rackunit</span><span class="p">)</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">is-a?</span> <span class="p">(</span><span class="nf">send</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">foo%</span><span class="p">)</span> <span class="nv">operator</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">foo%</span><span class="p">))</span> <span class="nv">foo%</span><span class="p">))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">is-a?</span> <span class="p">(</span><span class="nf">send</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">foo%</span><span class="p">)</span> <span class="nv">operator</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">bar%</span><span class="p">))</span> <span class="nv">bar%</span><span class="p">))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">is-a?</span> <span class="p">(</span><span class="nf">send</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">bar%</span><span class="p">)</span> <span class="nv">operator</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">foo%</span><span class="p">))</span> <span class="nv">bar%</span><span class="p">))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">is-a?</span> <span class="p">(</span><span class="nf">send</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">bar%</span><span class="p">)</span> <span class="nv">operator</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">bar%</span><span class="p">))</span> <span class="nv">foo%</span><span class="p">)))</span></code></pre></figure>
<h3 id="double-dispatch-with-generic-interfaces">Double Dispatch with Generic Interfaces</h3>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="nf">define-generics</span> <span class="nv">operand</span>
<span class="p">(</span><span class="nf">operator</span> <span class="nv">operand</span> <span class="nv">other</span><span class="p">))</span>
<span class="p">(</span><span class="nf">struct</span> <span class="nv">foo</span> <span class="p">()</span>
<span class="nt">#:methods</span> <span class="nv">gen:operand</span>
<span class="p">[(</span><span class="nf">define/commutative-double-dispatch</span> <span class="p">(</span><span class="nf">operator</span> <span class="nv">this</span> <span class="nv">other</span><span class="p">)</span>
<span class="p">[</span><span class="nf">foo?</span> <span class="p">(</span><span class="nf">foo</span><span class="p">)]</span>
<span class="p">[</span><span class="nf">bar?</span> <span class="p">(</span><span class="nf">bar</span><span class="p">)])])</span>
<span class="p">(</span><span class="nf">struct</span> <span class="nv">bar</span> <span class="p">()</span>
<span class="nt">#:methods</span> <span class="nv">gen:operand</span>
<span class="p">[(</span><span class="nf">define/commutative-double-dispatch</span> <span class="p">(</span><span class="nf">operator</span> <span class="nv">this</span> <span class="nv">other</span><span class="p">)</span>
<span class="p">[</span><span class="nf">bar?</span> <span class="p">(</span><span class="nf">foo</span><span class="p">)])])</span></code></pre></figure>
<p>The tests show the same argument-flipping behavior as for the
object system above.</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="nf">module+</span> <span class="nv">test</span>
<span class="p">(</span><span class="k">require</span> <span class="nv">rackunit</span><span class="p">)</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">foo?</span> <span class="p">(</span><span class="nf">operator</span> <span class="p">(</span><span class="nf">foo</span><span class="p">)</span> <span class="p">(</span><span class="nf">foo</span><span class="p">))))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">bar?</span> <span class="p">(</span><span class="nf">operator</span> <span class="p">(</span><span class="nf">foo</span><span class="p">)</span> <span class="p">(</span><span class="nf">bar</span><span class="p">))))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">bar?</span> <span class="p">(</span><span class="nf">operator</span> <span class="p">(</span><span class="nf">bar</span><span class="p">)</span> <span class="p">(</span><span class="nf">foo</span><span class="p">))))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">foo?</span> <span class="p">(</span><span class="nf">operator</span> <span class="p">(</span><span class="nf">bar</span><span class="p">)</span> <span class="p">(</span><span class="nf">bar</span><span class="p">)))))</span></code></pre></figure>
<h3 id="extending-the-example">Extending The Example</h3>
<p>First, we implement and test class <code class="language-plaintext highlighter-rouge">zot%</code>…</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="k">define</span> <span class="nv">zot%</span>
<span class="p">(</span><span class="nf">class</span> <span class="nv">object%</span>
<span class="p">(</span><span class="nf">super-new</span><span class="p">)</span>
<span class="p">(</span><span class="nf">define/public/commutative-double-dispatch</span> <span class="p">(</span><span class="nf">operator</span> <span class="nv">other</span><span class="p">)</span>
<span class="p">[</span><span class="nf">foo%</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">zot%</span><span class="p">)]</span>
<span class="p">[</span><span class="nf">bar%</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">zot%</span><span class="p">)]</span>
<span class="p">[</span><span class="nf">zot%</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">zot%</span><span class="p">)])))</span>
<span class="p">(</span><span class="nf">module+</span> <span class="nv">test</span>
<span class="p">(</span><span class="k">require</span> <span class="nv">rackunit</span><span class="p">)</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">is-a?</span> <span class="p">(</span><span class="nf">send</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">foo%</span><span class="p">)</span> <span class="nv">operator</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">zot%</span><span class="p">))</span> <span class="nv">zot%</span><span class="p">))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">is-a?</span> <span class="p">(</span><span class="nf">send</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">bar%</span><span class="p">)</span> <span class="nv">operator</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">zot%</span><span class="p">))</span> <span class="nv">zot%</span><span class="p">))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">is-a?</span> <span class="p">(</span><span class="nf">send</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">zot%</span><span class="p">)</span> <span class="nv">operator</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">foo%</span><span class="p">))</span> <span class="nv">zot%</span><span class="p">))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">is-a?</span> <span class="p">(</span><span class="nf">send</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">zot%</span><span class="p">)</span> <span class="nv">operator</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">bar%</span><span class="p">))</span> <span class="nv">zot%</span><span class="p">))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">is-a?</span> <span class="p">(</span><span class="nf">send</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">zot%</span><span class="p">)</span> <span class="nv">operator</span> <span class="p">(</span><span class="nf">new</span> <span class="nv">zot%</span><span class="p">))</span> <span class="nv">zot%</span><span class="p">)))</span></code></pre></figure>
<p>… and then implement and test struct <code class="language-plaintext highlighter-rouge">zot</code>.</p>
<figure class="highlight"><pre><code class="language-racket" data-lang="racket"><span class="p">(</span><span class="nf">struct</span> <span class="nv">zot</span> <span class="p">()</span>
<span class="nt">#:methods</span> <span class="nv">gen:operand</span>
<span class="p">[(</span><span class="nf">define/commutative-double-dispatch</span> <span class="p">(</span><span class="nf">operator</span> <span class="nv">this</span> <span class="nv">other</span><span class="p">)</span>
<span class="p">[</span><span class="nf">foo?</span> <span class="p">(</span><span class="nf">zot</span><span class="p">)]</span>
<span class="p">[</span><span class="nf">bar?</span> <span class="p">(</span><span class="nf">zot</span><span class="p">)]</span>
<span class="p">[</span><span class="nf">zot?</span> <span class="p">(</span><span class="nf">zot</span><span class="p">)])])</span>
<span class="p">(</span><span class="nf">module+</span> <span class="nv">test</span>
<span class="p">(</span><span class="k">require</span> <span class="nv">rackunit</span><span class="p">)</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">zot?</span> <span class="p">(</span><span class="nf">operator</span> <span class="p">(</span><span class="nf">foo</span><span class="p">)</span> <span class="p">(</span><span class="nf">zot</span><span class="p">))))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">zot?</span> <span class="p">(</span><span class="nf">operator</span> <span class="p">(</span><span class="nf">bar</span><span class="p">)</span> <span class="p">(</span><span class="nf">zot</span><span class="p">))))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">zot?</span> <span class="p">(</span><span class="nf">operator</span> <span class="p">(</span><span class="nf">zot</span><span class="p">)</span> <span class="p">(</span><span class="nf">foo</span><span class="p">))))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">zot?</span> <span class="p">(</span><span class="nf">operator</span> <span class="p">(</span><span class="nf">zot</span><span class="p">)</span> <span class="p">(</span><span class="nf">bar</span><span class="p">))))</span>
<span class="p">(</span><span class="nf">check-true</span> <span class="p">(</span><span class="nf">zot?</span> <span class="p">(</span><span class="nf">operator</span> <span class="p">(</span><span class="nf">zot</span><span class="p">)</span> <span class="p">(</span><span class="nf">zot</span><span class="p">)))))</span></code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
<p>Double dispatch is a useful addition to the object-oriented
programmer’s toolkit, and can be straightforwardly added to both
of Racket’s object systems using its macro facility.</p>
<hr />
<p>This post was written as executable, literate Racket. You can
download the program from
<a href="/files/double-dispatch-20160724.rkt">here</a>.</p>
Javascript syntax extensions using Ohm2016-02-05T08:57:58+00:00http://eighty-twenty.org/2016/02/05/js-syntax-extensions-using-ohmtonyg<p>Programming language designers often need to experiment with syntax
for their new language features. When it comes to Javascript, we rely
on <em>language preprocessors</em>, since altering a Javascript engine
directly is out of the question if we want our experiments to escape
the lab.</p>
<p><a href="https://github.com/cdglabs/ohm#readme">Ohm</a> is <strong>“a library and domain-specific language for parsing and
pattern matching.”</strong> In this post, I’m going to use it as a Javascript
language preprocessor. I’ll build a simple compiler for ES5 extended
with a new kind of <code class="language-plaintext highlighter-rouge">for</code> loop, using Ohm and the <a href="https://github.com/cdglabs/ohm/blob/572c7c95fa1aa6c2b225ac29b43bcb0434667c6f/examples/ecmascript/es5.ohm">ES5 grammar</a>
included with it.</p>
<p>All the code in this post is available in a <a href="https://github.com/tonyg/ohm-ecmascript-extension-example">Github repo</a>.</p>
<h2 id="our-toy-extension-for-five">Our toy extension: “for five”</h2>
<p>We will add a “for five” statement to ES5, which will let us write
programs like this:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="k">for</span> <span class="nx">five</span> <span class="k">as</span> <span class="nx">x</span> <span class="p">{</span> <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">"</span><span class="s2">We have had</span><span class="dl">"</span><span class="p">,</span> <span class="nx">x</span><span class="p">,</span> <span class="dl">"</span><span class="s2">iterations so far</span><span class="dl">"</span><span class="p">);</span> <span class="p">}</span></code></pre></figure>
<p>The new construct simply runs its body five times in a row, binding a
loop variable in the body. Running the program above through our
compiler produces:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="k">for</span> <span class="p">(</span><span class="kd">var</span> <span class="nx">x</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">x</span> <span class="o"><</span> <span class="mi">5</span><span class="p">;</span> <span class="nx">x</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span> <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">"</span><span class="s2">We have had</span><span class="dl">"</span><span class="p">,</span> <span class="nx">x</span><span class="p">,</span> <span class="dl">"</span><span class="s2">iterations so far</span><span class="dl">"</span><span class="p">);</span> <span class="p">}</span></code></pre></figure>
<h2 id="extending-the-es5-grammar">Extending the ES5 grammar</h2>
<p>We write our extension to the ES5 grammar in a new file <code class="language-plaintext highlighter-rouge">for5.ohm</code> as
follows:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">For5 <: ES5 {
IterationStatement += for five as identifier Statement -- for5_named
five = "five" ~identifierPart
as = "as" ~identifierPart
keyword += five
| as
}</code></pre></figure>
<p>Let’s take this a piece at a time. First of all, the declaration <code class="language-plaintext highlighter-rouge">For5
<: ES5</code> tells Ohm that the new grammar should be called <code class="language-plaintext highlighter-rouge">For5</code>, and
that it <em>inherits</em> from a grammar called <code class="language-plaintext highlighter-rouge">ES5</code>. Next,</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text"> IterationStatement += for five as identifier Statement -- for5_named</code></pre></figure>
<p>extends the existing ES5 grammar’s
<a href="https://github.com/cdglabs/ohm/blob/572c7c95fa1aa6c2b225ac29b43bcb0434667c6f/examples/ecmascript/es5.ohm#L432-L443">IterationStatement</a>
nonterminal with a new production that will be called
<code class="language-plaintext highlighter-rouge">IterationStatement_for5_named</code>.</p>
<p>Finally, we define two new nonterminals as convenient shorthands for
parsing the two new keywords, and augment the existing
<a href="https://github.com/cdglabs/ohm/blob/572c7c95fa1aa6c2b225ac29b43bcb0434667c6f/examples/ecmascript/es5.ohm#L85-L93">keyword</a>
definition:</p>
<figure class="highlight"><pre><code class="language-text" data-lang="text">five = "five" ~identifierPart
as = "as" ~identifierPart
keyword += five
| as</code></pre></figure>
<p>There are three interesting points to be made about keywords:</p>
<ul>
<li>
<p>First of all, making something a keyword <em>rules it out as an
identifier</em>. In our extended language, writing <code class="language-plaintext highlighter-rouge">var five = 5</code> is a
syntax error. Define new keywords with care!</p>
</li>
<li>
<p>We make sure to reject input tokens that have our new keywords as a
<em>prefix</em> by defining them as their literal text followed by
anything that <em>cannot</em> be parsed as a part of an identifier,
<code class="language-plaintext highlighter-rouge">~identifierPart</code>. That way, the compiler doesn’t get confused by,
say, <code class="language-plaintext highlighter-rouge">fivetimes</code> or <code class="language-plaintext highlighter-rouge">five_more</code>, which remain valid identifiers.</p>
</li>
<li>
<p>By making sure to extend <code class="language-plaintext highlighter-rouge">keyword</code>, tooling such as syntax
highlighters can automatically take advantage of our extension, if
they are given our extended grammar.</p>
</li>
</ul>
<h2 id="translating-source-code-using-the-new-grammar">Translating source code using the new grammar</h2>
<p>First, require the <code class="language-plaintext highlighter-rouge">ohm-js</code> NPM module and its included ES5 grammar:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="kd">var</span> <span class="nx">ohm</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">ohm-js</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">ES5</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">ohm-js/examples/ecmascript/es5.js</span><span class="dl">'</span><span class="p">);</span></code></pre></figure>
<p>Next, load our extended grammar from its definition in <code class="language-plaintext highlighter-rouge">for5.ohm</code>, and
compile it. When we compile the grammar, we pass in a namespace that
makes the ES5 grammar available under the name our grammar expects,
<code class="language-plaintext highlighter-rouge">ES5</code>:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="kd">var</span> <span class="nx">grammarSource</span> <span class="o">=</span> <span class="nx">fs</span><span class="p">.</span><span class="nx">readFileSync</span><span class="p">(</span><span class="nx">path</span><span class="p">.</span><span class="nx">join</span><span class="p">(</span><span class="nx">__dirname</span><span class="p">,</span> <span class="dl">'</span><span class="s1">for5.ohm</span><span class="dl">'</span><span class="p">)).</span><span class="nx">toString</span><span class="p">();</span>
<span class="kd">var</span> <span class="nx">grammar</span> <span class="o">=</span> <span class="nx">ohm</span><span class="p">.</span><span class="nx">grammar</span><span class="p">(</span><span class="nx">grammarSource</span><span class="p">,</span> <span class="p">{</span> <span class="na">ES5</span><span class="p">:</span> <span class="nx">ES5</span><span class="p">.</span><span class="nx">grammar</span> <span class="p">});</span></code></pre></figure>
<p>Finally, we define the translation from our extended language to plain
ES5. To do this, we extend a <em>semantic function</em>, <code class="language-plaintext highlighter-rouge">modifiedSource</code>,
adding a method for each new production rule. Ohm automatically uses
defaults for rules not mentioned in our extension.</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="kd">var</span> <span class="nx">semantics</span> <span class="o">=</span> <span class="nx">grammar</span><span class="p">.</span><span class="nx">extendSemantics</span><span class="p">(</span><span class="nx">ES5</span><span class="p">.</span><span class="nx">semantics</span><span class="p">);</span>
<span class="nx">semantics</span><span class="p">.</span><span class="nx">extendAttribute</span><span class="p">(</span><span class="dl">'</span><span class="s1">modifiedSource</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span>
<span class="na">IterationStatement_for5_named</span><span class="p">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">_for</span><span class="p">,</span> <span class="nx">_five</span><span class="p">,</span> <span class="nx">_as</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="nx">body</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">c</span> <span class="o">=</span> <span class="nx">id</span><span class="p">.</span><span class="nx">asES5</span><span class="p">;</span>
<span class="k">return</span> <span class="dl">'</span><span class="s1">for (var </span><span class="dl">'</span><span class="o">+</span><span class="nx">c</span><span class="o">+</span><span class="dl">'</span><span class="s1"> = 0; </span><span class="dl">'</span><span class="o">+</span><span class="nx">c</span><span class="o">+</span><span class="dl">'</span><span class="s1"> < 5; </span><span class="dl">'</span><span class="o">+</span><span class="nx">c</span><span class="o">+</span><span class="dl">'</span><span class="s1">++) </span><span class="dl">'</span> <span class="o">+</span> <span class="nx">body</span><span class="p">.</span><span class="nx">asES5</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">});</span></code></pre></figure>
<p>Each parameter to the <code class="language-plaintext highlighter-rouge">IterationStatement_for5_named</code> method is a
syntax tree node corresponding <em>positionally</em> to one of the tokens in
the definition of the parsing rule. Accessing the <code class="language-plaintext highlighter-rouge">asES5</code> attribute of
a syntax tree node computes its translated source code. This is done
with recursive calls to the <code class="language-plaintext highlighter-rouge">modifiedSource</code> attribute where required.</p>
<p>Our compiler is, at this point, complete. To use it, we need code to
feed it input and print the results:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="kd">function</span> <span class="nx">compileExtendedSource</span><span class="p">(</span><span class="nx">inputSource</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">parseResult</span> <span class="o">=</span> <span class="nx">grammar</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">inputSource</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">parseResult</span><span class="p">.</span><span class="nx">failed</span><span class="p">())</span> <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="nx">parseResult</span><span class="p">.</span><span class="nx">message</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">parseResult</span><span class="p">.</span><span class="nx">succeeded</span><span class="p">()</span> <span class="o">&&</span> <span class="nx">semantics</span><span class="p">(</span><span class="nx">parseResult</span><span class="p">).</span><span class="nx">asES5</span><span class="p">;</span>
<span class="p">}</span></code></pre></figure>
<p>That’s it!</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="o">></span> <span class="nx">compileExtendedSource</span><span class="p">(</span><span class="dl">"</span><span class="s2">for five as x { console.log(x); }</span><span class="dl">"</span><span class="p">);</span>
<span class="dl">'</span><span class="s1">for (var x = 0; x < 5; x++) { console.log(x); }</span><span class="dl">'</span></code></pre></figure>
<h2 id="discussion">Discussion</h2>
<p>This style of syntactic extension is quite coarse-grained: we must
translate whole compilation units at once, and must specify our
extensions separately from the code making use of them. There is no
way of adding a <em>local</em> syntax extension scoped precisely to a block
of code that needs it (known to Schemers as
<a href="http://www.schemers.org/Documents/Standards/R5RS/HTML/r5rs-Z-H-7.html#%_sec_4.3.1"><code class="language-plaintext highlighter-rouge">let-syntax</code></a>).
For Javascript, <a href="http://sweetjs.org/">sweet.js</a> offers a more Schemely
style of syntax extension than the one explored in this post.</p>
<p>Mention of sweet.js leads me to the thorny topic of
<a href="https://en.wikipedia.org/wiki/Hygienic_macro">hygiene</a>. Ohm is a
<em>parsing</em> toolkit. It lets you define new concrete syntax, but doesn’t
know anything about scope, or about how you intend to use identifiers.
After all, it can be used for languages that don’t necessarily even
<em>have</em> identifiers. So when we write extensions in the style I’ve
presented here, we must write our translations carefully to avoid
unwanted capture of identifiers. This is a tradeoff: the broad
generality of Ohm’s parsing in exchange for less automation in
identifier handling.</p>
<p>Ohm’s extensible grammars let us extend any part of the language, not
just statements or expressions. We can specify new comment syntax, new
string syntax, new formal argument list syntax, and so on. Because Ohm
is based on
<a href="https://en.wikipedia.org/wiki/Parsing_expression_grammar">parsing expression grammars</a>,
it offers
<a href="https://en.wikipedia.org/wiki/Scannerless_parsing">scannerless parsing</a>.
Altering or extending a language’s lexical syntax is just as easy as
altering its grammar.</p>
<h2 id="conclusion">Conclusion</h2>
<p>We have defined an <a href="https://github.com/cdglabs/ohm#readme">Ohm</a>-based compiler for an extension to ES5
syntax, using only a few lines of code. Each new production rule
requires, roughly, one line of grammar definition, and a short method
defining its translation into simpler constructs.</p>
<p>You can try out this little compiler, and maybe experiment with your
own extensions, by cloning its <a href="https://github.com/tonyg/ohm-ecmascript-extension-example">Github repo</a>.</p>