Saying Good-bye: HARP - Dylan Foundry

This begins a new series of blog posts that will continue over the next few months as we say "Good-bye!" to parts of Open Dylan.

Freedom comes when you learn to let go
Creation comes when you learn to say no

  -- Madonna, The Power of Good-Bye

We're beginning a process by which we'll start slimming down the compiler and the libraries, letting go of some major chunks of code, with the goal of improving the hackability of the compiler and enabling us to make new leaps in functionality.

What is HARP?

HARP is the Harlequin Abstract RISC Processor and was designed and developed at Harlequin in the late 1980s. It was used in Harlequin's LispWorks and later translated to Dylan for use in Harlequin's DylanWorks (which is now Open Dylan).

Clive Tong, an engineer at Harlequin in 1989, briefly described it as:

The compiler targeted an instruction set known as HARP (Harlequin Abstract RISC Processor), and then HARP instructions were translated into machine instructions using a template matching scheme. HARP had an infinite set of registers, and the register colouring happened as part of this templating processing.

Some additional details about the early design of HARP are available in Techniques for Dynamic Software Migration from 1988. Other early sources of information about it, written by Hunter and Knightbridge, appear to be lost to the sands of time.

HARP is a fairly complicated beast handling conversion from a generic instruction set to the target instruction set, register allocation, debug info generation, creation of platform-specific binary formats and more.

It is an impressive technical feat and is similar, to some extent, to other compiler frameworks such as those which exist within GCC and LLVM.

Why Are We Saying Good-Bye?

The version of HARP that is in Open Dylan contains support for the 32-bit x86 architecture, as it existed at some point in the 1990s. It doesn't know about subsequent additions to the architecture, nor does it know about differences in scheduling and other micro-architecture details.

While the maintainers of LispWorks have developed additional platform support, doing so for Open Dylan requires more resources than we have available.

To serve all of our target platforms, we would need to add support for 32-bit ARM, 64-bit ARM, 64-bit x86_64, as well as support for DWARF debug info, and better support for generating ELF and Mach-O executable formats.

At that point, we are better served by moving to using the LLVM framework and tools. LLVM is supported by a broad community

What Do We Gain?

By abandoning HARP, we gain back some flexibility as we are better able to extend the C and LLVM compiler back-ends for new functionality.

New Primitives

We would like to add new compiler primitives for things like popcount and SIMD instructions (SSE on x86/x86_64, NEON on ARM/ARM64). Up until now, we have been holding off on this sort of functionality as we didn't know how to correctly implement it in HARP.

Adding new primitives that map to hardware capabilities is pretty easy to do in the C and LLVM backends, so long as the C compiler and LLVM support the new functionality. Given that LLVM powers many system compilers and directly receives support from CPU makers, it is usually up to date and ready to go well before we are.

Debugging

The current HARP back-end only supports generating CV4 debug info for Microsoft Windows. It does not support modern Windows build and development tools which expect PDB files. While we are sacrificing our Windows support in the short term, in the longer term, when the LLVM back-end is ready, our Windows support will be much improved from what it is now, as we will be leveraging the resources and the work that has gone into the LLVM Windows support.

On other platforms, we will no longer be hobbled by the poor debug information present in the code generated by HARP. Instead, we'll have DWARF debug info from the C and LLVM back-ends. We have already started working with LLDB to improve support for Open Dylan.

Reduced Iteration Time

The HARP back-end contains a fairly significant amount of code across multiple libraries. This adds noticeably to our build process, especially when performing 3-stage bootstraps. Removing HARP will shorten this process.

Motivation

"When one burns one's bridges, what a very nice fire it makes."

  -- Dylan Marlais Thomas

Getting rid of HARP makes it easier to start making changes that require work on our compiler back-ends. This could lead to some exciting work getting done.

It also means that since (as we'll see in a moment), we're losing Windows support, it may help motivate us to get the LLVM back-end functioning and the default compiler back-end where it is supported.

What Do We Lose?

Our LLVM support is not yet functioning, so dropping HARP will leave us with a single functioning compiler back-end: the C back-end. As a result, until the LLVM back-end is functioning, we will not have support for the Windows platform.

Also, our support for the Memory Pool System (MPS) garbage collector only currently works for the HARP back-end. With the C back-end, we use the Boehm GC. We will work with the folks at RavenBrook to restore our support for the MPS GC.

When?

I already have a branch that removes almost everything related to HARP. It clearly leaves Windows in a broken state, which is where it will remain until we have the LLVM back-end in place.

I plan to land this branch sometime during the month of September.