Reducing Dispatch Overhead

Method dispatch in Dylan is the process by which the compiler and run-time collaborate to choose the right implementation of a function to call. This can get rather complex as a number of factors are involved in choosing the right method to call and whether that can be done at compile-time or deferred to run-time.

Unfortunately, full generic method dispatch at run-time in Open Dylan is currently not as fast as we would like. This means that when performance issues strike, they may well be due to the overhead of method dispatch.

Discussing and improving the overall performance of method dispatch isn't the subject of this post. That's going to require a fair bit of planning and work before that is resolved.

Instead, we're going to look at what to do when you're experiencing performance problems at particular call sites due to the overhead of method dispatch.

Sometimes, this is readily visible in the profiler:

/static/images/method_dispatch_in_profiler.png

Here, we can see that the percent-encode method in the uri library is going through dispatch to call member?. (I've translated from the names of the functions in C to their names in Dylan. The name mangling isn't that ...

read more »

There are comments.

Early Thoughts on Atom

I got a beta of the Atom editor soon after its initial release. The first thing that I decided to do was to add support for the Dylan language.

As we had a Textmate bundle available, this was an easy process:

apm init -p ~/.atom/packages/language-dylan \
  -c https://github.com/textmate/dylan.tmbundle

This was the start of a pleasurable experience with Atom. Creating and publishing a new package was easy and the command line tools were easy to use (and documented). After just a couple of minutes, I had a package that provided basic syntax highlighting and code folding for Dylan. It also worked on any Dylan file that I opened, unlike using our Textmate bundle in SublimeText which would occasionally hang on some files (while Textmate was fine). How's that for extensibility? :)

Next up, I decided to improve upon our new Dylan package. So I added support for auto-indenting and code snippets. This too was straight forward and easy.

As a comparison, I've also worked in the past with some other Dylan hackers on a plugin for IntelliJ to provide Dylan language support. While many things in IntelliJ are relatively easy to do, they all ...

read more »

There are comments.

Eliminating Copies in C-FFI

We've run into a situation fairly regularly when wrapping C libraries where we'd like to avoid making extra copies of data when passing it between Dylan and C code via the C-FFI. Related to this, we want to be able to use multiple types from the Dylan side of things such as <buffer>, <byte-vector> or <byte-string> without having to have separate code paths for each of them. Each of these classes can store raw byte data that we may want to share with other code.

An example that we'll work with in this discussion comes from bindings (in simplified and excerpted forms) for the LevelDB library:

extern void leveldb_put(
    leveldb_t* db,
    const leveldb_writeoptions_t* options,
    const char* key, size_t keylen,
    const char* val, size_t vallen,
    char** errptr);

<C-string>

The easiest way to bind leveldb_put would be to use <C-string>:

define C-function %leveldb-put
  input parameter db_ :: <leveldb-t*>;
  input parameter options_ :: <leveldb-writeoptions-t*>;
  input parameter key_ :: <C-string>;
  input parameter keylen_ :: <size-t>;
  input parameter val_ :: <C-string>;
  input parameter vallen_ :: <size-t>;
  output parameter errptr_ :: <char**>;
  c-name: "leveldb_put";
end;

However, this would mean that every key and value would have to be copied from an existing byte container (like <buffer>, <string> or <byte-vector>) to ...

read more »

There are comments.

Thoughts on a New Dylan Ecosystem

Dylan is in an unusual position for a programming language.

It has a pretty mature implementation, a specification, and a lot of high quality documentation. It has a long history and many people remember it from the early days of its history when it was being developed at Apple and other institutions.

On the other hand, the current community is fairly small as is the available set of libraries. This is typically a disadvantage, however it can be used to the community's advantage in a couple of ways.

Looking Forward

One way is that we can look forward and take advantage of what we know and want now without having to worry about backward compatibility.

A parallel might be seen with the early days of Node.js. Early on, Node.js was also working with a programming language with a specification and a mature implementation. However, Node.js was using JavaScript in a new way: to create server-side software employing asynchronous and non-blocking techniques. A big advantage that Node.js had was that there wasn't a large collection of libraries for doing server-side programming. The developers could (largely) start from a clean slate, producing libraries that fit in ...

read more »

There are comments.

A New IDE Approach

/static/images/DeftIDEA_smaller.png

We've previously written about the state of the Open Dylan IDE. In short, it is currently only available on Windows and looks like an application from the 1990s (because it is). Simply getting it running on Mac OS X or Linux would be a lot of work (an attempt was made to port the underlying GUI framework), not counting modernization and adding new features.

In late September, we had a seemingly crazy idea: What if we built a new IDE on an existing IDE framework. We've previously worked with Eclipse and didn't want to do that again. IntelliJ, however, officially supports IDEs for Java, Python, Ruby, PHP, and Objective C. Unofficially, it has support for a host of other languages including Erlang, Lua, Scala, HaXe and many others.

So, we wondered, what would it take to get some basics running for Dylan? In the end, the answer was that within 10 hours of first looking at IntelliJ plug-in authoring, we had it recognizing file types, doing some basic parsing for LID files and some other minor things.

This led to the creation of DeftIDEA, a new Dylan Environment For Tools running inside of IntelliJ IDEA.

Why did ...

read more »

There are comments.

Last Month's Hack-a-thon

Last month, we had our first hack-a-thon in many years. Attendance was spread out over the weekend and some great contributions got completed.

  • Carl Gay documented our regular expressions library.
  • Peter Housel made more solid progress on our LLVM compiler back-end.
  • Dustin Voss improved some aspects of our support for limited collections.
  • Ingo Promovicz improved support for locking I/O streams.
  • Bruce Mitchener worked on merging run-time libraries to help provide more consistent support for various features across the various compiler back-ends.
  • Francesco Ceccon worked on the gobject-introspection bindings and the generator for creating C-FFI bindings for the GTK+ family of libraries.
  • Hannes Mehnert addressed some issues in the networking library surrounding localhost connections.
  • Andreas Bogk returned from the dead and observed the goings on after a multi-year absence.
  • Tom Emerson began investigating providing support for Unicode in Dylan.
  • Robert Roland enhanced the sqlite bindings.

Thanks to everyone who participated and we're looking forward to our next hack-a-thon in a couple of months.

All of these changes and much much more will be present in our upcoming 2013.2 release!

read more »

There are comments.

Open Dylan's Compiler Back-ends

Open Dylan has 3 compiler back-ends currently:

  • HARP
  • C
  • LLVM

Harp

The HARP back-end generates native code for x86 processors and is used on Windows, FreeBSD and Linux. While for many years, this was our best and most mature compiler back-end, it is falling by the wayside and will be replaced by the LLVM back-end.

The issues with the HARP back-end are that it is 32-bit only and is more difficult to add support for things like new instructions, better scheduling, atomic operations and SSE support.

C

The C back-end generates C which is then compiled by either clang or gcc. We use the C back-end on Mac OS X, as well as Linux and FreeBSD on x86_64 processors.

The C back-end is relatively easy to work with and extend and is the easiest option for bringing up Open Dylan on a new platform. We are currently doing this with Linux on the ARM processor.

The C back-end used to generate code that was slower than HARP although in recent months, the C back-end has been optimized and many parts of it produce faster results than HARP.

The C back-end allows debugging the generated C (rather than at the Dylan ...

read more »

There are comments.

Binding nanomsg with melange

For those who haven't heard about it, nanomsg is a great library from Martin Sustrik. It provides high performance messaging with a simple API similar to that of BSD sockets. It bears some similarity to what someone might do with ZeroMQ and is licensed under the liberal terms of the MIT license. The nanomsg library is currently in a pre-alpha state.

I have recently written bindings for it for use with Dylan which are available as nanomsg-dylan.

In Dylan, we have a couple of options when writing bindings. We have a low level direct-c-ffi and a higher level C-FFI. Using the lower level interface is fairly tedious and verbose while using C-FFI is fairly convenient. But writing a binding using either involves a lot of work and hand-translation of the C APIs into the right Dylan definitions. (Currently, the direct-c-ffi system is not documented.)

This is where the melange tool is very useful. Melange can parse C headers and automatically generate the C-FFI bindings. While doing this code generation, it also handles details like automatically translations the C names into the correct Dylan names following the Dylan conventions.

Melange now has some preliminary documentation.

Using Melange

Melange generates a ...

read more »

There are comments.

Windows: Current State of Support

Related to the recent post Why is the OpenDylan IDE only on Windows?, another source of common questions are some of the requirements that we have for running on Windows and the state of 64 bit Windows support.

Building Open Dylan Applications on Windows

Building an Open Dylan application on any platform requires at least a linker. On Windows, it also requires a resource compiler for handling .rc files which controls things like an applications icons.

On Windows, we support debugging an application written in Dylan when using the Open Dylan IDE. To do this, we must have the debug data available. Our debug data on Windows is in CodeView 4 (CV4) format rather than the more modern PDBs in use today. This is partially a historical artifact of the fact that this code was written in the 1990s and partially due to the later formats not being publicly specified.

So, to link on Windows and support debugging, we need a linker that supports CV4 debug data. Unfortunately, Microsoft removed support for this in their linkers after Visual C 6. However, the PellesC linker, polink does support CV4 debug data.

PellesC also provides a resource compiler, porc.exe.

PellesC provides ...

read more »

There are comments.

Why is the OpenDylan IDE only on Windows?

We're often asked when the IDE will show up on Linux or Mac OS X.

Unfortunately, the answer to this is rather complex because the IDE consists of multiple pieces, all of which have separate portability issues.


The Existing IDE

DUIM - The GUI Framework

The biggest reason that the IDE is only on Windows is that it uses the DUIM framework which hasn't yet been ported away from Windows in a functioning and well-maintained manner. (An experimental GTK+ port has existed in the past, but it doesn't currently compile.)

Code Browsers and Inspectors

The code browsers and inspectors pull their information from frameworks that are independent of the GUI. It is possible to use this information separately from the IDE itself. The DIME environment for emacs supplies some of this functionality. The Hula project (now defunct) provided some of this functionality as well via a web-based interface.

This is actually an exciting set of features that our compiler provides. While many other languages have to glue IDE functionality on as a separate set of libraries, Open Dylan's compiler supports it internally from the ground up.

Debugging

The debugger depends on a couple of key components:

  • A ...
read more »

There are comments.