guile 2.2 omg!!!

Oh, good evening my hackfriends! I am just chuffed to share a thing with yall: tomorrow we release Guile 2.2.0. Yaaaay!

I know in these days of version number inflation that this seems like a very incremental, point-release kind of a thing, but it's a big deal to me. This is a project I have been working on since soon after the release of Guile 2.0 some 6 years ago. It wasn't always clear that this project would work, but now it's here, going into production.

In that time I have worked on JavaScriptCore and V8 and SpiderMonkey and so I got a feel for what a state-of-the-art programming language implementation looks like. Also in that time I ate and breathed optimizing compilers, and really hit the wall until finally paging in what Fluet and Weeks were saying so many years ago about continuation-passing style and scope, and eventually came through with a solution that was still CPS: CPS soup. At this point Guile's "middle-end" is, I think, totally respectable. The backend targets a quite good virtual machine.

The virtual machine is still a bytecode interpreter for now; native code is a next step. Oddly my journey here has been precisely opposite, in a way, to An incremental approach to compiler construction; incremental, yes, but starting from the other end. But I am very happy with where things are. Guile remains very portable, bootstrappable from C, and the compiler is in a good shape to take us the rest of the way to register allocation and native code generation, and performance is pretty ok, even better than some natively-compiled Schemes.

For a "scripting" language (what does that mean?), I also think that Guile is breaking nice ground by using ELF as its object file format. Very cute. As this seems to be a "Andy mentions things he's proud of" segment, I was also pleased with how we were able to completely remove the stack size restriction.

high fives all around

As is often the case with these things, I got the idea for removing the stack limit after talking with Sam Tobin-Hochstadt from Racket and the PLT group. I admire Racket and its makers very much and look forward to stealing fromworking with them in the future.

Of course the ideas for the contification and closure optimization passes are in debt to Matthew Fluet and Stephen Weeks for the former, and Andy Keep and Kent Dybvig for the the latter. The intmap/intset representation of CPS soup itself is highly endebted to the late Phil Bagwell, to Rich Hickey, and to Clojure folk; persistent data structures were an amazing revelation to me.

Guile's virtual machine itself was initially heavily inspired by JavaScriptCore's VM. Thanks to WebKit folks for writing so much about the early days of Squirrelfish! As far as the actual optimizations in the compiler itself, I was inspired a lot by V8's Crankshaft in a weird way -- it was my first touch with fixed-point flow analysis. As most of yall know, I didn't study CS, for better and for worse; for worse, because I didn't know a lot of this stuff, and for better, as I had the joy of learning it as I needed it. Since starting with flow analysis, Carl Offner's Notes on graph algorithms used in optimizing compilers was invaluable. I still open it up from time to time.

While I'm high-fiving, large ups to two amazing support teams: firstly to my colleagues at Igalia for supporting me on this. Almost the whole time I've been at Igalia, I've been working on this, for about a day or two a week. Sometimes at work we get to take advantage of a Guile thing, but Igalia's Guile investment mainly pays out in the sense of keeping me happy, keeping me up to date with language implementation techniques, and attracting talent. At work we have a lot of language implementation people, in JS engines obviously but also in other niches like the networking group, and it helps to be able to transfer hackers from Scheme to these domains.

I put in my own time too, of course; but my time isn't really my own either. My wife Kate has been really supportive and understanding of my not-infrequent impulses to just nerd out and hack a thing. She probably won't read this (though maybe?), but it's important to acknowledge that many of us hackers are only able to do our work because of the support that we get from our families.

a digression on the nature of seeking and knowledge

I am jealous of my colleagues in academia sometimes; of course it must be this way, that we are jealous of each other. Greener grass and all that. But when you go through a doctoral program, you know that you push the boundaries of human knowledge. You know because you are acutely aware of the state of recorded knowledge in your field, and you know that your work expands that record. If you stay in academia, you use your honed skills to continue chipping away at the unknown. The papers that this process reifies have a huge impact on the flow of knowledge in the world. As just one example, I've read all of Dybvig's papers, with delight and pleasure and avarice and jealousy, and learned loads from them. (Incidentally, I am given to understand that all of these are proper academic reactions :)

But in my work on Guile I don't actually know that I've expanded knowledge in any way. I don't actually know that anything I did is new and suspect that nothing is. Maybe CPS soup? There have been some similar publications in the last couple years but you never know. Maybe some of the multicore Concurrent ML stuff I haven't written about yet. Really not sure. I am starting to see papers these days that are similar to what I do and I have the feeling that they have a bit more impact than my work because of their medium, and I wonder if I could be putting my work in a more useful form, or orienting it in a more newness-oriented way.

I also don't know how important new knowledge is. Simply being able to practice language implementation at a state-of-the-art level is a valuable skill in itself, and releasing a quality, stable free-software language implementation is valuable to the world. So it's not like I'm negative on where I'm at, but I do feel wonderful talking with folks at academic conferences and wonder how to pull some more of that into my life.

In the meantime, I feel like (my part of) Guile 2.2 is my master work in a way -- a savepoint in my hack career. It's fine work; see A Virtual Machine for Guile and Continuation-Passing Style for some high level documentation, or many of these bloggies for the nitties and the gritties. OKitties!

getting the goods

It's been a joy over the last two or three years to see the growth of Guix, a packaging system written in Guile and inspired by GNU stow and Nix. The laptop I'm writing this on runs GuixSD, and Guix is up to some 5000 packages at this point.

I've always wondered what the right solution for packaging Guile and Guile modules was. At one point I thought that we would have a Guile-specific packaging system, but one with stow-like characteristics. We had problems with C extensions though: how do you build one? Where do you get the compilers? Where do you get the libraries?

Guix solves this in a comprehensive way. From the four or five bootstrap binaries, Guix can download and build the world from source, for any of its supported architectures. The result is a farm of weirdly-named files in /gnu/store, but the transitive closure of a store item works on any distribution of that architecture.

This state of affairs was clear from the Guix binary installation instructions that just have you extract a tarball over your current distro, regardless of what's there. The process of building this weird tarball was always a bit ad-hoc though, geared to Guix's installation needs.

It turns out that we can use the same strategy to distribute reproducible binaries for any package that Guix includes. So if you download this tarball, and extract it as root in /, then it will extract some paths in /gnu/store and also add a /opt/guile-2.2.0. Run Guile as /opt/guile-2.2.0/bin/guile and you have Guile 2.2, before any of your friends! That pack was made using guix pack -C lzip -S /opt/guile-2.2.0=/ guile-next glibc-utf8-locales, at Guix git revision 80a725726d3b3a62c69c9f80d35a898dcea8ad90.

(If you run that Guile, it will complain about not being able to install the locale. Guix, like Scheme, is generally a statically scoped system; but locales are dynamically scoped. That is to say, you have to set GUIX_LOCPATH=/opt/guile-2.2.0/lib/locale in the environment, for locales to work. See the GUIX_LOCPATH docs for the gnarlies.)

Alternately of course you can install Guix and just guix package -i guile-next. Guix itself will migrate to 2.2 over the next week or so.

Welp, that's all for this evening. I'll be relieved to push the release tag and announcements tomorrow. In the meantime, happy hacking, and yes: this blog is served by Guile 2.2! :)

14 responses

  1. rain1 says:

    Congratulations! Thank you for keeping the wonderful scheme language alive and well! The guile compiler is a fantastic tool both for the practical getting work done and the theoretical - learning scheme/programming/compiler implementation details.

  2. John Cowan says:

    This is good news, because I am about to reinstall my large suite of Schemes on my home system using Ubuntu on Windows, so I'll hold off on installing Guile. I do hope I don't have to wait hours, like I did the last time I installed Guile from source.

  3. Julian Graham says:

    Andy, you rock! Thanks so much for all your hard work on this. Can't wait to start using the official release.

  4. Javier Sancho says:

    Thank you so much, Andy! Your work and your posts are undoubtedly taking us to the top.

  5. gasche says:

    I think you should seriously consider writing an academic article about Guile, as a way indeed to present and transfer your knowledge and design choices for the system. For example, the deadline for article submission for the next "Trends of Functional Programming" symposium is on May 5 (the conference is June 19-22 in Canterbury, UK), and looking at last year's programme ( http://www.tifp.org/ ) could give you an idea of what the conference feels like -- it is more open to implementation-oriented works than some other conferences, which is why I thought of it as a reasonable first step. Surely you could enlist the help of some colleagues that were recently in academia or still there for the "how do I structure/write a damn paper?" and "how do I get feedback on my draft?" parts.

  6. gasche says:

    Sigh, frames; http://tfp2016.org/accepted.html

  7. Charles says:

    Congratulations on 2.2! It's been a pleasure to watch Guile improve under your watch and see the new applications it has found as a result. I look forward to trying out the latest. Also, thank you for all the informative and edifying blog posts on language design and implementation.

  8. amz3 says:

    Thanks for this awesome release. I am not not a compiler/VM dev just a user but AFAIK Guile offers what's required to build software. It's seems to me Guile 2.2 is similar in terms of features compared to CPython 3.6 except that Guile doesn't have a GIL.

    Guile only lakes more users :)

  9. RobertX says:

    How does Windows fit into this? Is it even a target?

  10. Johnny says:

    I just looked through ecraven's benchmarks. Sure they are benchmarks, and they onlt show a part of the truth, but consistently being AT LEAST 2x faster must mean something :)

    Well done!

  11. Adam says:

    When you talk about "the multicore Concurrent ML stuff" that you have yet to write about, is that stuff that is publicly available in your fibers repoo, or something else? In the ML world, this has been the subject of ongoing research but I'm curious what you've done.

  12. wingo says:

    Heya Adam,

    I understand that Manticore is a working multicore CML system. The work I have is in Fibers and is mostly in operations.scm, and pretty much follows Manticore. See also channels.scm in that same directory, and internal.scm for the scheduler implementation. In Fibers we coalesce the "poll" and "do" phases into a single "try" phase; otherwise it's pretty much the same as Manticore, though the channel protocol is atomic (no spinlocks). I need to do some more benchmarking and will blog about that sometime soon :)

    Andy

  13. Adam says:

    Andy - Thanks! Really interesting stuff, hope you blog about it soon. I'm hoping more people learn about CML since it solves a lot of the problems I have as a UI programmer.

  14. mats says:

    I looked through the benchmarks by ecraven, and the speedup is incredible! Ported a bunch of python things (project euler solutions) to guile2.2 and had an instant 3-10x speedup.

Comments are closed.