I'm telling you

29 February 2008 5:18 PM (scheme | meta | guile | git)

A little while ago I got a mail from my good friend Leif, on whose laurels or electrons I have been resting, freeloading on his colocated server. Perhaps this was an acceptable arrangement when I was teaching in Namibia, earning five dollars a day, but now that I earn my keep programming for the man it is a less tenable position.

In any case, the mail went like this: "Wingo, find a new hosting arrangement." Or something to that effect. So I bought a slice and started to migrate my services there.

It is at this point in the story that I experienced a bit of revulsion at the thought of migrating my blog product, powered by Wordpress, to the new server. Wordpress is (1) plagued with security vulnerabilities, and (2) written in PHP, neither of which I was excited to have on my fresh clean slice.

Hence, the bad idea: what if I wrote my own weblog software, so that I wouldn't have to install PHP?

Then, the idea whose worth I have not yet fully evaluated: what if I wrote said weblog software in Scheme?

Then, constraints result in awesomeness: I considered how I could use a database from my usual Scheme implementation, Guile. There are no standard database bindings for Guile, nothing like Perl's DBI or Python's DB-API, and the only binding that I knew of, for PostGres, was bitrotten. (Incidentally the PG binding has been updated within the last couple weeks.) But then I thought, why not use git's object database as a persistent store?

tekuti is born

So in this way I turned a terrible idea into a great one: write weblog software, written in Scheme for fun, and using Git as the persistent storage layer. Three weeks later, if you are reading this, Tekuti has the basics working great.

If you visit my web site, it's probably not very impressive. Some things aren't even implemented yet, for example the tags/ tree, and archives aren't yet exposed to the user. But under the hood there is much new excellence, and quite a bit of old excellence. By way of explaining what I mean, first a bit of background about the architecture.

Tekuti is implemented as a long-running Guile process sitting behind Apache. Tekuti listens on a TCP port, and Apache connects to the port for every request that it receives, sending headers and post data, and receiving headers and body to send to the client. This protocol is implemented by the misnamed mod_lisp, which actually has nothing to do with Lisp, but is just used as a way to use Apache as your frontend while processing requests with some kind of long-running process. FastCGI is similar, but with a more complicated protocol.

Like I mentioned, Git's object database is used as a persistent store. Tekuti is configured to look at a particular directory for a "bare" git repository. The format of the store is rather simple: each post is in a tree. There is a blob for post metadata, and a blob for the raw post content. There is a comment tree under each post tree, and each comment is one blob, somewhat like an rfc 822 mail message.

Clearly, however, this format is not appropriate for all access requirements of weblog software. For example, a request to the main page would involve scanning all of the metadata blobs, parsing post timestamps out of all of them, then taking the latest N timestamps, and showing those posts. This process might work fine if you have 10 posts, but with 200 it is a bit more problematic.

Hence, Tekuti keeps an index of the persistent store. The index is extensible. It currently stores a pre-parsed and time-ordered copy of post metadata, and a mapping between tags and posts for making a tag cloud or for navigating by tag. In that way showing the last N posts is a simple matter of looking at the index, then pulling specific post contents out of the database with git show (and subsequent post-processing).

The names of the post trees are made in such a way that going to 2008/01/19/hackings directly translates into a tree name, so that lookup of individual posts is O(1).

I should mention that the entirety of Tekuti, modulo a couple of local procedures, is written in the functional style. Once allocated, data is never mutated.

Oddly enough, this immutable style of programming promotes the mutability of the program itself. If you run tekuti with the --repl argument, it spawns the main listening and processing loop in a thread, leaving the main thread to run a read-eval-print loop. If you redefine a function in the REPL, it is picked up immediately by the processing thread. My current deployment strategy is to run it within a screen session, so my REPL is always available to me. I just have to SSH into my server, run screen -r, and I can diagnose any problems or change any code that I want to, without restarting the Guile process.

I used similar facilities while developing the software, via Guile's Emacs integration code, known as GDS. If you run Tekuti with --gds, it will spawn a thread to connect to a locally running Emacs. At that point, you can redefine code in the running Tekuti process from within Emacs. It's an incredible reduction in the standard edit-compile-run cycle, effectively removing the "compile" step.

it's all functional programming

Git actively supports the functional programming style: if you have a reference to a tree, that tree will never change out from below you. The only kind of imperative modification that Git supports, besides the interning of new objects into the database, is updating the value of "refs", which conveniently has a compare-and-swap operator, git-update-ref. So not only is git's object database excellent for functional programming, it supports concurrency natively. Indeed, the algorithms that one must use to update a ref are exactly those used in lock-free datastructures. Neat stuff!

So, to summarize: the core of Tekuti is a tail-recursive loop, a là erlang, in which after every iteration we do a goto to the new code of the function. At the beginning of the iteration, we get the value of the "master" ref, and if it's different from the one Tekuti saw last, we do a reindexing operation (on only those parts of the tree that changed). Then we read the data from the mod_lisp socket, dispatch the request, and then -- oh, something I didn't discuss --

All this without harming a single angle-bracket. No <a href="">, no <p>; all of the data is represented as s-expressions (instead of x[ml]-expressions). Hence we have the entire power of the SXML toolkit at hand: pre-post-order, functional tree-fold input parsing, and you never have to think about that stray ampersand in your text, because strings and XML are distinct data types.

the bad, the ugly

Instructions on getting the source code, and importing your old Wordpress blog posts, are available at It's two thousand lines of scheme. The user-level side of things is OK, although it lacks pingback support, and the archives and tag browsing stuff isn't quite finished yet.

On the side of the admin, the one who writes the weblog posts, things are, um, spare. I committed this post directly to the git repository. There's some scaffolding for an admin interface which as yet fails to be awesome. In addition, I'm not sure how well things work when you're starting from an empty repository; probably best to import from wordpress first.

You'll want the latest Guile 1.8, ideally with threading support so you can do things like --repl and --gds. Then you'll want the latest guile-lib, preferably from the development repository.

things I learned

So, as I was hacking this thing, I allowed myself the liberty to investigate the meta-level, practices and code that could help me to hack better. I now use paredit-mode when hacking Scheme code in Emacs, which is a revolution. I now miss it when I hack C. I figured out how to link Emacs and a running Guile application via GDS, which is both crazy futuristic and also crazy 20 years old. (It's pretty much like SLIME for you common lispers.) I learned how to use mod_lisp and Guile's sockets, and lots of ickiness surrounding HTTP: status codes, decoding post data, implementing basic authentication (and base64 decoding!), encoding of URLs, Last-Modified and If-Modified-Since, etc.

Guile's built-in exception handing is a bit ad-hoc, so I learned SRFI-34 and SRFI-35 to try to systematize things a bit more. Mixed results. The same conclusion probably applies to learning the intricacies of `format', including its iteration constructs. PCL good book.

I investigated writing this all in R6RS scheme, but the best-looking compiler, ikarus, is not available for my development machine, which runs PPC. Also, R6's support for dynamic programming still isn't there. I'd like to look at it again at some point, perhaps with an eye towards porting; I think that for me, Tekuti will be the litmus test for R6's validity as a programming language. We'll see.

what's with the name

Ote ku ti kutya mOshiwambo, "tekuti" otashi ti "ote ku ti".

Te estoy contando que en oshiwambo, "tekuti" quiere decir "te estoy contando".

I'm telling you that in Oshiwambo, "tekuti" means "I'm telling you".

32 responses

  1. Andy Wingo says:

    Hm, seems that the post link in the atom feed is busted. If you see any other bugs, please comment. Thanks!

  2. Thomas Kappler says:

    Sounds awesome, both the Scheme and the git part!

    Can you give a rough estimate about how much the code depends on Guile, i.e., how much work it would be to port it to, say, PLT?

    Giving "" as website URL gave me a "Bad URL" error, even though it works fine.

  3. Andy Wingo says:

    Thanks for the feedback!

    Like I say, tekuti's only about 2000 lines of scheme, so there is an upper bound to how much work it would be to port it. Given that PLT has a pretty deep library stack, I wouldn't think that it would be more than about 20 hours of work.

    That said, I think a more interesting port would be to port to some kind of standard scheme, be it r6rs or whatever enhanced r5rs proposals people are bandying about these days.

  4. Jakub Narebski says:

    Do you use git directly, calling git commands, or do you access repository structure? I recall that there was discussion on git mailing list about storing maildirs in git, and what could be changed in repository organization to make it faster and more sane. But unfortunately I don't remember details, and don't know if it would help you or not.

    What makes me wondering is why you use Bazaar as SCM for tekuti, and not Git...

  5. Jakub Narebski says:

    If/when 'tekuti' is out of alpha, could you post announcement to git mailing list, (you don't need to subscribe to post), and perhaps add info about it to GitWiki:


  6. shanawar wht says:

    Any glowing shine plus a glowing skin color is almost everything everyone would like for. Appearances play a vital role today. People need to look presentable normally glamorous. healthwatchwi

  7. shanawar wht says:

    It’s for certain any person would come across unexpected situations, actions, crisis and a number of the life thunder or wind storms that always are out connected with control. These kind of events regularly affects male’s living actually or perhaps monetarily. Well-known fact below, is in which no-one can easily put themselves from these kinds of difficulties, if they’re planning to happen or perhaps actually guard themself against their particular severe outcomes. phoenixazinsurance

  8. shanawar wht says:

    Many people are linked together through social media marketing these nights. Mostly everybody has balances on famous social media sites now-a-days no matter the age group. Where these social media sites enable people coming from different locations, cultures, nationalities and also languages to email each some other, these web sites also become a program for legal activities. newmediacareers

  9. shanawar wht says:

    Each and every summer, the International Language Keep an eye on ranks the most notable Fashion Capitals with the World. Singapore and also Berlin broke in to the Top Five for initially this yr. Singapore will be ranked eighth on earth, coming inside behind simply Hong Kong inside the Asia record. fashiontradefairs

  10. shanawar wht says:

    Stamping companies, one just like the 55 print is in fact very great for many diverse businesses. Printing must print out there advertisements, advertising banners, literature, visiting charge cards, pamphlets, leaflets etc. The purposes which is why printing can be utilized are many. halfpricelawyers

  11. kk says:
  12. ADIL says:

    River map of United statesRiver map of United statesRiver map of United statesRiver map of United statesPrintable calendar 2017Printable calendar 2017Printable calendar 2017
    print usa calendars print usa calendars printable templete calendars printable templete calendars printable templete calendars printable templete calendars tenplete printable calendars tenplete printable calendars tenplete printable calendars tenplete printable calendars tenplete printable calendars free printable calendars free printable calendars free printable calendars free printable calendars

    printable calendars printable calendars printable calendars printable calendars printable calendars printable calendars printable calendars printable calendars printable calendars printable calendars printable calendars printable calendars printable calendars printable calendars

    printable calendars printable calendars printable calendars printable calendars 2017 printable calendars 2017 printable calendars print free calendars printable calendars templeteprintable calendars templete printable calendars templete

    This is an excellent post i seen.I have to thanks to you to share it. It is really what I wanted to see hope in future you will continue for sharing such a excellent post.

  13. asd says:


  14. kerja dari rumah says:

    The information you have posted is very useful. The sites you have referred was good. Thanks for sharing...

    Love what you're doing here guys, keep it up!..

  15. melancong says:

    This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! keep up the good work

  16. rahsia uber review says:

    This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post.

  17. FORT LEE says:

    This is a smart blog. I mean it. You have so much knowledge about this issue, and so much passion. You also know how to make people rally behind it, obviously from the responses.

  18. yuma az upholstery cleaning says:

    There's no doubt i would fully rate it after i read what is the idea about this article. You did a nice job..

  19. Criminal Lawyer says:

    this blog was really great, never seen a great blog like this before. i think im gonna share this to my friends..

  20. kerja dari rumah dengan ilancee says:

    Great tips and very easy to understand. This will definitely be very useful for me when I get a chance to start my blog.

  21. meaning of head jewelry says:

    This is a great post. I like this topic.This site has lots of advantage.I found many interesting things from this site. It helps me in many ways.Thanks for posting this again.

  22. pizza white plains ny says:

    This was among the best posts and episode from your team it let me learn many new things.

  23. dyson v8 absolute usa says:

    Thank you very much for sharing such a useful article. Will definitely saved and revisit your site

  24. instagram business tools says:

    Well, this got me thinking what other workouts are good for those of us who find ourselves on the road or have limited equipment options.

  25. buy instagram likes says:

    As a seller of legal steroids, you can buy Crazy Bulk products, explore stacks and finally get the body you’ve always wanted

  26. says:

    You completed a number of nice points there. I did a search on the issue and found nearly all people will have the same opinion with your blog.

  27. 1000 free followers on instagram says:

    I think this is a standout amongst the most critical data for me. What"s more, i"m happy perusing your article. Be that as it may, ought to comment on some broad things

  28. says:

    Thank you very much for sharing such a useful article. Will definitely saved and revisit your site

  29. says:

    Well, this got me thinking what other workouts are good for those of us who find ourselves on the road or have limited equipment options.

  30. tech startups says:

    This is also a very good post which I really enjoy reading. It is not everyday that I have the possibility to see something like this.

  31. refinance rumah malaysia says:

    My friend mentioned to me your blog, so I thought I’d read it for myself. Very interesting insights, will be back for more!

  32. Toronto dentist says:

    I have been searching to find a comfort or effective procedure to complete this process and I think this is the most suitable way to do it effectively.

Leave a Reply