wingolog

regarding decadence

10 June 2008 6:07 PM (gnome | guadec | decadence | gl | against boredom)

Quand les gens s’aperçoivent qu’ils s’ennuient, ils cessent de s’ennuyer.

Paris 1968

Regarding my last post, Benjamin said to me that "the biggest omission was a solution to the problem". Well, I don't have a solution, not really anyway. But while the topic is fresh, and it seems to have resonated with a few people, I do have some thoughts on the topic.

technology

I seem to recall Christian complaining about the new GDM animations on login, that they feel slow. Well that's because we have a static conception of interface, that things are static by default, and that motion is the abnormal case. This is a mindset that distorts how we make programs (static with sprinkled animation), and one that does not reflect the way that hardware works these days.

Three years on, Jon Smirl's breakdown of the Free graphics stack is still relevant. Go read it! But instead of his conclusion ("make free applications faster through a new X server"), I would draw the line differently: let us make our applications with GL. Then the applications that we make will be new, 100 times a second.

This hertzian rebirth liberates the computer to react to us, and to anticipate our desires.

For example, as the user starts to focus on an actionable surface, that surface might become more prominent at the same time as other, related actions present themselves. To take a most basic example, if you are typing in a word processor, the mouse cursor goes away. (Or can be made to do so.) If you then grab the mouse again, probably you want to click somethingorother; yet the clickable surfaces are present even when typing, unnecessarily, and are equally small once you do grab the mouse.

Or to take a more familiar example, the "expanding icon" behaviour of the Mac OS dock contextually expands the icon that you are focusing on, and anticipates focus of adjacent choices; and it's something that we still do not have, years later.

It is for these reasons that I think that a GL-based canvas like Clutter is an excellent option for GTK+. The library provides nouns and verbs that compose to make a dynamic, pleasing interface; and if you are programming in a fast language, you can implement your own nouns and verbs, which then form part of the GL rebirth/draw loop.

Another thing to think about is that the mouse is simply a way of indicating focus and a limited form of selection. Our verbs are point, click, and drag. New input methods are here, or coming soon: things like multitouch, head tracking, and gestural recognition. These input methods do not map exactly to the mouse. In fact, once you have such capabilities, other, more "normal" parts of standard interfaces start to look restricting. Menu bars are made for mice and framebuffers; the panel starts to look superfluous; etc.

Obviously this strategy has problems when it comes to power consumption; I do not know the solution. I assume the OpenGL ES people have thought about this somehow.

audience

Whether or not you think these ideas are good or bad, you can rest assured that they will not come to pass in GNOME as it is now.

The problem, to my mind, is that we have painted ourselves into a corner of serving a fictional clientele that is not ourselves. As Havoc says:

Is GNOME for UNIX and shell users? Is it for Linus Torvalds? Is it for ourselves? Is it for American college students? 35-year-old corporate office workers with an IT staff? And are we willing to tell whoever it's not for to jump in a lake?

...There is a default audience if a project doesn't find a way to choose one deliberately, and it's what I've called "by and for developers." If there's no way to stay out of that gravity, it's better to embrace it wholeheartedly IMO...

I think we should have a project for hackers. A GNOME skunkworks, if you will; a place in which hackers can create something new for ourselves, in which innovation and even failed experimentation is possible and encouraged. If something developed there is useful to a wider public, as we would hope would be the case, then by all means folks can pull it out, and start to put it somewhere more stable.

Note that "for hackers" does not necessarily entail complexity; it entails only that complexity which is necessary. Physics does not strive to make ugly theories, string theory aside; we as hackers, though oft maligned, neither want such things. Working together, with the best spirit of peer review, we could come out with something with power, and simplicity, and diversity.

Such a project would question everything: what is our interface metaphor? Is it a desktop? Is it a space, like Neuromancer? Is it a set of overlapping planes, like in Minority Report? Is it something else? What is it populated with? How do we compose separate parts into one graphical whole? How do we choose parts of the whole to focus on, to interact with? What is the nature of a part of the whole? How do focus and applications interact? Does the phrase "switching applications" make sense, and if so, how do we do so, or start new applications? Do applications exist in windows? How do multiple users interact with one live system, when we run GNOME on our TV and on our laptop on the arm of the sofa, and using the wiimote as an input device?

interlude

I think we can learn a couple of things from the Pyro experiment. In case you don't recall, Pyro was an attempt to bring web developers out of the confines of the browser window, to let them manipulate the whole "desktop". It was a really neat hack. It seems to have failed, also; I don't hear much about its uptake, a year later. At the risk of poking Alex's old wounds, we should probably wonder why.

If I were to guess, I would say that Pyro failed for cultural reasons. Its target "audience" was web developers, and also those parts of GNOME that would be comfortable changing desktop development into web development. But web developers like to work for an audience of users, and of these there were not many.

But more than that, and reasonable people may disagree, I think that Pyro failed because it wasn't exciting to the developers that we do have, to our culture. I don't think this situation was particularly gratifying to Alex, who (I would imagine, I do not know) found more fulfillment in other parts of his life. Ah well. "One must bear a chaos inside to give birth to a dancing star."

Regardless of your judgment on Pyro, I think that there's still loads of interesting work to do on "the client side", not least experiments with the GPU and self-renewing interfaces. So we should embrace the hackers that we do have, and see where that takes us.

steps

Let's take a number of cases of existing applications, and see what they might look like, coming out of the skunkworks.

The web browser breaks out of its four walls.

Marvin is browsing planet gnome. The web page fills up the entire screen. He finds some link of interest, and selects it. The Planet GNOME page seems to recede in distance as the new page fades in to replace the full screen.

He reads for a while, somehow scrolling the page, then realizes he wanted to go back to some other link. He bangs the mouse against the edge of the screen, or gives a certain hand gesture, and the web page shrinks down from fullscreen to just one page in a series of pages fading off into the distance, representing his browsing history. As he runs his mouse over that series, the pages pop up under the cursor, with contents if available. The effect is similar to that of flipping pages in a phone book. or passing the mouse over the Mac OS dock.

The "window manager" expands its scope, and admits new forms of input.

Marvin middle-clicks on a link to open it in a new "window". The existing train of web pages slides left, off the screen, leaving space for the new page to fade in. Marvin reads for a while, then decides he wants to get some hacking done. He holds up his hand, pushes back the space, pans his applications around until he is on Emacs, and lets his fingers fall into a fist. Emacs comes close to him and fills him and the screen with its radiant light.

Alternately, Marvin could pan via moving the mouse while holding down the shift key, or pressing some appropriate key combination that does not conflict with his carefully-crafted emacs keybindings.

A photo browser fills the screen.

Erinn, having just finished hacking the good hack, wants to enjoy her photos on the TV in the living room. She sits down on the couch, and grabs the wiimote. She presses a button to open a media navigator, showing her all of the media shares on the home LAN, neuromancer-style. She pilots into the living room machine, selects the photos, and begins browsing. The photos are arranged on a timeline rolling into the distance; she focuses in on the set from last year's GUADEC, and begins flipping through them.

OK, so you see where I want to go with this. Try it yourself, perhaps taking "video chat" as an example. How are calls received? How are they made? How does video chat interact with your photo app, with the rest of your applications?

My point is that although we have reached a kind of local equilibrium with the "desktop" metaphor, the existence of the GPU gives us all kinds of possibilities, unique capabilities of the box connected to the monitor. My ideas (and these are not all mine, I stole most of them from folk at work) are those that I see from here, which is not very far. We can do better than this.

conclusion

I recognize that these are words from a marginal player in GNOME, and a Schemer at that. But I think that somehow, a skunkworks is how GNOME will "revision up", whether it comes from the free community, or whether it is dropped on us from some corporate lab.

The concrete step that interested hackers should take is to learn GL, to play around with it. Use the texture_from_pixmap extensions to pull in a WebKit window or two, toss them around. Use Clutter, if that's what you feel like. Build a video chat application (the Telepathy people swear it's possible), build a new window manager (with tight Ruby bindings), make a mockup. Then if your mockup is inspiring, and organic enough to live in, we can start a GNOME skunkworks to play around in.

But above all, make it for you. Pay no attention to mental questions of how your mother would see this, those questions will fix themselves in time. A focus on beauty and simplicity and power cannot fail to make something interesting. Code against boredom!

12 responses

  1. Samuel says:

    Come on, just let me write web applications in pygtk and I have enough of the web on my GNOME desktop. Please.

  2. daniels says:

    While I'm not a designer (but feel compelled to point out the sheer irritation the Office 'ribbon menus' caused: smarter people than I have brought up the notions of consistency and predictability as good things), I will note that the way to save power in OpenGL ES, isn't. 3D engines are very relatively power-hungry beasts: on the desktop, they consume more than most of the rest of the machine, and in embedded/consumer devices, they consume vastly more than most everything else (excluding the backlight) combined.

  3. Tinkilinki says:

    Thanks for inspiring writing! I associate myself with your views. GNOME should learn from Mac OS X - especially neat animations in applications. Look is important, look sells.

  4. Mickey says:

    I don't think that a citation of mai 68 is a good idea for a post about decadence. If there is an obsession in france is the decadence, the french's decadence :)

  5. Alex Graveley says:

    Regarding Clutter as a way forward, my concern is that it will be outmoded by the Web before we can do much that is interesting.

    Once 3D comes to the web, in the next 2-3 years (there today with the right Firefox extension), how long do you think it will take for a Clutter-like JS API to emerge on top of it? I'd bet 6 months tops.

  6. Alex Graveley says:

    Additionally, Pyro didn't fail so much as it's development aborted. Chris Toshok got busy working on Moonlight. I decided to focus on my career until Firefox 3 came out, which enables a faster, rich Pyro experience beyond just experimentation.

  7. Sieb says:

    "But above all, make it for you. Pay no attention to mental questions of how your mother would see this, those questions will fix themselves in time."

    Oh dear. That's how we got into this mess in the first place- usability issues really don't "fix themselves in time".

    If you write an application that's genuinely intended only for personal use, then worrying about your mother is not a problem of course. If you harbour any ambitions of your mother ever using it, and you don't think about her up front, then you'd better be happy for her not to use it for many years to come.

    (If anybody believes this isn't true, can we have Undo in Evolution and Nautilus 2.24, please?)

  8. Zakaria says:

    If GNOME want to have skunkwork project, I think that project must be about ZUI (Zoomable User Interface) http://rchi.raskincenter.org/index.php?title=ZUIs.

    For a bit demo about see humanized version of it
    http://humanized.com/weblog/2007/05/18/die_desktop_die/

  9. Mike Gran says:

    I do a bit of math and science coding, and truth be told, the few times I tried to code GNOME/GTK, I always got hung up at the HIG. For better or worse, most GUI toolkits force one to first consider the layout of the GUI. Even though the 'model' is the fun part, one always starts with an expansive and rigid 'view'. And this view is expected to conform to a heavyweight standard with significant functionality.

    That's why I always preferred web/HTML coding or console coding over an GUI system. In HTML, there is no de jure HIG. If I make a web form that contains just one text entry or one button and spend the rest of the time on the model and the business logic used to create the result, no one will beat me over the head about not having a help menu, a font chooser, or an About box. Web and console application users have been taught to expect HTML GUIs that are severely constrained and to find them unsurprising. GNOME users have been taught to have applications that 'just work' and conform to expectations.

    Coding GNOME is no fun, because, no one will recognize the coolness of your hack until you've worked on the HIG completeness of your ap. I code free software for myself, but, I want the kids to think I'm cool, too. The distance from crazy idea to getting the nod from one's peers is just so much shorter in the world of web than in the world of GTK.

  10. sapphirecat says:

    While in the photo viewer, the phone rings, and Joe answers. His gestural-interface feels the grabbing and tipping, interprets it as "emptying", and deletes the photo. (Drinking would also tend to do this.)

  11. Chris Wallace says:

    Thanks for the great articles and refs here. Jon Smirl saved me wondering further about Brendan Eich's comments ~ "why can't X just die". Certainly interesting to appreciate some of the engineering involved - and reflect on the detailed GL engines/system becoming a component.

    Indeed it would be a shame not to use all that work elegantly! You don't have to be one of the fan-boi on Youtube to feel it's a matter of pride and fun, which as you note, can nicely fuel development (and inspire, yet not a good steering mechanism).

    Moreover in gaining world-domination, we want trailblazers on the frontiers. There are recruits & converts who will judge us by our "cover" as well as our functionality and usability. I know I've envied some swish Apple glitz (and now some ZUI :), even though I can survive WIMPily.

    Maybe the consolation of the latter is feeling more environmentally responsible! ;) Perhaps the software-fallback path could also be voluntarily invoked to allow low-power builds/mode. Presumably not as well as 2D, so there'll be some energy for serious configuration management involving their maintenance.

    Definitely relating to blazing the frontiers, I really want to fire up people to see pyrodesktop's coals are hot stuff. It's time to strike and stoke this fire. By aligning towards becoming the javascript arm/channel of the GOD project (GNOME Online Desktop, ultimately augmented with Weave-ing, plus maybe a me.dium option), and boosted with OpenSocial aspects like gOS has, it could be a contender for the FF3 Extension Award. While it's main function is as a experimental space for innovative interface & application integration, it will be taking full advantage of the JS security, native font rendering, and off-/on-line awareness.

    Also, rather than being at risk of GL 3D kinda-VRML+ZUI? browser future, surely a pyrodesktop implementation on clutter would be the ticket to that and the JS API Alex Graveley foreshadows. The future is beckoning much as Neo gestures in the Matrix ;-) Before the depth-of-field switcher(oo)s, I didn't know what extra interface metaphors there could be for me - or Mum. People can have the freedom to be "decadent" through kinda point releases, or through window-dressing. One important thing is though, GNOME's future necessarily lies with the new kids on the block.

    Meanwhile on another front, let's all do our part in data for developing speech recognition: http://www.voxforge.org and esp for those in North America this month: http://www.voice2type.com/submit_speech/

  12. Ricky Youngblood says:

    I really don't care about this kind of stuff. Sometimes you post too much of it and I stop reading your blog for half a year on accident.

    However, to add to the dialogue, my Gnome running machine was exiled to a friend's closet a couple of months ago so that I could get on with my life.

Leave a Reply