Settling down with ClojureScript

I recently blogged about using ClojureScript for a new internal application at work. After some initial difficulties, I’m finally up to speed with it. I thought it’s about time I shared some more insights.

Since the application we’re building offers a REST API to be consumed by multiple clients, we decided to eat our own dog food from the start and make the web frontend use the same API. So what we’re doing with ClojureScript is a pure client-side web project.

Generating HTML

Since we’re making a pure client-side application, we needed some kind of client-side templating. I didn’t feel like bothering with HTML after getting accustomed to Hiccup in Compojure, so we used the dom-helpers namespace from the TwitterBuzz ClojureScript demo. Now we can generate markup like this:

(dom/build [:div#content
            [:h1 title]
            [:div#messages
             [:div#message-input
              [:div
               [:label "Your name:"]
               [:input#author-name-input {:type "text"}]]
              [:textarea#message-textarea]
              [:button#send-button "Send message"]]
             [:div#message-list]]]))

The dom-helpers are basically a nice abstraction for Google Closure’s DOM manipulation utilities. I’m sure that ClojureScript will include something like that in the future.

Avoiding Google Closure

Knowing that amazing web applications like Google Docs use it, I really wanted to give Google Closure, which is included in ClojureScript, a chance. The utility functions are actually quite handy, but the lack of documentation makes using most parts, e.g. the UI widgets, a royal pain. Add to that the fact that it feels more like using a cumbersome Java framework than a JavaScript library, I decided to avoid it where possible. Google Closure is technically sophisticated, but lacks elegance and convenience.

Building with Leiningen

When we started to use ClojureScript, we built it from source and invoked the ClojureScript compiler from a Makefile. I can’t begin to tell you how weird this felt. Since we’re using Leiningen to build the backend, we decided to use it for the frontend as well, so I wrote a custom Leiningen plugin that performs the tasks we need, e.g. calling the ClojureScript compiler and packing the output into a WAR file. We’re not planning to release this plugin, it’s pretty hard wired towards our needs and not really thought through.

Instead, I started to contribute to lein-clojurescript, first making it work with a stable release of Leiningen. The author likes my ideas about offering multiple modes for development and production, and making the respective options configurable in the project.clj, so that’s the plugin you want to use.

Promising stuff

REPL in the browser

When ClojureScript was first announced a while ago, the REPL used Rhino for JavaScript evaluation and DOM manipulation was not possible. The ClojureScript developers have addressed that problem and made it possible to use the Browser as evaluation environment. It looks promising, but I didn’t really find the time to play around with it yet.

Compiling automatically

Compiling without optimizations is pretty fast, but manually invoking the compiler before refreshing the browser still feels wrong when doing web development. cljs-watch attempts to solve that problem by watching your ClojureScript sources, recompiling as soon as anything changes, e.g. you save a file. I trie it, but couldn’t make it work right away, so I postponed that. I guess something similar can be build into lein-clojurescript.

Problems

As I told you at the beginning of this post, I’m pretty happy with ClojureScript by now. We had to put some extra energy into setting up our development environment and build system, but now we’re up to speed. A few problems, however, remain.

No unit testing

I haven’t yet invested the time to figure out how to execute unit tests for ClojureScript code, it will probably be easier now that we build with Leiningen. This is something I’ll have to approach soon, working without tests feels really bad.

Still no release

I really don’t know why the Clojure/core folks announced ClojureScript without having a release candidate of ClojureScript ready, or even a tag. That hasn’t changed. At first, we forked the ClojureScript project on GitHub and created a tag to ensure reproducible builds. Now that we use lein-clojurescript, we use the ClojureScript version from Clojars, but that doesn’t feel much better – it’s still an arbitrary snapshot.

Conclusion

Despite the problems, ClojureScript made a very good first and second impression. It proved very stable (I didn’t come across a single compiler weirdness, of which I’ve seen plenty in GWT), which made me take languages that compile to JavaScript seriously again. Instead of whining about programmer inconvenience and immature tooling, I’ve decided to contribute, starting with lein-clojurescript. Once that’s powerful enough to replace our custom plugin, I’ll see if I can help out with ClojureScript itself, it really needs an initial release.

Back to game development

After busying myself with other things for a couple of months, I feel a strong urge to return to game development. I haven’t blogged about this, but my wife and I work on a few games in our spare time. The one that’s most likely to ship this year is a puzzle game for Android, and I’m determined to make that happen.

I’ve been a little less motivated to work on that particular game because, well, it’s a puzzle game. The code is pretty much final, but most of the game content is missing; there are only a few boring levels with ugly programmer art at this point. Designing levels for a puzzle game on pen and paper doesn’t really work for me, so I began to write a level editor. A major feature will be the possibility to create random levels, so that I will hopefully be able to generate a level, attempt to solve it and rate its difficulty.

I really enjoy Android development, but one of the things that keeps annoying me is that you’re stuck with Java. I love Java as a platform, but I never felt much love for the Java language, far too limited for my taste. Fortunately, I can use Clojure for the level editor. It is possible to write Android apps in Clojure and other JVM languages, but there is apparently a performance penalty involved, and I can’t have that. I guess I’ll use C++ for our next Android game, despite the fact that I won’t be able to use the pretty neat Android framework.

Anyway, stay tuned.

Adventures with ClojureScript

Since Clojure is one of my favourite programming languages, I took a chance and used the brand new ClojureScript on a small project.

I was sceptical when ClojureScript first came out: Having used Google Web Toolkit for years has made me a bit nervous about compiling other languages to JavaScript. Such a layer of indirection can seriously complicate things, and make development feel disturbingly like magic.

On the other hand, I thought that Clojure, being a Lisp dialect, would translate much more nicely into JavaScript – which is very similar to Lisp – than Java. So I gave it a go.

Working with ClojureScript

ClojureScript is very young, so there is hardly any tool support at this point. Neither Maven nor Leiningen can build ClojureScript projects, so I wrote a good old Makefile.

Compiling felt weird, because the usual JavaScript workflow is to just refresh the page. With ClojureScript, I had to invoke make first, which is fortunately pretty fast, at least if you don’t have much code.

I typically spend a lot of time in the browser’s JavaScript shell, so I expected to use the ClojureScript REPL extensively. It compiles ClojureScript into JavaScript on-the-fly and interpretes that with Rhino, a JavaScript engine for the JVM. That doesn’t only sound weird, it’s also a bit unstable. I had mysterious exceptions all the time, and at some point it even crashed my machine. I didn’t investigate these problems further, because I really had to get stuff done. What I really want – DOM manipulation – is apparently not possible in the ClojureScript REPL anyway, and the normal Clojure REPL was sufficient for everything else.

As for debugging in the browser, I had to break that habit as well: The JavaScript generated by ClojureScript was hardly readable, even with optimisations turned off and pretty printing.

These are quite a few limitations, and as expected, they did slow me down. Then again, being able to use Clojure instead of JavaScript made up for that a bit (I really like JavaScript, but I feel that I’m more productive with Clojure).

Functions and properties

ClojureScript worked like I expected it to most of the time, but there was one particular weirdness that made me blink: If you call a JavaScript function, you will get the function’s source code as a return value, not the function’s return value. It does make sense from a technical point of view, but I’m pretty sure it will make every single aspiring ClojureScript programmer curse at some point.

This is how you would normally execute a function of an object:

(.someFunction someObject)

And this is how you execute a function and get it’s return value in ClojureScript:

(. someObject (someFunction))

I couldn’t find any documentation on working with properties in ClojureScript, so I’ll share what I figured out:

Get a property:

(.someProperty someObject)

Set a property:

(set! (.someProperty someObject) newValue)

Working with Google Closure

ClojureScript comes with the Google Closure Library, a large collection of JavaScript utilities and widgets, created by Google and used on amazing web applications like GMail and Google+. I had been curious about Closure, so I decided to use it for this project.

My first impression is that it is less elegant and concise than jQuery and other JavaScript libraries I’ve used. I had to write an annoying amount of boiler plate code to create a simple UI programatically. In Clojure, where things are usually very elegant and concise, this felt even weirder – not too different from using Swing in Clojure on the JVM.

To make things worse, the documentation isn’t up to Google standards, but I guess that would change as the library becomes more popular. If it does.

Conclusion

ClojureScript is interesting, but apparently not intended for production use yet – there hasn’t even been a release. It needs to mature, providing a stable REPL and a nice functional layer above the Google Closure Library (or some other library, or something written in ClojureScript).

With languages that compile to JavaScript becoming more common and with the prospect of debugging support, ClojureScript is definitely something I will keep an eye on.

Contributing to Chromium

I always wanted to work on a web browser. Probably because I’m both a front-end focused web developer and a low-level loving C++ programmer, and web browsers are among the few types of software that require such a combination.

Moreover, I recently felt the urge to lay hands on large, established code bases. I believe that working with existing code will take me further in my quest of becoming a better programmer than starting yet another small project. So I decided to contribute to Chromium, the open source project behind Google Chrome.

Why Chromium?

Well, mostly because it’s my favourite browser, and because it is in my opinion the one that pushes the web the most right now. I also think that a few more non-Googlers working on it would be good for the project. Not that I distrust Google on this, but I believe that an open source project, especially such an important one, needs contributors from all over the industry. A too dominant player can, even with good intentions, harm a project in the long run: Remember what has happened after Oracle bought Sun.

Another reason why I picked Chromium was that I thought it was a small, modern, lightweight piece of software, unlike Firefox, whose code base appears to be an ancient monster. Guess what? Chromium’s code base is a monster too, at least in terms of size. Including dependencies, we’re talking about several gigabyte of source code here.

The code

Nonetheless, the code quality looks good to me. Google’s (partially weird) code style is followed consistently and the code I’ve seen so far looked clean and polished. I don’t understand the complete architecture yet, but what I’ve seen and read seems promising. I have only two problems with the code:

Monolithic build

I found it difficult to work on just one area, in my case the UI. You have to statically link all of Chromium (which takes about 10 minutes on my, admittedly slow, netbook) in order to see even a small change. I would prefer it if there was some way to fiddle with design and layout with a faster turnaround. Some people on the #chromium IRC channel seem to be unhappy with this as well.

Redundant UI code

Considerable parts of the UI code are effectively duplicated for each supported platform: Windows, Mac and Linux. The Windows code is written with WinAPI and WTL, the Mac code with Cocoa and Objective-C and the Linux code with GTK. I’m sure they have their reasons, one of which is very likely performance, but I still think there could be more code reuse among the platforms. I have seen essentially the same code on different platforms, written in a slightly different way, probably by a different person. As is the problem with code duplication like this, some bugs get fixed only in some versions of the code. The status bubble (the thing that appears if you move the mouse cursor over a link) is a good example of this: It slides perfectly on Windows, flickers a bit on Mac and disappears completely on Linux.

Contributing

In order to find something to contribute, I had a look at the Chromium issues marked as GoodFirstBug and picked one that concerned the sliding of the status bubble (see above) on Linux. After identifying the problem and fooling around with the code, I noticed that this was a non-trivial GTK problem, the kind of thing I’d like to avoid with my first patch, not having used GTK before. So I picked an issue with the downloads tab which was all in all about one hour of work, naturally not counting compilation time. To my delight, the download tab’s UI was written in HTML/JS, which made the UI work a no-brainer.

It was fun to work with the code, but the compile times were pretty annoying. My desktop box at home is quite fast, but I rarely use it ever since I became a dad. So I worked on the train, which means on my netbook. I usually don’t have any problems programming on it – even Eclipse is usable – but Chromium development was tough. A trivial change meant that I would have to wait for 10-15 minutes. A git pull meant that I had to compile over night. It was mind-numbing.

Was that worth it? Yeah. Sure, my issue wasn’t a big deal, but now I can tackle more difficult ones, maybe even become a committer if I stay motivated. That would allow me to shape the future of the web, which is a considerable level-up for a web developer. Not sure if I can keep my motivation when having to work on that slow netbook though. Maybe I can use some of the 20% time at work for this, I guess Chromium development would be much more enjoyable on a powerful iMac.

Update

My company being awesome, I was indeed able to hack on Chrome during 20% time. I implemented Kiosk mode for Mac. As expected, it was more enjoyable with the iMac. I have a lot of 20% projects going on, but I’m definitely planning to contribute more in the future.

Eclipse Color Theme

About half a year ago, I began to work on what has become my most popular open source project up until now: Eclipse Color Theme, a plugin that makes it possible to use colour themes in Eclipse. I thought this was a good time to talk a bit about the history and future of the project.

The black on white ages

It all started with me being fed up with Eclipse not supporting colour themes in any reasonable way. Since Eclipse was mandatory at my old job, I was forced to stare at it all day. I don’t like to stare at black on white text all day, so I had to find a way to use colour themes.

Before I continue, you need to know that Eclipse preferences are a mess. A complete and utter mess. Every plugin can store arbitrary key value pairs of data, and there is no way to export or import preferences selectively, it’s all or nothing.

There are no central colour settings, so every plugin stores their own, in the format of their choice. In practice, this means that even if you actually change all the colours for the Java editor manually, you will have to do it again for the JavaScript editor. And the XML editor, and the JavaScript editor, and whatever other editor you want to use. This is highly inconvenient.

And because Eclipse preferences are such a mess, all you can do to share a colour theme you created is to export your preferences and have someone else import them. And since your preferences contain all your Eclipse settings, it will completely mess up the other person’s settings. Bah.

So I guess it’s safe to say that using colour themes in Eclipse without going insane was impossible. The only solution I saw was to create a plugin that would take care of changing each editor’s preferences according to a standarised colour theme format, without messing any other settings up.

I created a prototype that supported just the Java editor and a single hard coded colour theme to see if this would work. It did, so I added another colour theme and published version 0.1 of Eclipse Color Theme on the Eclipse Marketplace.

The colour themes revolution

I never thought that many people would be interested in having colour themes for Eclipse. Probably just a small bunch of geeks like me, coming from Vim and Emacs. Searching the Internet for Eclipse colour themes revealed just a handful of people sharing their exported preferences or asking about theme support, but it were really few. There was even an ancient ticket about colour theme support in Eclipse’s bug tracker, for which about 4 people voted during all that time. So I thought I’d create the plugin more or less for myself.

Turns out I was wrong, I received many emails of appreciation, even a couple of donations. So I was quite motivated to improve the plugin and add support for more editors, at first XML, HTML, JavaScript, CSS and C++. At that time, my own needs where met, but I kept adding things that were requested via email or GitHub.

At some point, Roger Dudler contacted me and told me that he was planning to create a website where people could create colour themes for Eclipse, and whether I wanted to join forces. I did, so we both worked on plugin and website together (well, I did only a few things on the website so far), and eclipsecolorthemes.org was born, allowing users to create their own colour themes with a WYSIWYG editor.

Did I tell you how wrong I was about nobody being interested in colour themes for Eclipse? I was. Within a few weeks, Eclipse Color Themes climbed to the top 4 of Eclipse Marketplace with thousands of installations, and hundreds of themes on eclipsecolorthemes.org.

This huge demand created a constant flood of emails, asking us to support new editors or reporting problems. We decided to make the plugin more modular and easier extendable, using Eclipse plugin features like extension points. Roger, who had some experience with Eclipse RCP development, did that conversion mostly by himself.

Future

Maintaining the plugin is not too much work, so Roger and I are able to concentrate on other projects. Our own needs are long met, especially since I don’t use Eclipse on a daily basis anymore. (At my current job, everyone uses IntelliJ IDEA – whose theme support is only slightly better than Eclipse’s – and I’ve switched back to Emacs for C++ and JavaScript development.) Nonetheless, my ambition is to release Eclipse Color Theme 1.0, preferably this year, and there are still a few issues to be solved and improvements to be made.

One interesting topic for the future is Eclipse 4, which will introduce a new, themable UI. If I understood it correctly, plugins can either use the old or the new UI technology, which means it will probably take a while until all important plugins make use of it. Maybe it will make sense to support Ecilpse 4, we’ll wait and see.

Since Eclipse Color Theme is (in theory) modular, i.e. support for new editors can be added by other plugins, it would be nice to split it into multiple plugins, e.g. one for each Eclipse plugin package (JDT, WTP, CDT, PDT, …). Ideally, the developers of the package would also maintain the colour mappings, but I guess that’s wishful thinking.

If there is one thing I learned in this project, it’s how remarkably motivating it is to have lots of users and lots of feedback. Thanks to all of you writing emails, creating themes and donating money, you’re a great source of motivation.

Test-driven development in the shipyard

As regular readers might recall, I began to use test-driven development for my private projects a couple of months ago. At my new job, I finally get the chance to use it on larger codebases, and there is one particularly useful technique I learned so far: In the absence of a better name, I’ll call it the shipyard.

Photograph: Tup Wanders

 

In TDD, before you write any code, you write a test case for it. The test doesn’t have to compile, you can go ahead and build your dream world of how classes, methods or functions should interact with each other in order to achieve your goal. In the second step, you create stub implementations to make the test compile. The nice thing about this is that you get to design an API that cleanly communicates its purpose. However, you still have to think upfront about where to put the code, and who should get access to it. If you’re, like me, trying to keep the scope of everything reasonably small, you’ll have to put some serious thought into this. So you’re stuck where it hurts most; writing your first test case.

This is where the shipyard comes into play: Instead of putting the code you are about to write in its final place, you deliberately keep all of it in the same file as your test case. That way you don’t have to think about whether it will become a utility or part of the code that needs it, and you don’t have to jump around in your codebase all the time during development. As soon as the code does everything you wanted it to, it can become part of the bigger picture, just like a completed ship leaves the shipyard to fulfil its purpose.

Photograph: Official U.S. Navy Imagery

 

If you put the code into place upfront, not only do you have to put in some extra effort, you will also have to move it somewhere else if your initial choice was a poor one. Another benefit is that in the shipyard you can commit your progress into version control without exposing not yet finished code to the rest of the application.

However, I see one potential problem of this approach: You could write nicely tested code that does exactly what you want, but when you are about to integrate it, you notice that it isn’t really needed, or not in that form. That hasn’t happened to me so far though.

On a new road

That’s it, today was my last day at update, I’ll be working for Viaboxx starting next week.

I’ll really miss my coworkers and, considering that I’ve been here for four years, switching jobs is making me a bit nervous and a bit sad.

On the other hand, I’m really looking forward to my new job. From what I’ve seen, the development team is agile, highly effective, open-minded and each developer managed to impress me. They have several interesting projects, including parcel machines, mobile applications and web applications. Furthermore, they have 20 percent time, an awesome office, ergonomic chairs and iMacs with ridiculously large screens.

Consider me excited.

Fukushima

As you probably know, there have been disturbing incidents at several reactors in Fukushima, Japan lately. I’m not sure how the mass media in your country handle the issue, but those here in Germany are certainly going completely insane over it, to an extent that even Germans start to think they might be in danger.

Don’t get me wrong, this is a terrible incident and I don’t support nuclear energy (mostly because storing nuclear waste worries me). Nonetheless, as educated human beings, we should approach such events factual, not emotional.

So what is radiation? I’m no physicist, but I know that radiation is not some rare, deadly poison – it’s everywhere. Sure, in very large doses, it can damage your health, so can pretty much anything on this planet. As Paracelsus put it:

All things are poison, and nothing is without poison; only the dose permits something not to be poisonous.

So it’s the dose we should talk about, not the fact that there is radiation near Fukushima, because radiation is everywhere. XKCD posted a very nice chart comparing radiation doses from various sources.

So how much radiation is there around Fukushima? Greenpeace published their monitoring results. The problem with data is that we have to find some we can trust. I consider Greenpeace a sufficiently trustworthy source on this matter.

Now we can get ourselves an idea of how bad the situation in Fukushima is. When looking at the Greenpeace data right now, I see some dark red bubbles north west of Fukushima. Since that colour indicates the highest readings, we can infer that 11 µSv/h is the highest radiation dose around Fukushima. So that’s 264 µSv/day and 96.36 mSv/year. According to the XKCD chart, this is twice the allowed dose for US radiation workers, which is 50 mSv/year.

That’s certainly not good. But how bad is it? Another look at the XKCD chart reveals that the lowest yearly dose clearly linked to increased cancer risk is 100 mSv/year. That’s awefully close, so these people might in fact suffer from an increased cancer risk if exposed to this amount of radiation for a whole year. And how big would that cancer risk actually be? Probably lower than that caused by smoking, according to Nature News.

That was the worst region in the area, how about those south of Fukushima, further away (about 1/3 on the way to Tokyo)? The readings say 0.4 µSv/h, which is 6.9 µSv/day and 3.5 mSv/year. I don’t think that’s a dose to worry about.

I’m no radiation expert, so both my data and my analysis could be wrong. If you’re sceptical, do some research on your own and see what you can find out. And, as Douglas Adams wisely put it in (and, in fact, on) The Hitchhiker’s Guide to the Galaxy:

Don’t panic.

Algorithms

It’s been a while since I last posted to this blog, mostly because I’ve been incredibly busy being a dad, moving to a new appartment and a new job (I’ll blog about that one).

Nevertheless, I recently found an excellent excuse to finally work on my basics, so I got myself a copy of Introduction to Algorithms (not the Knuth, but still extensive) and began to deepen my knowledge. I’m enjoying it so far, and there’s lots to learn since the topic wasn’t covered in such detail at my university.

Nearly everybody I told about my endeavour said I’ll never need that knowledge. I respectfully disagree. Sure, I probably won’t need to implement sort or matrix multiplication algorithms in my day job, unless that day job involves low level systems programming. But algorithms are everywhere, and studying them doesn’t mean to learn all the existing ones by heart, but mostly to learn how to design efficient algorithms for any problem. The most useful skill is in my opinion to transform real-world problems into problems for which a proven efficient solution already exists, graph problems for instance are everywhere, visible only to the trained eye.

Since all mathematical proofs and no code make Felix a dull boy, I decided to implement every single algorithm explained in the book on algorithms.ubercode.de. I’m also using this as an opportunity to make my first pure HTML5 site and to try Node.js.

The one thing I’m not happy with is that I had to use in-place algorithms in order to visualise the sort process. For instance, my implementation of merge sort originally looked like this:

function merge(array1, array2) {
    var array = [];
    // Merge array1 and array2 into array
    return array;
}
 
function sort(array) {
    if (array.length == 1)
        return array;
 
    var middle = Math.floor(array.length / 2);
    return merge(sort(array.slice(0, middle),
                 sort(array.slice(middle));
}

I find this a lot more readable than what I have now. But since this is a recursive implementation, I wasn’t able to regularly send the whole array from the worker to the main script in order to display the process. The current algorithm modifies the array directly, not returning anything, which clutters the function calls with various indizes and bloats the code. On the other hand, this is very close to the pseudo code in the book, and the performance is likely better. Still, let me know if you can think of how to update the progress with a functional implementation.