Gosu’s Inconceivable non-ClassLoader (Take 1)

Two feature requests stand out on Gosu’s short list for our next milestone release. Primarily they stand out because the demand for them overshadows all other requests combined. And it happens that both features are blocked on what appears to be a virtual impossibility. To give you a hint as to their nature, if you’ve considered using Gosu in any of your current Java projects and chose not to, the absence of at least one of these features is probably why you passed over Gosu. If you’re thinking, “Hard to use Gosu from Java” or “No class files”, you’ve nailed it — as if you’re not thinking that.

Yep, Gosu currently does not produce Java class files on disk. There’s no command line compiler, instead the runtime dynamically compiles source to bytecode and defines Java classes directly from memory. While there are benefits in doing it this way, there are significant drawbacks to consider. Performance is an obvious problem. The extra time it takes to compile Gosu may impact startup time and introduce lag in initial usage of an application’s features. Web applications can mitigate the issue by “warming up” the type system where critical parts are preloaded before users log in. Unfortunately, this strategy won’t work for other kinds of applications like scripts and command-line tools where startup time can be crucial.

Closely related to the problem of not having class files is basic Java interoperability. Gosu interoperates seamlessly with Java… if Java is being used inside Gosu. Turn that around, however, and we have a lot of room for improvement; specifically Gosu classes aren’t directly usable inside Java. I’ll go into detail about this in a bit, but basically the core problem deals with the fact that Java’s class loader has no knowledge of Gosu’s class loader. This is the nature of class loaders in general where a Java application loader is oblivious of custom loaders created after the application initializes. Consequently we have no way of instructing a Java app loader to delegate to our loader. As a result calling or referencing Gosu from Java involves low-road solutions such as reflection or extracting intermediary Java interfaces from Gosu classes. Not quite the seamless experience we’re looking for, hence the demand for better Java-to-Gosu interoperability. And it all stems from having to define a special class loader.

To illustrate the problem consider the following two classes:

package com.abc; 
public class JavaClass {
  private GosuClass obj;

package com.abc 
class GosuClass {
  var obj: JavaClass

    | System      |       Loads system classes
    |             +------>
    | ClassLoader |       eg. java/lang/String.class
Ask parent |
    | Application |       Loads application classes
    |             +------>
    | ClassLoader |       eg. com/abc/JavaClass.class
Ask parent |
    | Gosu        |       Loads Gosu classes
    |             +------>
    | ClassLoader |       eg. com/abc/GosuClass.gs

(For the purposes of this example let’s take it on faith we can compile JavaClass; we’ll touch on that later. Also let’s assume Gosu is initialized, its class loader locked and loaded.)

Ok. Regardless of what kind of application we run (command line, web, etc.) the host environment loads our compiled Java classes for us from the class path. In our generalized example the Application ClassLoader loads JavaClass, but JavaClass fails to verify because com.abc.GosuClass can’t be found. This is because the application loader doesn’t delegate to our Gosu ClassLoader, it delegates upward to its parent – it has no idea the Gosu loader even exists. Conversely, if we explicitly load com.abc.GosuClass using the Gosu ClassLoader, the JVM has no trouble verifying the reference to JavaClass because the Gosu loader involves its parent, the Application loader.

Now you might think that if we go ahead and write class files to disk, there’d be no need for a special loader because the normal Java loader could load our class files — two birds, one stone. And you’d be right if all of Gosu’s classes could be statically compiled to disk. Ah, but there’s the rub. First, and I’ll spare you the particulars, there are a few lingering implementation details with Gosu’s runtime that on occasion rely on compiling bits of code dynamically; these will go away eventually, but we’re stuck with them for now.

More critically, Gosu’s parser and compiler are necessarily incorporated into the runtime. A few corner cases in the language require these services e.g., eval(), but mostly the open type system drives this design. Types in Gosu aren’t required to resolve as Java classes; instead Gosu provides abstractions for type loaders and type information. At runtime Gosu classes compile to conventional Java classes, yet other custom types that are not directly source code oriented e.g., Guidewire’s web framework, aren’t represented this way. Instead such types provide runtime type information for method, constructor, and property invokers. For instance, a web page method typically wraps a user-defined chunk of Gosu. Instead of requiring the web page framework to manage a bytecode oriented type system for all the snippets of Gosu associated with web pages, Gosu allows the snippets to execute loosely at runtime as programs or fragments. Thus a method invoker can simply instruct Gosu’s runtime to execute its anonymous chunk of Gosu, directly from source. But that chunk of Gosu will ultimately compile down to an anonymous Java class which is not discoverable from the Java application’s primary class loader.

The problem isn’t quite as knotty as it may seem. Because these particular chunks of Gosu are anonymous — they’re just nameless scripts — there’s no opportunity for Java classes to reference them by name. Basically this means the Java application loader should never be required to load any of these classes. Therefore we can get away with defining our own special class loader for these, which is what Gosu has done all along. There’s a deeper problem, however. Our special class loader parses and compiles these classes, which involves locking in our type system. Java also has locks for its type system; have a look at ClassLoader, note the synchronized methods especially on loadClass():

protected synchronized Class<?> loadClass(String name, boolean resolve)

Suffice it to say we flirt with deadlock every time our special loader does its job. We lessen the likelihood of this deadlock by ensuring we acquire both our lock and our class loader’s monitor atomically, if not we release both, wait a bit, and try again:

while( !TypeSystem.tryLock() ) {
  wait(100); // release this class loader's monitor obtained from the enclosing synchronized method
try {
finally {

Mission accomplished! Well… we’re not quite out of the woods here. What if our loader compiles a class that references another class from our parent loader, the application loader? Its monitor must be acquired too. We’re back to square one. It turns out deadlock arising from this particular scenario is quite rare, but it’s there, waiting. The only way to avoid it is to take our special loader out of the picture and somehow have the application loader compile and load our classes. Inconceivable.

But wait, there’s more. The JVM considers classes, abc.A and abc.B to be in different packages if they’re loaded in separate loaders. Thus even though A and B are declared in the same logical package, if A is a Java class and B is a Gosu class, because our special loader loads B, B will not have access to A’s package-protected (“internal” in Gosu speak) members. Likewise, A will not have access to B’s internal members. We currently overcome this limitation by generating bytecode for internal method calls and so forth as reflective calls. It works, but it’s an ugly hack with performance ramifications we’d rather avoid. Again the only way to get this right is for the Java class loader to do our bidding. Inconceivable!

Lastly, and most importantly, although we understand and appreciate the benefits of deploying class files, part of the appeal of Gosu is that it doesn’t require them. You can simply write code, say in your favorite text editor, and run it, no IDE or build step necessary, an aspect of dynamic languages our static language currently enjoys — some might say the best of both worlds. What’s more, legacy Gosu libraries don’t have class files. We’d like to support them as they are. So we’re back to square one yet again. Are we sure Java’s application loader can’t be made to somehow load our Gosu classes? “As I told you, it would be absolutely, totally, and in all other ways inconceivable.” (Vizzini to Inigo Montoya)

It would seem so. A class loader can only load the classes it’s designed to load, right? Some of you might be thinking of an unforgivable hack involving AspectJ or some such where we rewrite a portion of the class loader’s bytecode, tailoring it to load our classes. I like the energy, but let’s not; too many different class loaders out there to screw up and they’re all moving targets, it just won’t hold. We couldn’t take that approach anyway because Oracle’s Binary Code License Agreement prohibits the modification of system classes such as sun.misc.Launcher$AppClassLoader which we desperately need to control. Here’s another idea. A lot of application class loaders derive from URLClassLoader, even Java’s standard AppClassLoader does this. We could reflectively call addURL() and modify the class path dynamically. But what URL would we add? The whole point here is to get the class loader to resolve Gosu class names from source, not class files. The URLs we normally add are file system directories and jar files. What we need is a URL for… what?

A better question to begin with is, what does URLClassLoader do with the URLs as it attempts to resolve a class name. For instance, let’s say my class path is comprised of a single directory:

-classpath /my/application/classes

The application loader will have the corresponding URL:


URLClassLoader finds classes in a given URL using the protocol handler associated with the URL’s protocol. Java provides stock protocol handlers for standard protocols such as file, http, ftp, etc. A protocol handler’s primary responsibility is to establish a connection to a given URL in the form of a java.net.URLConnection. Most protocol handlers extend java.net.URLStreamHandler and implement openConnection( URL ). So, essentially, URLClassLoader delegates responsibility of finding a class or resource on its class path to the openConnection() methods corresponding with the protocol handlers of the URLs on its path. Given our single directory example, let’s say URLClassLoader handles a call to findClass() with argument “com.abc.Foo”, let’s assume our directory contains this class. First, the loader transforms the class name to a relative file name: “com/abc/Foo.class”. Then it creates a new URL for the class by appending the relative name of the class file to the directory URL:


Then it calls URL.openConnection() to establish a connection to the class file on disk. Next, the URL finds the protocol handler associated with the “file” protocol and delegates to it. From there the resulting URLConnection provides a stream to the contents of the file and the rest is history. I’ve skipped over a lot of implementation details but that’s the gist.

Interesting. The nature of protocol handlers is to find content wherever it exists. In the case of the file protocol it looks for files on disk. Likewise, the http handler finds resources at an address on the web. In theory we could define a handler for finding gosu classes in a Java application. How would that work? Let’s say we define our protocol as “gosuclass”. Fine.  Now we need to define our subclass of java.net.URLStreamHandler. But wait. Looking over the rules for adding a protocol handler it’s not so simple. Check out the curious process for adding a new handler:

http://docs.oracle.com/javase/1.4.2/docs/api/java/net/URL.html#URL(java.lang.String, java.lang.String, int, java.lang.String)

There are three options. The first option suggests we define our own URLStreamHandlerFactory for creating protocol handlers. The intention, I think, is to centralize control of protocol handler creation so we have a consistent experience with… protocol handling. Fair enough if you’re after controlling all of protocol handler creation, which we aren’t. Adding insult to injury the way we go about adding our factory pretty much rules out its use. From the documentation for URL. setURLStreamHandlerFactory():

“This method can be called at most once in a given Java Virtual Machine.”

Well, I’m pretty sure most web servers call this method way before we’ll get a crack at it. Which is what we want because we need to get a handle to the server’s factory so our poor factory can delegate to it for finding the web server’s other handlers. Since it can be called only once, we’re out of luck. Strike one!

Next on the list describes a convoluted convention where the system property for ” java.protocol.handler.pkgs” provides a list of packages containing handlers. The handlers must be named as follows:


Where <package> is one of the packages in the list and <protocol> is the name of the protocol and then, of course, Handler is the name of your URLStreamHandler subclass. We do all of this and depending on the execution environment (web server, rich client, command line) we get mixed results. The worst of it, on the web server it flat just doesn’t work, ClassNotFoundExceptions abound. What’s the deal? It turns out that the URL class tries to load protocol handlers using Class.forName(), which uses the caller’s class loader. Well this means URL’s class loader since the call originates from that class. On a typical web server Java system classes such as URL are loaded in a different loader than our application classes, therefore it never finds our Handler class. Strike two!

Our third and final option is a dud. Basically the same as the second one except the “default system package” is consulted for the Handler. Strike three! Yeerrr Out…

…wait a sec…

We’re not asking for much, we’d just like to add a bleeping protocol handler.  Where and how are these handlers managed inside Java? I mean how complicated can this really be?  Following the debugger it’s all too easy. The URL class maintains a static Hashtable (how retro) mapping protocol name to handler. That’s it. Couldn’t they just have… I’ll spare you my emotional diatribe regarding pragmatism in software design, or the disregard of it in the Java libraries. Anyway, all we need to do here is resort to a bit of reflection hackery, feel the requisite amount of shame, and move on. For your viewing pleasure:

private static void addOurHandler( Handler handler ) throws Exception {
  Field field = URL.class.getDeclaredField( "handlers" );
  field.setAccessible( true );
  Method put = Hashtable.class.getMethod( "put", Object.class, Object.class );
  put.invoke( field.get( null ), handler.getProtocol(), handler );

Now that we’ve force fed our protocol handler to Java we can confidently add our gosuclass URL to the class loader’s path. This part is easy albeit not entirely guilt-free as it involves another reflective call to protected method, addURL():

private static void addOurUrl() {
  URLClassLoader urlLoader = findUrlLoader();
  if( !(Arrays.asList( urlLoader.getURLs() ).contains( url )) ) {
    Method addURL = URLClassLoader.class.getDeclaredMethod( "addURL", URL.class );
    addURL.setAccessible( true );
    addURL.invoke( urlLoader, url );

The protocol handler itself is pretty simple. It’s only task is to produce a GosuClassURLConnection, which does all the work :

protected URLConnection openConnection( URL url ) throws IOException {
  GosuClassesUrlConnection connection = new GosuClassesUrlConnection( url );
  return connection.isValid() ? connection : null;

GosuClassUrlConnection “connects” with the URL for a Gosu class. Connecting doesn’t involve anything more than resolving the name of the class, which it accomplishes by a call to Gosu’s TypeSystem.getByFullName( className ). If the type exists and is an instance of a Gosu class, the connection is successful. Actual compilation can wait until the class loader asks for the content of the class, which it obtains via a call to our URL connection’s getInputStream() method. This is where we dynamically compile the class and return an input stream for the resulting bytecode and in the process complete the “inconceivable” task of making the application loader do our work for us. No more special loader. Gosu classes, both as class files and source files, are now for all intents and purposes regular Java classes to the JVM; they are usable directly from Java code as Java code.  Hallelujah!

As an added bonus our compiler generates Gosu classes with a static block to initialize Gosu. Initialization includes the critical steps to inject our protocol handler and add the gosuclass URL to the app class loader’s class path. This may seem chicken-and-egg-like because static block execution is part of class loading. Right, but recall the other more dynamic bits of Gosu that are not written to disk. Also recall we support both modes of execution: from source and from class files — the source file-based classes need to load too. The idea is that Java application developers incorporating Gosu, essentially meaning applications executed with Java.exe and not Gosu.exe, will likely choose to compile the Gosu classes they create in their projects to class files. As such with Gosu app code precompiled developers aren’t bothered with wedging Gosu initialization in their app’s to support dependencies on Gosu code not compiled to class files. In other words, the first Gosu class file that loads will automatically setup Gosu so that source-based classes and such will be loaded via our protocol handler. The only requirement for using Gosu in a Java app is that the core Gosu jars be included on the class path. Other than that there should be no difference whether your Java app’s classes are Java or Gosu.

That about wraps it up. Essentially what we’ve done is implement a virtual class loader in the form of a URLConnection. It does the work of finding and producing bytecode for a given Gosu source file; everything a real class loader does aside from actual class definition. Which brings us back to what were after all along. Unbeknownst to the Java application class loader, it is now loading and defining our Gosu classes simply by deferring to our clandestine gosuclass protocol handler. Gosu no longer has a “special” class loader” and now sits at the cool table with the other Java classes. “HE DIDN’T FALL?  INCONCEIVABLE.” (Vizzini to Inigo Montoya)

There were some things I deliberately glossed over or just didn’t cover here. Mostly I avoided compiler stuff dealing with how we compile Java with Gosu class references in it. Well, most of this is still a work in progress. I have a side project where I’m tricking javac into compiling Java with Gosu type references in it. I’ll write about that next, but in summary javac can be made to compile with Java and Gosu intertwined. In fact with the technique I’m using Gosu’s entire type system can be used from within Java with no modifications made to javac. I also skipped over IDE related specifics. Each IDE has its own way of supporting (or not supporting) multi-language integration. The IntelliJ IDE plugin we’re currently focused on has all the support we need — I have another side project with Gosu-in-Java integration limping along in IJ. In fact the guys at JetBrains graciously fixed a blocking bug for me in their most recent release — a two week turnaround!  Those guys are awesome.

What Differentiates Gosu From Other Languages?

A recent question posed on our forum at gosu-lang:

What is that differentiates Gosu from other ‘similar’ languages and that makes Gosu unique?

I couldn’t agree more regarding the importance of differentiating the language.  In fact there isn’t much point in producing a new language if it can’t set itself apart in a significant way.  And I mean measurable significance in terms of us day-job programmers who are looking to get stuff done faster, better, and cheaper for our customers.

Many of the newer JVM languages offer only grammatical improvements over Java.  Sure, Java is a dated language still without properties, closures, composition, extensions, etc., but we can still build the same systems using Java with only marginally more effort than with most of the other newer languages.  And I mean total effort including the impact of the language on design, testing, refactoring, and general ease of understanding, not just the insignificant effort in the number of characters you have to type.  In fact most of the newer languages, I’ll argue, make our lives harder in ways that trump whatever modern features they offer.  For instance, newer dynamic languages leave us in the dark ages of pre-IDE wisdom — no parser feedback to speak of, no deterministic code completion, limited navigation, almost no discoverability, no trustworthy refactoring, etc.  Not to mention no basic static analysis and type checking from the compiler.  Not that bad if you’re writing something smallish in a text editor.  Otherwise, over time I’ll wager a net loss in productivity for most development teams heavily invested in a dynamic language, not that they’ll ever know it. I can make this claim from experience here at Guidewire.  Our success in developing a huge stack of technology, including Gosu, can lend itself in part to state-of-the-art tooling.  Further I’ll say rather confidently that we would have likely failed as a company without it.

What’s striking is what lures a budding development team away from Java isn’t the dynamic languages per se, but the systems atop the languages.  For instance, Rails is the primary reason for the spike in Ruby usage.  Before this phenomenon dynamic languages were pretty much left for dead or a sideshow at best.  Like functional languages they had several opportunities to overtake the imperative OOP ones, but have always failed to gain lasting traction as general purpose languages. My observation is that most red-blooded programmers prefer imperative OOP because it makes sense to them.  That’s really all there is to it, it just makes plain sense to human beings because that’s how we naturally model the world around us.  But Java has given us such a horrible experience with building web applications that we are looking outside the marriage so to speak.  It’s that bad.  I blame the hodgepodge of half-baked tools that are the result of EJB and the JCP fiasco — the same bureaucracy that has delayed basic language features like properties, closures, etc.  The point I’m making is that it’s not the dynamic languages so much that are drawing folks from Java as it is the systems built around these languages.  They, at least in the fledgling stage of a project, provide real, measurable productivity gains, which is what matters.  As you’ll see this is the point I’m working toward with Gosu.

Before we move on to Gosu, though, I’d also like to share my disappointment with some statically typed JVM languages.  Of the few newer ones none significantly helps us programmers advance the ball.  Take Scala.  It’s probably the most popular new kid on the block, yet it’s feature-set and type system are so overwhelmingly complicated most salary paying companies rightly keep it out of their code bases.  Don’t get me wrong, in many ways I like and respect Scala; It’s designer understands the advantage of static typing and it has quite a bit in common with Gosu, but its roots are functional and its feature-set is academically driven (read, hard for us nut crackin, hard workin OO programmers to understand so we can get stuff done and keep it done).  A web search exposes more than enough rants on Scala’s complexity, basically the same reasons functional programming on the whole remains a niche – a lot of simple ideas combined in ways to form overwhelming complexity.

If complexity isn’t enough to deter you, Scala’s flexibility should.  Most language grammars leave wiggle room for expression resulting in mini-dialects, but successful languages thus far tend not to vary too much in dialect.  Java’s success, for instance, is due in part to this characteristic — if you’re an experienced Java programmer, you won’t have much trouble reading and understanding Java, no matter where you go.  Not so with Scala.  If you’re a Java programmer experimenting with Scala, you’ll tend to write Java-like OO code in it.  If you’re a “clever” programmer, guess what?  You can pretty much write Haskell in Scala… Type classes? Monads? Implicit parameters? No problem.  The point is that Scala, beyond its horrifyingly complicated set of features, or perhaps because of them, is also a Tower of Babel.  Again I’m more concerned with the 99-percenter OO programmers who’re paid to impress customers than with the 1-percenter programmers who’d rather impress their peers.  So if you can somehow hire yourself a steady stream of green-blooded Haskell hackers, maybe Scala is for you, but probably Haskell is more for you.

Still think maybe Scala could be in your future?  Witness Kotlin.  Scala is so danged complicated the wizards at JetBrains invented a new language to take its place, namely Kotlin.  I was there when its authors introduced the language at the JVM Language Summit last year where they proclaimed it a Scala-lite of sorts because they basically began with Scala and removed a lot of stuff until they were satisfied (I’m paraphrasing, but that’s the gist).  On the whole Kotlin feels like a reasonable language, but like most Java alternatives, it doesn’t really offer much beyond some grammatical improvements, some of which are difficult to argue as improvements, like multiple inheritance and operator overloading.  Have we not learned from our errors with C++?  Language designers should be made to fix at least one good hard bug in an ancient C++ MFC application before they form an opinion on multiple inheritance, operator overloading, or macros.  I’ve seen some things, man.  In any case it’s hard to argue Kotlin isn’t a net improvement over Scala.  But as I’ll explain, it doesn’t matter.

The intention here is to talk about Gosu and what it has to offer not only in terms of what sets it apart from other languages, but also, and more importantly, why Gosu can help normal development teams become significantly more productive.  As I pointed out earlier the other new languages provide only marginal gains via typical grammatical changes.  I know that’s rather bold of me to say, but consider any of the Java challengers.  None of them offer anything more than an improved Java and/or functional programming to a degree.  There really isn’t much there that warrants a massive shift from Java – a closure here and there reduces some typing and may (or may not) improve readability, properties in the architecture can help relieve some pain, class extensions are a breath of fresh air, but really these are just improvements on Java’s base OO design.  And Java’s going to have most of that stuff sooner or later; Java 8 is just around the corner.  Please don’t misunderstand.  I loathe Java’s slow progress.  No properties?  Really guys?  Wait for Java 8 for closures?  Ugh!  Defender methods?  Sigh.  So for now there’s a niche market for JVM languages to offer features at a faster pace and Gosu does not hold back in that department.  We think we have a nice balance of features, so if it’s closures or properties or enhancements or delegates or what have you, Gosu will not disappoint.  But that’s not important. The features are there because Gosu should have them, but they are not the raison d’être.  Why the language then?  We are so explosively enthusiastic about Gosu because of one giant, game changing idea: Gosu’s Open Type System.

I’ve written extensively on the subject as have others at Guidewire.  To my surprise it has been challenging to convey the idea.  So far it’s a technology you have to experience for yourself to realize its benefits.  Evangelizing it is akin to demonstrating the qualities of Blu-ray on DVD, but harder.  It demos so naturally that most people miss what just happened, even after we say, “Look! We just had an intelligent conversation with a toad!”  In programming we tend to concentrate on the conversation and not the participants (the types), because who cares, they’re just types and of course they can talk.  It’s difficult to get across that the types involved in our demonstrations are really just raw data with which you normally can’t communicate, let alone have the compiler tell you whether or not your conversation makes sense.

The idea itself is simple: programmatic expansion of the type system.  The general idea isn’t new, some call it meta-programming.  But meta-programming as it exists is reserved for dynamic languages where there’s little benefit for development since it exists only at runtime; the dynamic language parser can’t help me discover anything about the new type or the fact that the type even exists.  The unique and enormous achievement with Gosu is that we’ve unified two otherwise discrete concepts –data and static types.  Programmers can now work directly with raw data as first-class types and do it statically leveraging all the productivity features offered by modern, high quality IDEs.  But there’s a key piece of information missing in that last marketing slogan of a sentence – the batteries are not included.  Worse, we have to manufacture the batteries in the form of IDE plugins.

Deep down I’ve always felt the key to Gosu’s success as an open source language was to have a strong presence in one or more IDEs. After all that is where static languages really shine; tooling rules.  Our first foray into the IDE world took us through the gauntlet and ultimate dead-end for languages other than Java, namely Eclipse.  I won’t get into the details other than the JDT was designed exclusively for one thing, the Java language.  To those other static language providers who dare to build a truly useful and competitive plugin in Eclipse, I bid you well; make the best of your orgy with the JDT classes and AspectJ weaving.

IntelliJ IDEA is our IDE of choice here at Guidewire and has been for ten years.  As an experienced user of both IDEs I feel confident in my view that IntelliJ is notches above Eclipse in every way: features, quality, and architecture.  We didn’t integrate Gosu with IntelliJ at first for a couple of reasons: it wasn’t free and it wasn’t the market leader.  Well I suppose the main reason was that I had started the initial development of the Eclipse plugin to get the ball rolling in advance of our open sourcing Gosu; it seemed like the right direction at the time.  Live and learn.  As a company up until nearly a year ago we were still kidding ourselves that we could build our tooling around Eclipse.  Then after talking with our IDE team about the full-retard scenario they were facing with weaving etc. I became convinced we had to punch out, and quick.  As it happened shortly before that time I was also exploring IntelliJ’s plugin architecture as an alternative.  I had developed a simple Gosu plugin to provide debugging and basic highlighting for our internal devs.  It was a smallish project, but I learned enough to know that JetBrains designed a truly open architecture for custom language plugins.  Fast forward six months… without getting into details I began exploring a proof of concept project on fully integrating Gosu into the world of IntelliJ.  It wasn’t long before I was convinced that not only should I continue down this path, but also that Guidewire should redirect its Eclipse effort toward IntelliJ.  This was the IDE for Gosu and, therefore, at least to my caveman brain, for Guidewire.  It wasn’t an easily won battle, but it was well worth the fight.

Now Gosu is well on its way to achieving professional quality tooling in IntelliJ, which I have to say given JetBrains’ high bar is quite an achievement.  So thanks to our hard working IDE team we’ve finally released our first preview of the plugin.  As the “preview” qualifier suggests it ain’t perfect, but it’s at a stage now where you can be productive with it, which is a huge milestone for us.  Because until now Gosu’s powerful type system was not demonstrable to the outside world.  Now that it is I’m optimistic about Gosu’s future in the wide world of open source.  But as I pointed out earlier regarding Ruby’s success with Rails, Gosu’s success also depends on contributing systems such as web frameworks, OR layers, and the like.  But unlike Ruby on Rails, due to Gosu’s static typing and full IDE support it has quite a bit more potential to scale both in terms of performance and project size.  And to make matters worse for the other new languages (let’s not pretend we don’t keep score in this game), Gosu has an unfair advantage with its open type system.  And some smart people have been busy bees building impressive frameworks and systems leveraging every ounce of it.  This is where the rubber really meets the road.

Now with Ronin, the web framework for Gosu created by Gus Prevas, and Tosa, the SQL-focused OR layer for Gosu authored by Alan Keefer and Carson Gross, programmers can be extremely productive building hardcore web applications in Gosu with IntelliJ.  Where else can you build web applications where the links are statically verified?  Where else can you directly reference your SQL DDL and access queries with static types using code completion, all with no code generation? What other static language lets you modify types corresponding with your web content, even the structure of the types, and immediately see the results in the browser?  (I’m going to keep going with this.)  Can your JVM language directly expose arbitrary XML schemas where elements are types and attributes are properties with no code generation?  Unless you have Dana Lank’s XSD/XML framework and your language is Gosu, it can’t.  And even if it could, it couldn’t dream of having an IDE like IntelliJ refactor the name of an XML element and automatically rename all references to it in your code.

This is the kind of differentiation I’m talking about.  Sure, base language features like closures and properties enhance the experience for programmers on many levels, but they don’t increase overall productivity anywhere near the scale of the open type system.  With it Gosu offers something entirely new to programming; another dimension if you will.  The open type system breathes life into the otherwise dead world of raw data.  This is the world we struggle with as programmers.  This is where we waste precious time and resources reading and deciphering and parsing and mapping and reinventing, and we do this over and over. The cracks and crevices between data and our world of objects is where the bad bugs lie. With Gosu suddenly data has the potential to be directly accessible as typed objects, be it XML, WSDL, properties, SQL, data markets, business rules, or you name it.  As the list of data domains supported by Gosu’s type system grows, so does the potential for new and interesting applications to leverage them.  But most importantly, the potential for real productivity gains is available to everyone right now with Gosu, Ronin, Tosa and XML on IntelliJ.  I invite you to take a test drive and let us know what you think on our forum.  We’re always more than happy to help with questions and are excxited to respond to any feedback you might have.

Gosu CR IntelliJ Plugin (and Other Goodies)

Today, we’ve finally released our IntelliJ IDEA plugin for use with the Gosu Community Releases, along with a new release of Gosu itself.  A few open source projects using Gosu will also be releasing updated versions to work with the new Gosu and IntelliJ plugin releases.  The official Gosu CR announcement is over on the gosu-lang group:


While the official IntelliJ IDEA plugin announcement is on the new gosu-idea group:


(Note that these releases are not part of any current Guidewire products and won’t work with any current Guidewire products; these are simple the standalone “community release” versions of the Gosu language and the plugin).  I’ve been playing with the plugin for several weeks as part of getting Tosa ready for an associated release, and I can definitely say that they make programming in Gosu a lot faster and a lot more fun.  So if you’ve been turned off by Gosu in the past due to the weak tooling support, especially if you’re used to Java IDEs, now’s the time to start checking it out!

An Apology for Agile: When to (Not) Use It and How to Make It Work

After reading this post about scrum yesterday, I went over to comment on the article on HackerNews, and was genuinely saddened by the overall negative tone of the comments there around scrum/agile. It seems like a fairly large percentage of people have only had negative experiences with “agile” processes and are either actively hostile to them or, at best, don’t see the point.

My overly-long comment response got eaten by what appeared to be a server timeout, but I think a blog post is a better explanation anyway. So here’s what you might consider an apology for agile process, wherein I’ll discuss what problems it solves, when not to use it, and how to avoid screwing it up if you do decide to use it.

In this discussion, I’ll refer to “agile” as a general thing, even though there are many variants on it. For purposes of this discussion, “agile” is a development process that involves timeboxed iterations of fixed length, work broken down into small-scale stories, and a product owner who generates (most of) the stories and decides on their priorities. That’s a simplification, and there are many variants on that theme, but for purposes of this post that’s what I mean.

We’ve All Got Problems, Right?

Anyone writing code is, kind of by definition, following some sort of process, even if it’s not an explicit process: your “process” might consist of writing code, testing a few things, deploying your website, and then repeating the cycle. So there’s always some current or default process that you’re following, and the only reason you should ever consider switching processes to agile (or anything else) is to solve a problem. If everything’s working great for you, then stop reading right now: keep doing what you’re doing!

The analogy that comes to mind here is, oddly, barefoot running. I struggled for years with shin splints and knee problems to the point where I could run, at most, once a week, on a trail, or else I’d get horribly injured. Eventually I tried barefoot-ish running (in those weird toe shoes) as a way to try to avoid those injuries and, for me, it’s worked wonders. There are people out there who evangelize barefoot running as if everyone should do it no matter what, because it’s better for you or “more natural” and so forth, but you know what? If you’re a runner and you don’t have recurring knee or lower leg injuries and you’re happy running, then keep doing whatever it is you’re doing! It’s working! The last thing you want to do is fix something that’s not broken. If you get shin splits and knee pain, by all means, try the barefoot thing; it might help. But if it ain’t broke, don’t fix it.

A development process is like that: if what you’re doing is working, keep doing it. If someone tries to tell you there’s One True Way to develop software and that if you’re not doing full-time pairing with test-driven development using story cards and iterations and daily standups . . . well, just ignore them, and be content in the knowledge that if that guy is your competitor, your business is going to do just fine.

What Agile Does

So what problems does agile solve? Primarily, agile solves problems that arise due to the interaction between developers and product owners. If you’ve got a division of labor where one person or set of people is responsible for defining what the product should do and prioritizing the features, while other people are responsible for the actual implementation, then you’re likely to run into these problems; if there’s no such division, then you’re much less likely to run into these problems, and agile is likely to be far less helpful. I can think of seven specific problems that agile helps address. Don’t have these problems? Then agile isn’t going to help you.

The first problem is killing developer productivity by constantly shifting priorities and directions. The classic problem here is that a developer starts working on feature A, but gets interrupted because the product owner decides suddenly that feature B is more important. The next day, after talking with a prospective customer, the product owner decides that feature C is really the most important thing. As a result, the developer is left with a bunch of half-finished work, and everyone loses. The primary mechanism for addressing this in agile is the iteration/sprint, which is supposed to be a “time box” where priorities are adjusted only at the start, but not within the time box.

The second problem is an inability for product owners to make well-informed tradeoffs around priorities. For example, in order to decide which out of features A, B, C, and D to work on, it’s important that a product owner know how much relative work those features are. D might be the most important individual feature, but if A, B, and C combined as are much work as D, then that combination of features might be more compelling. Without reasonably-accurate estimates, a product owner can’t make those tradeoffs. Agile attempts to address that with relatively-estimated, small-scale stories (which are ideally also fairly independent, but that’s often easier said than done).

The third problem is having too many unnecessary meetings and status checks. I realize this sounds odd given the number of meetings and ceremony that often accompanies agile (standups, estimation meetings, acceptance meetings, retrospectives, demos . . .), but the theory behind the daily standup meeting is to give everyone who cares about the project’s status a well-known place to listen in, so they don’t bug people for status updates at random one-off times.

The fourth problem is an inability to accurately predict when a project will be done, primarily with the aim of either cutting scope or moving the deadline (or adding developers, which is always dicey), and knowing that one of those will need to be done as much ahead of time as possible. If you’re doing continuous deployment, this probably matters a whole lot less. If you’re releasing packaged software with a hard ship date, it matters a lot more. Agile attempts to address this by empirically measuring the team’s “velocity” and then comparing that to the number of “points” of work left, which (in my experience) tends to work much, much better than just constantly estimating and re-estimating when things will be done. (More on this later, because it’s probably the most important bit.)

The fifth problem is frustrated developers due to poorly-defined features. I’ve been in situations where developers attempted to start work on a particular feature only to find that the product owner hadn’t really thought it through, and that tends to just lead to a bunch of frustration and wheel-spinning: at best you waste time while you wait for a hastily-conceived answer, at worst the developer just makes their own decisions about how things should work and manages to get it completely wrong. Agile attempts to address this problem via story generation and estimation; if you can’t estimate a story, or it seems way too big, it’s a pretty good sign it’s not well defined yet.

The sixth problem is the temptation to adjust the quality knob in order to meet a date target. This one is pretty self-explanatory, I’d imagine, to anyone who’s ever actually developed any software. Agile attempts to address this by getting a shared definition of “done-ness” up front, and then providing accurate information around progress such that other levers can be pulled instead.

Lastly, this isn’t so much a problem per se, but agile builds in time for reflection on the product and the process. The rhythm of iterations gives you natural points for retrospectives where you analyze what’s working and try to change what’s not.

Again, don’t have those problems? Then agile probably isn’t going to buy you much. Have those problems? Maybe it’ll help.

Let me also take this chance to say that development processes work best when they’re voluntarily adopted by the team in question in response to real problems that they want to address. When the developers themselves see the process as something they want, because it helps them do their work well, you’re much more likely to succeed than when the process is imposed on the developers by some outside agent to solve their problems. As a general rule, trying to get developers to do anything which they don’t perceive as helping them do their work is going to be a failure. Developers are happy and think they’re doing awesome work but product owners feel like they can’t prioritize and have no visibility? You’re in for some rough conversations if you’re a product owner or manager trying to impose a new process. Developers are frustrated by constantly changing priorities, vaguely-defined features, and constant nagging about when things will be done? They’ll likely be much more receptive to trying agile.

One problem that agile most definitely doesn’t solve is a dysfunctional organization. Agile evangelists sometimes spin it as a way to make a dysfunctional organization functional, which is precisely wrong the wrong thing: agile can help competent, well-intentioned people be more productive by improving the communication and information flow between different parties in the development process. If people are incompetent, if management and development are at odds on fundamental issues (management wants stuff done as fast as possible, developers don’t want to cut corners), or if the developers don’t trust the product owners to make prioritization decisions around features, agile isn’t going to solve any of those problems. Agile can perhaps help build back trust in an organization by allowing a team to be successful, but if the organizational prerequisites aren’t there, it’s not going to work. And, of course, agile is definitely not a guarantee of success, it’s just one thing that can, in the right situations, help make success more likely.

When Everything Looks Like A Flat-Head Screw

Allow me to make another poor analogy. A development process such as agile is tool, so think of agile as a flat-head screwdriver. The general problem domain for flat-head screwdrivers is effectively “attaching things to other things,” but a flat-head screwdriver is only useful if you’re going to attach those things with flat-head screws. If you have a phillips-head screw, you can maybe wedge a flat-head screwdriver in at an angle and try to use it, but if someone tells you to do it, you’re going to think they’re an idiot. If they ask you to pound a nail in with it, you’re really going to be annoyed. Sometimes, you need a different type of screwdriver, sometimes you need a hammer, and sometimes you just need superglue. Just because you’re attaching two things together doesn’t imply that you need a flat-head screwdriver or that one will even be helpful. And if your only exposure to flat-head screwdrivers is in situations where you really need a hammer, you’re going to think that flat-head screwdrivers pretty much suck, and you’ll wonder why any idiot would ever want one.

Unfortunately, there are agile disciples who seem to omit all those little subtleties. “Flat-head screwdrivers are awesome for attaching things together!” they say. “If it doesn’t work for you, you must be doing it wrong.” Or maybe, just maybe, you really just need some glue, and a screwdriver isn’t going to help at all . . . The one-process-fits-all evangelists have really done a lot to harm the popular perception of agile, I’m afraid.

So when should you *not* use agile? Agile works best when you have a team of people working with a product owner in a known technology and problem domain on definable features that can be relatively estimated with reasonable reliability and built and delivered incrementally. It doesn’t work so hot for, well, anything else. If you’re prototyping something, if it’s a small scale effort (i.e. a few days) you can fit it into agile by doing a timeboxed spike, but for large-scale prototypes it’s inappropriate. If it’s a fundamentally difficult or new problem domain, where you can’t reliably predict even the relative difficulty of any task, agile isn’t going to work. If there are major non-linearities to the work such that things are highly unpredictable, agile isn’t going to work. For example, performance tuning doesn’t fit the model at all: you have no idea how long it’ll take to make something run 50% faster, or if it’s even possible, so writing a story card that says “Make process X run 50% faster” is pretty pointless since there’s no possible way to estimate it. If you’re doing a large-scale refactoring or re-architecting project that’s really an all-or-nothing thing, agile doesn’t work so well; estimates are likely to be unreliable, and you don’t have the option to cut scope by shipping some stories out of that but not others. Hopefully you get the idea. The core of agile is really relative estimation of small-scale stories, and if that’s not possible then it’s not going to work, and things are going to break down pretty severely. In my experience agile also doesn’t deal well with large-scale architectural decisions; those have to be made outside the process, before it starts, or at some point during development you have to temporarily jettison agile as you re-architect things, then re-start your sprints.

Agile is neither useless nor a suicide pact. Use it when it works, do something else when it doesn’t, and use the built-in feedback mechanisms to adjust the process based on the problem domain. If you’ve got a bunch of screws to turn, use a screwdriver. If you’ve got a bunch of nails to bang in, by all means put the screwdriver down and go get a hammer instead. Don’t assume everyone else has flat-head screws to turn, but don’t assume that everyone else has nails either: there’s room in this world for all kinds of tools, and people using ones you don’t find useful are quite probably just solving a different set of problems than you are.

How To Not Screw Up Agile

Here at Guidewire we’ve done our fair share of experimenting, and I think we have a pretty good idea of what doesn’t work, as well as what can work in the right circumstances. And in my perspective on the world, the two non-obvious-and-easiest-to-screw-up aspects of agile that are critical to its success are relative estimation and agreement on “doneness.” We certainly didn’t do those at first, and I hear of lots of other teams that make the same mistakes, so I can only assume it’s an oversight that a lot of people make. The two things work together, and without them everything else in agile kind of falls apart.

Agreeing on doneness means deciding ahead of time what it means for a story to be finished. Sometimes people call this “done done,” since it’s not uncommon for a developer to say something like, “Oh yeah, the FooBar widget is done, I just need to write the tests for it,” with the implication that merely being done doesn’t actually mean work has halted, so it’s not really finished until it’s really truly done done.

Anyway, it’s absolutely critical that doneness be defined up front and that everyone, product owner and developers and QA and anyone else involved agree on it. Does it mean that unit tests are written? That QA has signed off and all identified bugs fixed? That code documentation was added? That it was code reviewed? That it was merged into the main code line? That customer-facing documentation was added? It doesn’t matter what the answers are, it just matters that there are answers that everyone agrees to. (Obviously, though, you want your answers to correlate with the condition the code needs to be in for you to ship it.)

Relative estimation means that stories are estimated in a relative fashion, relative to other stories, rather than in some absolute measure of days. That’s generally done in terms of “points,” with all 1 point stories being about the same size, all 2 point stories being roughly twice as much work as the 1 point stories, and so on. A game like planning poker is often used to help the team converge on estimates to improve their accuracy, and accuracy usually improves over time as the team becomes more familiar with the problem domain. Those relative estimates are then mapped back to actual days by empirically tracking how many stories the team actually gets “done done” in a given period of time, known as the team’s “velocity.” Note that velocity is a team metric, not an individual metric: if you change the team composition, the velocity will change, and the team as a whole can be either greater than or less than the sum of its parts, depending on how well people work together. Also note that velocity is likely to bounce around a bit, especially early in a project, so in practice you’ll often use something like the running average of the team’s velocity for planning purposes. Not perfect, because this is software development that we’re talking about and it’s inherently unpredictable, but it’s far better than anything else I’ve ever seen anyone try.

Relative estimation is much easier to do reliably than absolute estimation; not 100% reliably, of course, but more reliably. Absolute estimation requires a developer to take too many things into account: development time, test-writing time, documentation time, bug-fixing time, the probabilistic chance that something will blow up and go horribly wrong, even general overhead from meetings or other interruptions. It even requires you to take into account individual differences, since Alice might do a story in 1 day that takes Bob 2 days. Taking all of that into account is all really hard to do, as it turns out. Rather than saying something will take “2 days,” which might mean “2 days of uninterrupted work if everything goes smoothly and nothing blows up and I only write a few tests,” it’s much easier to say “this is 2 points because I think it’s about twice as much work as that thing I said was 1 point.” You don’t have to estimate your overhead, or how much time you have to spend on development versus testing, you just measure it. You don’t even have to take who does the work into account, so long as both Alice and Bob take twice as long to finish 2-point stories as they do to finish 1-point stories; Alice could work twice as fast as Bob, and the math still all works out, because you’re measuring overall team velocity, not individual velocity. Maybe your 4 developers get 40 points of work done in 10 days, and maybe they get 8 points of work done; it doesn’t really matter, so long as the estimates are about right relative to each other. That’s how fast you’re working, so now you can start to get an idea of how long the project will take, and make decisions accordingly.

The other crucial advantage is that relative estimation doesn’t pressure people into rushing things to meet the time estimate they gave. As a developer, if you say something is 2 days of work, and you’ve been working on it for 4 days, it’s very, very tempting to just call it done and move on; the psychological pull is pretty strong there. If you said it’s 2 points of work, and the points to days mapping is computed and somewhat variable anyway, it’s much easier to just keep working until it’s done. Maybe our velocity will be lower this sprint as a result, or maybe it was a poor estimate and it was more work than I thought, or maybe that’s just how long 2-point stories take. It’s much easier to just keep working until it’s “done done” if you don’t give a fixed date estimate up front.

I really can’t over-emphasize how important these two things are. If you don’t do these, you’re likely not going to have a good experience with agile. Here’s kind of how things tend to break down. Say you don’t agree on what “done done” means up front. Suddenly, everyone is tempted to adjust the quality knob when the going gets rough, which is exactly what no one really wants. (Note: if that is in fact what management wants, you have deeper organizational problems which agile isn’t going to fix.) Or perhaps you count stories as “done” which aren’t really “done done,” which gives you an inflated velocity for the initial stages of the project, so maybe you think you’re getting 20 points of work done an iteration when in reality you can only get 14 done but you’re fudging the numbers. Now you have two problems: you’ve got 6 points of unscheduled off-the-grid work lurking in the future *and* you’ve got everyone making their plans based on the team doing 20 points of work per iteration, which completely throws off everyone’s ability to plan and prioritize. Or say you try to do absolute estimation instead of relative estimation. Now there’s pressure to cut corners to meet the estimates, and the estimates themselves are wildly inaccurate, but no one is really sure how inaccurate they are because you’re not really rigorous about tracking it or making sure that things are “done”, so once again your ability to measure your rate of progress is gone. Once you lose that visibility into the real rate of progress, not only does it throw off your ability to plan your development schedule and cut scope/shift dates/take other action as necessary, but it starts to make people nervous. Product owners know that they don’t really know how things are going, so they start bothering developers outside of standup meetings and changing priorities mid-iteration to try to exert some control over the process, or they make hasty decisions to try to course-correct, or they just freak out and kill the project because they have no idea if it’ll be done on time and on budget or if it’ll come in 6 months late and 200% over budget, and if it comes in 200% over budget they’ll get fired.

So if you take away just one thing from this ramble of a blog post, let it be this: relative estimation and agreement on doneness are absolutely critical to the success of the agile process.

Summing It Up

So that’s what I’ve got for you this time around. In summary: agile is useful in certain circumstances, it solves specific problems that you may or may not have, it’s only worth trying if you do in fact have those problems and if it seems like a good match to your problem domain, and relative estimation and “done doneness” are essential to the success of the process.

Why There Have Been No Posts Lately…

I apologize for the extended absence of posts here on the dev blog, but I’ve been busy with a new Open Source project:


The JSchema Project is an attempt to define a very simple schema system for JSON documents. JSON, for those who are not familiar, is a subset of Javascript that has proven to be a useful and less verbose alternative to XML for data-interchange:

    "group" : "Developers",
    "members" : [
      { "id" : 1, "first_name" : "Joe", "last_name" : "Smith" },
      { "id" : 2, "first_name" : "Jennifer", "last_name" : "Mitchum" }

It is easy to parse and produce, and integrates well with newer front-end technologies written in Javascript.

The motivation for this project, it will not surprise you, was Gosu. An excellent engineer at Amica, JP Camara, had been playing around with a JSON-based Type Loader for Open Source Gosu. I contacted him about adopting a schema-based approach, and began looking at the industry standard, JSON Schema. It quickly became apparent that JSON Schema was too complicated, and that a small extension to JP’s simple and elegant template approach would yield an expressive schema language, with a Gosu Type Loader to boot.

To give you a taste of how simple JSchema is, here is the schema for the JSON example above:

    "group" : "string",
    "members" : [
      { "id" : "integer", "first_name" : "string", "last_name" : "string" }

I hope that schema makes intuitive sense to most people: JSchema was designed such that the schema corresponds closely to the documents it describes, and it should be easy to take an example JSON document and transform it into a JSchema schema.

The JSchema specification is still very young but I think it is reasonably complete. We may add some more core data types (a ‘bytes’ datatype for raw byte data seems like it might be useful, for example) but the core ideas are working out well in our Gosu implementation (I’ll blog more about that in a later post).

If you are interested in the specification and/or participating in its design (or making the website prettier), you can fork it here. Participation is very welcome!

JVMLS Talk Is Now Up

You can watch the video here:


Many thanks to the folks at Oracle for getting it up.

Faking Builders With Gosu

One of the features of Groovy (being aped by Kotlin) is builders, a nice syntactic way to compose nested content.

A canonical example of this, I think, would be building an HTML page. The Groovy page has an example of this:

html {
    head {
        title"XML encoding with Groovy"
    body {
        h1"XML encoding with Groovy"
        p"this format can be used as an alternative markup to XML"

        / an element with attributes and text content /
        ahref:'http://groovy.codehaus.org' ["Groovy"]

        / mixed content /
        p [
            "This is some",
            "text. For more see the",
            ahref:'http://groovy.codehaus.org' ["Groovy"],
        p "some text"

Gosu does not support builders, but I thought it would be fun to see how close we could get to them with the syntactic tools that Gosu does have. Using names arguments and default parameters, in particular, I think we could get pretty darned close.

So, the first trick I’m going to use in faking builders is something I call the Keefer Gambit: I’m going to create an empty HtmlUtil interface and then create an enhancement of that interface that supplies all the methods you will use to build html. This allows you to mix in the HTML functionality wherever you want it by simply implementing the (empty) interface. This (hack|elegant solution) for mixins in Gosu was first pointed out by Alan Keefer, hence the name.

OK, given that general approach, the syntactic tools we can use to fake builders are named arguments and default parameters. Let’s look at implementing the html element method in our enhancement:

  function html( head : String = "", body : List<String> = null) : String {
    var bodyContent = body == null ? "" : body.join( "\n" )
    return "<html><head>\n${head}</head>\n<body>${bodyContent}</body>\n</html>"

Nothing too fancy here: there are two parameters, each with a default value. The function will return these two elements, wrapped up in the appropriate tags. If the parameters are omitted then the default values are used, and you can use named parameters when invoking this method because it is defined in Gosu.

I’m not going to walk through the implementation of each method, rather I’ll just point you to the sample code I put up on GitHub here:


But using that code, I can write this code in an Example client:

    :head = head(
        :title = "XML encoding with Gosu"
    :body = {
        h1( "XML encoding with Gosu" ),
        p( "this format can be used as an alternative markup to XML" ),
        /* an element with attributes and text content */
        ahref( 'http://gosu-lang.org', "Gosu"),
        /* mixed content  */
            "This is some",
            "text. For more see the",
            ahref('http://gosu-lang.org', "Gosu"),
        p( "some text" )

Now, I’ll admit that isn’t quite as clean as the Groovy: we violate the DRY principle when we generate the head element, for example. And things are a bit dicier around when you use parens vs. curly braces. Also, I have to admit, this is a relatively simple example. But I think that this is a pretty clean translation nonetheless, and it was doable without adding any new syntactic features to Gosu.

Gosu, unlike Groovy, Scala or Ruby, requires parentheses around method arguments, and it doesn’t support more exotic things like var-args (it has a nice list literal instead) or implicit arguments: we favor regularity over succinctness beyond a certain point. Despite this, I’m comfortable with the expressiveness of the syntax of Gosu.

I’d rather see innovations in libraries, leveraging The Open Type System rather than much continued innovation in the core language syntax.

I’ll be at OSCON this week. I’m always happy to talk gosu and/or drink beer if you are up here.

Blocks Inside Out

In my previous post I gave some practical applications of blocks (a.k.a closures) in Gosu. In this post I’m going to look at one particular method mentioned in that article, where(), and try to show how the concept of blocks emerges from code you might be writing today. I hope by the end of it you’ll understand what I mean when I say that:

    “Classes help us abstract concepts, blocks help us abstract algorithms”

As a refresher, where() is a method in Gosu available on anything implementing Iterable<T>. It takes a single block and returns a new sublist of items where the block returns true.

Now, backing up a step, let’s look at some code that filters a list written in vanilla Java:

  List<String> shortStrings = new ArrayList<String>();
  for( String s : aListOfStrings ) {
    if( s.length() < 10 ) {
      shortStrings.add( s );
  return shortStrings;

Very standard Java code. I want to focus on the bolded part of the algorithm: it’s a little bit of logic that actually provides the filter. Note that it is very specific to this particular application of the filter algorithm: this bit of logic is deeply baked into the for-loop. This makes it difficult to extract the code around it, which is all fairly general, into a reusable bit of software.

Now, if filtering lists of strings on length was common enough in your code, you might extract this logic into a function and make the maximum (+1) length of the strings a parameter:

  List<String> findStringsLessThanLength( List<String> aListOfStrings, int length ) {
    List<String> shortStrings = new ArrayList<String>();
    for( String s : aListOfStrings ) {
      if( s.length() < length ) {
        shortStrings.add( s );
    return shortStrings;

This has generalized the operation a bit: you now can pass in varying lengths and get different results. However you are still stuck with the fact that this method can only filter strings with a length less than some number. If, for example, you wanted strings greater than some length, you’d need to either introduce another method or, yikes, introduce another parameter.


At this point, depending on how crazy you are willing to get and how much you hate copy and pasting the same algorithm everywhere, you may choose to introduce a Predicate concept to your project. This allows you to pass an anonymous inner class to a method, where the anonymous inner class expresses the filter critera:

  return Iterables.filter( aListOfStrings, new Predicate<String>() {
    public boolean apply( String s ) {
      return s.length() < length;

Hmmm. Well, some people might think that’s better. I’m not so sure.

The problem (in my eyes) is that the amount of syntax overwhelms the core concept: we are just filtering a list, for goodness’ sake.

Looking back at the original implementation:

  List<String> shortStrings = new ArrayList<String>();
  for( String s : aListOfStrings ) {
    if( s.length() < 10 ) {
      shortStrings.add( s );
  return shortStrings;

It’s really just that bolded part we care about. Wouldn’t it be nice if we could pass just that to a function?

That’s exactly what blocks let you do:

   return aListOfStrings.where( \ s -> s.length() < 10 )

The block (again, the funny bit after the ‘\‘) has a single parameter, a string named ‘s‘, and returns true if s has a length less than 10, false otherwise. You can think of the block as a mini-function, if that helps, or as an mini-anonymous-inner-class. All blocks do is take arguments and return values, in an conveniently-inlineable syntax. And, of course, you write whatever expression you’d like in the bolded area, so this is a truly general solution to filtering things.

That’s what I mean when I say that blocks help you abstract algorithms: ideally, you’ll never have to write that filtering code ever again. Or that sorting, mapping or partitioning code either.


Effective Blocks

One of the cool features in Gosu is closures, colloquially known as “blocks”. Blocks act kind of like inline function definitions, allowing you to plug bits of logic into other algorithms. (One my standard lines in the talks is “Classes are a tool for generalizing concepts, blocks are a tool for generalizing algorithms.” Pithy and, hey, maybe even true.)

Trying to describe blocks in general terms doesn’t appear to be a great way to explain them: people get lost in the terminology and syntax. The way that I finally “got” blocks wasn’t by staring intently at a text book or wiki page, it was by seeing them in action.

In this vein, I’m going to describe some common programming problems you might face and then show you how blocks can help you solve them elegantly. I hope this will help you understand blocks in Gosu and how use them effectively.

“I’ve got a List of X and I need a List of Y”

This is perhaps the classic problem that blocks can help solve:

  • “I’ve got a list of Claims, and I need a list of Claim IDs.”
  • “I’ve got a list of Employees, and I need a list of last names.”

To address this problem, Gosu has added the map() function to Iterable objects. Taking the two examples above, you can write this code:

  var ids = claims.map( \ c -> c.ID )
  var lastNames = employees.map( \ emp -> emp.LastName )

The block is the funny little thing starting with a ‘\‘. Between the ‘\‘ and the ‘->‘ are the arguments to the block (remember, blocks are kind of like a mini-function) and after the ‘->‘ comes the body of the block.

Taking a look at the first line in the code example, we have a block that takes an argument c of type Claim (Gosu infers this type) and that returns the ID of that claim.

The map() function then uses this block to run through all the claims in the claims list and create a new list, filled with the ID’s of each claim.

Nice, eh?

Here’s another example:

“I’d like to sort List X by attribute Y”

Some concrete examples might be:

  • “I’d like to sort a list of Policies by their effective date.”
  • “I’d like to sort a list of Contacts by their age.”

Gosu has an orderBy() function for this:

  var sortedPolicies = policies.orderBy( \ p -> p.EffectiveDate )
  var sortedContacts = contacts.orderBy( \ c -> c.Age )

In Java, you might use a Comparator (and then try to remember when to return -1, 0, or 1) in conjunction with Collections.sort(). I hope you’ll agree that the Gosu above is easier to write and also reads more clearly.

The advantage of the block-based approach becomes even more glaring when you want to add secondary sort columns:

  var sortedPolicies = policies.orderBy( \ p -> p.EffectiveDate )
                                .thenBy( \ p -> p.Status )

Imagine the Java Comparator for that. Ugh.

“I’d like all the elements in List X where Y is true”

This is another classic:

  • “I’d like all the Claims that are past due.”
  • “I’d like all the Employees that are active.”

Gosu has an where() function for exactly this:

  var pastDueClaims = claims.where( \ c -> c.DueDate < Date.Today )
  var activeEmployees = employees.where( \ emp -> emp.Active )

The where() function returns a new list where any items that do not meet the criteria expressed in the block are filtered out.

Very nice.

“I’d like to split a List of X up on attribute Y”

This is a more advanced usage of blocks, but it can come in quite handy in some situations:

  • “I’d like to split a list of automobiles up based on their color.”
  • “I’d like to split my friends up by their birthday month.”

Gosu has a partition() function for this:

  var carsByColor = cars.partition( \ c -> c.Color )
  print( "All red cars: " + carsByColor["Red"] )
  var friendsByBirthdayMonth = friends.partition( \ f -> f.Birthday.Month )

The partition() function splits a list up based on the block you pass in. So, in the first line of code in this example, we split a list of cars up by their color, and get a map, where the key is a color and the value is a list of cars of that color. In the next line we print the list of cars that are red.

You could partition on any attribute: make, model, year, whatever. Blocks help generalize the operation.

Composing Methods

Gosu really starts to sing when you compose these methods:

  • “I’d like to get the ids of claims past due, ordered by the claim due date”

We can compose three methods together to solve this quite succinctly and clearly:

  var pastDueClaimIdsByDueDate = claims.where( \ c -> c.DueDate < Date.Today )
                                       .orderBy( \ c -> c.DueDate ) 
                                       .map( \ c -> c.ID ) 

The corresponding java code would be more verbose, and would probably conflate the various operations, making it more difficult to understand what the code was doing.

“Hmm, this is all data structure manipulation…”


The sweet spot for blocks (in Gosu, at least) is in data structure manipulation. There are other areas they can come in handy (callbacks, listeners) but usually you’ll be using them to transform one data structure into another.

Personally, not a day that I’m writing java code goes by that I don’t wish I had blocks. Once you get used to them, it can be very hard to go back. I hope that this quick overview helps you use blocks effectively in your own Gosu code and have a bit more fun when you are coding.

Gosu in Dr. Dobbs

Gosu is this month’s Dr. Dobbs Language of the Month:


Scott McKinney, the creator of Gosu, talks about some of the history and features of the language, and once again points out the Open Type System as the core innovation of Gosu.