Gosu’s Secret Sauce: The Open Type System

Earlier last week we finally released Gosu to the general public, the JVM language we use for configuring our enterprise applications at Guidewire.   Gosu has been in the works for several years here beginning at the dawn of our engineering history in 2002.  Over the years the language has evolved to meet the growing demands of our applications.  Initially we created Gosu because we needed an embeddable, statically typed, scripting language for business rules.  We needed static typing so that we could 1) statically verify business logic and 2) build deterministic tooling around the language via static analysis e.g., for code completion, navigation, usage searching, refactoring, etc.  Tooling was, and is now more than ever, a vital part of our technology offering.  For instance, we don’t want to burden our users with having to commit our entire API to memory in order to be productive with our platform.  But it doesn’t end there.

Our platform, like that of most large-scale enterprise application vendors, consists of a relatively tall stack of technologies, most are developed in-house.  The stack is comprised of the following subsystems in no particular order: Database Entity/OR layer, Web UI framework, Business Rules, Security/Permissions, Web Services, Localization, Naming services, Java API, Query model, XSD/XML object models, various application-dependent models of data, and Gosu.  A primary challenge with any enterprise application is to adapt to customer requirements through configuration of its subsystems.  As I’ve stated, Gosu is the language we use to accomplish that, but how does, say, the Gosu code in the Web UI or Rules access the Entity layer and XSD-modeled information?  In other words, if a primary goal of Gosu’s static type system is to help users confidently and quickly configure our applications, how can it possibly help with domains such as these which are external to the type system?

This is where Gosu separates from the pack.  Unlike most (all?) other programming languages, Gosu’s type system is not limited to a single meta-type or type domain.  For instance, Java provides one, and only one, meta-type, namely java.lang.Class.  If you want to directly represent some other type domain to Java, you’re out of luck.  Your only options are to write a class library to access information about the domain or resort to code generation.  Neither option is desirable or even adequate in many cases, but as Java programmers we don’t know any better.  Dynamic languages such as Ruby provide varying degrees of meta-programming, which can be a powerful way of dynamically representing other type domains.  But with dynamic  languages we escape type safety, deterministic tooling, and other advantages of static typing, which defeats the purpose of our primary goal of exposing other type domains to the language.  Our customers, for instance, would be at a terrible disadvantage without the ability to statically verify usages of various domains in their application code.  Thus, Gosu’s type system must provide a similar (or better) level of flexibility provided by dynamic meta-programming, but without sacrificing static typing — a tall order.  How did we do it?

Gosu’s type system consists of a configurable number of type domains.  Each domain provides a type loader as an instance of Gosu’s ITypeLoader interface.  A type loader’s primary responsibility is to resolve a type name in its domain and return an implementation of the IType interface.  This is what is most unique about Gosu — it’s type system is open to other domains to participate with first-class representation.  For instance, Gosu does not discriminate between a Gosu Class a Java Class or Entity Type or XSD Type or what have you; they’re all just types to Gosu’s compiler.  This unique and powerful feature extends all the benefits of static typing to a potentially huge range of otherwise inaccessible domains.

Before we consider a specific domain, it is important to understand that not all type loaders in Gosu are required to produce bytecode.  Those that do, like Gosu classes, implement an interface for getting at the corresponding Java bytecode. Those that don’t provide call handlers and property accessors for reflective MethodInfo and PropertyInfo evaluation.  Note all types provide TypeInfo, see IType.getTypeInfo().  For instance, the parser works against the TypeInfo, MethodInfo, etc. abstractions as the means for a level playing field between disparate types. At runtime, however, unless a type provides a Java bytecode class the MethodInfos and PropertyInfos also are responsible for handling calls.  Not requiring bytecode at runtime accommodates a potentially much wider range of type loaders and makes it possible  to quickly prototype loaders. 

With that understanding, we can jump to some specific examples.  Let’s take a look the Gosu Class type loader. It’s job is to resolve names that correspond to the domain of Gosu classes, interfaces, enumerations, enhancements, and templates (primarily .gs* family of files).  The result is an instance of GosuClass, which implements IType etc.  One of the methods in there is getBackingClass().  Essentially, this method cooperates with Gosu’s internal Java classloader to compile the parse tree from the Gosu class to a conventional Java class.  So it follows that Gosu’s compiler performs dynamic compilation direct from source; there is no separate build process or persisted classfiles. This does not imply that the Java class loader is transient, however.  It is not.  Since Gosu is a statically typed language it’s classes are compiled to conventional bytecode and, thus, don’t require any classloader shenanigans as with most dynamically typed languages.  But I digress.

What connects the Java bytecode class to its Gosu type is the IGosuObject implementation; all Gosu classes and most other types implicitly implement this. The getIntrinsicType() method is instrumental, for example, in making reified generics work in Gosu.  An instance of a Gosu class, or any type in the entire type system for that matter, can produce its underlying type via getIntrinsicType().  And from there Gosu’s type system takes over. For instance, you can reflectively access Gosu-level type information from Java code via TypeSystem.getFromObject( anyObject ).

It’s worth repeating that Gosu classes are no more important to Gosu’s type system than any other type.  For instance, XSD types resolve and load in the same manner as Gosu classes.  Given an XSD type loader (we have one internally), Gosu code can import and reference any element type defined in an XSD file by name.  No code generation, no dynamic meta-programming, and all with static name and feature resolution.  The XSD type defines constructors, properties, and methods corresponding with elements from the schema.  In addition it provides static methods for parsing XML text, files, readers, etc.  The result is an instance of the statically named XSD type corresponding with the element name, which can be used anywhere a Gosu type can be used, because it *is* a Gosu type!  Here’s a very simple example that demonstrates the use of an XSD type:

From a sample driver.xsd file:

<xs:element name=”DriverInfo”>
 <xs:complexType>
  <xs:sequence>
   <xs:element ref=”DriversLicense” minOccurs=”0″ maxOccurs=”unbounded”/>
   <xs:element name=”PurposeUse” type=”String” minOccurs=”0″/>
   <xs:element name=”PermissionInd” type=”String” minOccurs=”0″/>
   <xs:element name=”OperatorAtFaultInd” type=”String” minOccurs=”0″/>
  </xs:sequence>
  <xs:attribute name=”id” type=”xs:ID” use=”optional”/>
 </xs:complexType>
</xs:element>
<xs:element name=”DriversLicense”>
 <xs:complexType>
  <xs:sequence>
   <xs:element name=”DriversLicenseNumber” type=”String”/>
   <xs:element name=”StateProv” type=”String” minOccurs=”0″/>
   <xs:element name=”CountryCd” type=”String” minOccurs=”0″/>
  </xs:sequence>
  <xs:attribute name=”id” type=”xs:ID” use=”optional”/>
 </xs:complexType>
</xs:element>

Sample Gosu code:

uses xsd.driver.DriverInfo
uses xsd.driver.DriversLicense
uses java.util.ArrayList

function makeSampleDriver() : DriverInfo {
  var driver = new DriverInfo(){:PurposeUse = “Truck”}
  driver.DriversLicenses = new ArrayList<DriversLicense>()
  driver.DriversLicenses.add(new DriversLicense(){:CountryCd = “US”, :StateProv = “AL”})
  return driver
}

Notice the fully qualified names of the XSD types begin with xsd.  That’s just the way the XSD loader exposes the types; a type loader can name them any way it likes, but it should do its best to avoid colliding with other namespaces by using relatively unique names.  Next you may have discovered the name of the XSD file, driver, as the namespace containing all the defined element types.  That’s also a convention the loader chooses to define.  What’s most interesting, however, is how seamlessly the XSD types integrate with Gosu.  You can’t distinguish them from Gosu classes or Java classes, and that’s the whole idea!  All type domains are created equally.  Notice you can even parameterize Java’s ArrayList with the DriversLicense XSD type.  Isn’t that sweet?  And of course all the powerful static analysis tooling applies including code completion, feature usage, refactoring, etc.

Another interesting tidbit: Type domains aren’t limited to tangible resources like files on disk. For instance, one evening I decided to see what it would take to provide a dynamic type in Gosu, similar to C#’s dynamic type.  In theory, I thought,  the type system could handle a dynamic type as just another type domain.  And wouldn’t you know, it could. Well, in the spirit of full disclosure I did have to tweak the type system internals a bit to better handle the “placeholder” concept, but just a little.  Aaanyway, unlike most type domains this one consists of just a single type name, namely dynamic.Dynamic.  Note Gosu requires that a type live in a namespace, so we can’t name it just Dynamic.  It was surprisingly simple to implement the type loader, you can download the source from our website here (http://gosu-lang.org/examples.shtml).  In a future blog I’ll cover the implementation of the Dynamic type loader (or some other one) step-by-step.

Another super cool type loader is the one from the RoninDB project by Gus Prevas (http://code.google.com/p/ronin-gosu/).  Here Gus provides a zero-configuration JDBC O/R layer via Gosu type loader.  The tables are types, the columns are properties, and no code generation; it just works.  Pretty cool, eh?  Download it and give it a spin.

So far I’ve only touched on the basic concepts underlying Gosu’s open type system.  But understanding the core idea is critically important to appreciate Gosu’s full potential.  More so, if you plan to use Gosu in your personal or professional software projects.  Because once you see the light with respect to type domains and the benefits they provide, it’s hard to think of developing enterprise applications without them.  To that end I promise to blog more and shed more light on building type loaders and the type system in general.

---