I have always found that the burn-up and burn-down charts are very informative and fit to the iterative story-based development very well. Every project that I work on, I will try to figure out different ways to generate burn-up and burn-down charts.

Two months ago, I took the job on putting Platform on Sprints. After some consideration, I have decided to follow the setup that I have for the AF team, creating the stories in the form of JIRA issues. However, the chart generation that I had for AF team was still semi-manual, which means that it takes a couple of minutes to download, and a couple of minutes to update the stats every morning. The worst part is that when I get busy or sick, I will forget.

So my first action item was to figure out how to generate the same kind of charts with the push of a button. The idea seems to be easy enough:

1. Figure out the search criteria to retrieve all the JIRA issues of the backlog
2. Count the issues that are in different states
3. Update the data with the counts, and check it into Perforce.
4. Refresh the chart with the updated data

Number one and two were actually not that hard, because Guidewire GScript has a nice WebServices support. With a few tries, I was able to count the beans.

Here is an example of the data generated. I think you get the idea just looking at it.

Date,Day,Closed,Deferral Requested,In QA,Open Stories,Open New Features,Open Bugs
10/09/2009,41,55,0,1,13,7,40
10/12/2009,42,55,0,1,14,7,40
10/13/2009,43,56,0,0,14,7,40
10/14/2009,44,56,0,0,16,7,41
10/15/2009,45,56,0,0,21,8,42
10/16/2009,46,58,0,1,19,8,42
10/19/2009,47,58,0,2,28,8,42
10/20/2009,48,58,0,6,26,8,42
10/21/2009,49,58,0,6,26,8,42
10/22/2009,50,58,0,7,25,8,44

Number three took less time but a bit of research because Perforce Java library’s API is not exactly straightforward.

It took me a while to figure out how to do the last one. After looking into JFreeChart and Google Chart API, I eventually turned to my dear friend, Tod Landis, who is also my partner at Entrance, and he quickly drafted an entrance script for me. Based on it, I was able to write a template that can be used for all teams within a few hours.

PLOT
  very light yellow  area,
  light yellow filled circles and line,
  DATALABELS
  very light orange  area,
  light orange filled circles and line,
  very light red  area,
  light red filled circles and line,
  very light blue area,
  light blue filled circles and line,
  very light gray area,
  dark gray filled circles and line,
  very light green area,
  dark green filled circles and line,
  all AXISLABELS
WITH
  ZEROBASED
  TITLE "Sprint"
  TITLE X "Days"
  TITLE Y "Points"
  SCALE Y 0 100 0
  CLIP
  legend
  gridlines
  collar
  no sides
SELECT
  `Open Bugs`,
  `Open Bugs`,
  date,
  `Open New Features`,
  `Open New Features`,
  `Open Stories`,
  `Open Stories`,
  `In QA`,
  `In QA`,
  `Deferral Requested`,
  `Deferral Requested`,
  `Closed`,
  `Closed`,
  day
from report;

Please note this is the final PLOT script, there are other SQLs run before this to import the data into the MySQL database, sum up the data to produce a stacked chart, and even out the labels.

And I now have this chart generated automatically every morning with the help of a windows scheduler.

One thing we’re currently working on, for a variety of reasons, is compiling down our in-house programming language into Java bytecode (and just for the record, I can’t make any promises about when we’ll be done or around when or even if this stuff will make it into future product releases).  Part of the fun therein is learning about the internals of the JVM, as well as finding all the crazy edge cases in your own language.  And few things are as, um, “fun” or have as many edges as try/catch/finally statements.  So rather than delving into philosophy or agile methodology this time around, I’ll go into detail on an area of the JVM that most people never have to (or probably want to) delve into.

If you’d asked me two weeks ago, I would have assumed that finally blocks are a feature implemented by the JVM:  it’s such a core language thing, it’s got to be built-in, right?  Much to my surprise, the answer is no:  finally blocks are implemented by inlining the finally code at all possible exits from the try or associated catch blocks, wrapping the whole thing in essentially a “catch(Throwable)” block that rethrows the exception when it finishes, and then adjusting the exception table such that the catch clauses skip over the inlined finally statements.  Huh?  (Small caveat:  prior to the 1.6 compiler, apparently, finally statements used sub-routines instead of full-on code inlining.  But we’re only concerned with 1.6 at this point, so that’s what this applies to).

That probably doesn’t make any sense, so let’s rewind a bit with some JVM basics around exception handling.  Exception handling is built into the JVM, and it’s done so in the form of declaring try/catch blocks within a given method.  What you’re saying is “between point A and point B, any exceptions of type E should be handled by the code at point C.”  You can have as many such declarations as you need, and when an exception is propogated to that method the JVM finds the catch block that matches based on the extent and the type of exception it handles.

So with a simple example:

public void simpleTryCatch() {
  try {
    callSomeMethod();
  } catch (RuntimeException e) {
    handleException(e);
  }
}

You might end up with bytecode like the following (I’m showing it in the format the ASM Eclipse plugin does:  it’s an invaluable tool for learning how the Java compiler works, and I find it’s formatting pretty easy to read.  The “L0″ and such are code labels.):

public simpleTryCatch()V
TRYCATCHBLOCK L0 L1 L2 java/lang/RuntimeException
L0
  ALOAD 0
  INVOKEVIRTUAL test/SimpleTryCatch.callSomeMethod()V
L1
  GOTO L3
L2
  ASTORE 1
  ALOAD 0
  ALOAD 1
  INVOKEVIRTUAL test/SimpleTryCatch.handleException(Ljava/lang/RuntimeException;)V
L3
  RETURN

So we’re saying the catch statement covers the full extent of the try block (though not the GOTO statement at the end), and that on a RuntimeException we should transfer control to L2.  If the try statement complete, it jumps over the catch statement and continues on.  When the RuntimeException handler is invoked, the exception is on the top of the stack, so we store it to a local variable slot.  We then load the “this” pointer and the exception argument in order to invoke the exception handler.  The catch block then falls through to the end; if there were additional catch blocks, it would jump over them.

So let’s add in a finally block and another catch statement and see what happens to the bytecode.  Suppose we have the following totally contrived method:

public void tryCatchFinally(boolean arg) {
  try {
    callSomeMethod();
    if (arg) {
      return;
    }
    callSomeMethod();
  } catch (RuntimeException e) {
    handleException(e);
  } catch (Exception e) {
    return;
  } finally {
    callFinallyMethod();
  }
}

In this case, we end up with this far-less-obvious bytecode:

public tryCatchFinally(Z)V
TRYCATCHBLOCK L0 L1 L2 java/lang/RuntimeException
TRYCATCHBLOCK L3 L4 L2 java/lang/RuntimeException
TRYCATCHBLOCK L0 L1 L5 java/lang/Exception
TRYCATCHBLOCK L3 L4 L5 java/lang/Exception
TRYCATCHBLOCK L0 L1 L6
TRYCATCHBLOCK L3 L7 L6
TRYCATCHBLOCK L5 L8 L6
L0
  ALOAD 0
  INVOKEVIRTUAL test/SimpleTryCatch.callSomeMethod()V
L9
  ILOAD 1
  IFEQ L3
L1
  ALOAD 0
  INVOKEVIRTUAL test/SimpleTryCatch.callFinallyMethod()V
L10
  RETURN
L3
  ALOAD 0
  INVOKEVIRTUAL test/SimpleTryCatch.callSomeMethod()V
L4
  GOTO L11
L2
  ASTORE 2
L12
  ALOAD 0
  ALOAD 2
  INVOKEVIRTUAL test/SimpleTryCatch.handleException(Ljava/lang/RuntimeException;)V
L7
  ALOAD 0
  INVOKEVIRTUAL test/SimpleTryCatch.callFinallyMethod()V
  GOTO L13
L5
  ASTORE 2
L8
  ALOAD 0
  INVOKEVIRTUAL test/SimpleTryCatch.callFinallyMethod()V
  RETURN
L6
  ASTORE 3
  ALOAD 0
  INVOKEVIRTUAL test/SimpleTryCatch.callFinallyMethod()V
  ALOAD 3
  ATHROW
L11
  ALOAD 0
  INVOKEVIRTUAL test/SimpleTryCatch.callFinallyMethod()V
L13
  RETURN

So what’s going on here?  (Note that the labels are numbered in the order they’re created by the compiler, not by the order they appear in the code). First of all, you’ll notice that both exception handler blocks are now split in two, from L0 to L1 and from L3 to L4.  That’s because the range from L1 to L3 is where the finally block is inlined due to the return statement.  Since exceptions thrown during the finally block shouldn’t be caught by catch blocks associated with the same try statement, that range has to be excluded from the exception table.  The exception entries with no type declared are from the finally block.  It has to handle any exception thrown from the try statement or from the catch blocks, but it also has to exclude any inlined finally statements so that the finally block doesn’t catch exceptions thrown by that same finally block.  It has to have three splits, because in addition to the inlined finally within the try block, the catch(Exception) block also contains a return statement.  You may also be surprised to see that the finally block appears a total of five times in the code.  You’ll see the first inlined finally from L1 to L3, corresponding to the return statement in the try block.  The second finally block is a bit more confusing:  in this case it’s being inlined into the end of the first catch block, which then jumps over the finally code that’s there in the fallthrough case.  (I personally would assume that it would jump down to the fallthrough case, rather than inlining the code again).  The third time it appears is from L8 to L6, around the early return statement within the second catch block.  The fourth time the code in the finally block appears is from L6 to L11, which corresponds to the exception case:  that’s to ensure the finally block is executed in the case of an unhandled exception thrown out of the try block or any of the catch blocks.  Notice that it stores the exception as normal, performs the call in the finally statement, and then loads and rethrows the exception.  The last finally block is for the fall-through case:  it’s where the end of the try block jumps to.

If we were to have nested try/catches or try/finallys, things would get even stranger.  A return from the inner try statement needs to have both the inner and outer finally inlined into it, and the exception table needs to be set up such that exceptions thrown by the inner finally are caught by the outer catch statements and the outer finally, and exceptions thrown by the outer finally aren’t caught by anything.  At this point you can probably start imagining the kind of state your compiler needs to carry around in order to know what to to inline where and how to partition the exception table correctly.

It’s interesting, for me at least, to see how the JVM designers chose to push the finally statement into the compiler rather than building it into the VM; clearly it can be built by the compiler, thus making the VM that much simpler, it just makes life a little rougher for folks like us who are building another language on the JVM.

Knowing how things are implemented by the compiler can actually make it a little easier to figure out what happens in certain odd circumstances.  For example:

try {
  return "foo";
} finally {
  return "bar";
}

will end up returning “bar” since the finally statement is inlined before the return statement, so the return from the finally block executes first and the original return statement never does.

String value = "foo";
try {
  return value;
} finally {
  value = "bar";
}

will return “foo” since the value to return is stored in the stack prior to the finally statement executing, and then restored and returned afterwards.  (My example didn’t show that, but that’s what you’d see if you were to look at the bytecode).  So the finally block changing the “value” variable has no affect on the in-progress return statement.  And lastly, something like:

while(true) {
  try {
    return "foo";
  } finally {
    break;
  }
}
return "bar";

will return “bar”.  This one surprised even me at first, but it makes sense once you realize that a break statement is just a GOTO in bytecode, so when the finally block is inlined as part of the inner return statement, that GOTO statement executes before the RETURN instruction and completely skips over it to the end of the loop (the same logic holds true for a continue statement inside a finally block).

On our end, we’ve decided to disallow return, break, and continue statements within the finally block due to the rather unexpected semantics (and C# disallows them as well, so I feel like we’re in good company with that decision).

If anyone actually found this enlightening, I’ll consider doing some future posts about other interesting issues we’ve run into when generating bytecode, both due to the JVM itself and due to the occassional impedance-mismatch between our language and Java around things like closures and enhancements and generics.

It’s often the case that the hardest things to get motivated to do are those with a very speculative chance of success.  If you put in a lot of time at the gym, you’re going to get in better shape.  If you spend time cooking, you’ll probably have some reasonable to eat at the end.  If you work hard at your job, there’s at least a pretty good chance that you’ll be rewarded for it.  For all those activities, there’s a straightforward and somewhat predictable correlation between effort and results.

Other endeavors, however, are far messier.  If you start a business, it’s likely you’ll spend a huge amount of time and end up failing.  If you really commit to a relationship, there’s always a chance that you’ll invest a ton and come away with nothing.  And the same generally holds true for most software projects, where it’s likely that you’ll build something only to find no one use it.  The amount of effort required to build a programming language, for example, is astronomical, yet the expected number of users for any new language essentially rounds to zero.

So how does one go about getting motivated to undertake an effort that’s likely to end in failure?  Well obviously, you have to be willing to fail, but how do you go about that?  At least one technique I’ve found is to focus on the fact that success is stickier than failure.

What does that mean exactly?  That means that success tends to last a while, but that failure is almost always temporary (and if it’s not . . . well, then you have a different set of tradeoffs to consider).  It might take you three or five tries to start a successful business; so what?  A successful business can last for decades, potentially.  Your heart gets broken; so what?  If you keep trying, odds are good that some day you’ll end up in a stable, long-term relationship.  No one uses your open-source project, or the product managers don’t like how you’ve implemented a feature.  So what?  You figure out why not and try again, or perhaps you’ll find some existing project to devote your energies to instead.  Eventually, chances are you’ll hit on something worthwhile, and that project will stick around for a long time and deliver more than enough utility to compensate for the effort seemingly wasted on the failed efforts.

Of course, that theory only works if the cost of failure is mainly the loss of the associated resources; if you’re in a position where trying to force some new internal project on your organization could kill the company if it fails, the set of tradeoffs is a bit different.  But usually that’s not the case; people are usually much more worried about feeling like failures, or about wasting time and resources, or about the way their friends or their family or their managers will react to them if they fail.  But the funny thing is, once you have a success or two, no one really remembers the failures, because they’re so transient when compared to the successes.  No one remembers that it took you three tries to really nail some feature; all they remember is that you nailed it, and that it’s so good that no one has to touch it anymore and you can worry about different things.  And no one remembers all those stupid ideas for internal projects that you’ve had and abandoned over the years, they just remember that one awesome idea that you had that’s become totally indispensable to the company.

Failure is a temporary condition, a mere rest stop on the way to success.

I tend to try to make parallels between my work and whatever else is going on in my life, even when they’re inappropriate.  Case in point:  I got married at the end of July (yay!) which meant that my wife and I spent a large percentage of our free time for the prior year or so planning the wedding (d’oh!).  As anyone who’s gone through that knows, it’s kind of a never-ending list of things to do and, more specifically, of decisions to be made.  Not only that, but each decision is about something that’s costing you hundreds or even thousands of dollars.  Normally, when making decisions involving large sums of money, I tend to be pretty careful; I read product reviews online, I comparison shop, etc.  But that all takes time, and when you have to make several such decisions per day for months and months, you just can’t put that level of effort into each and every decision.  In the end, if you don’t want to go crazy you have to just accept that you’re going to make some mistakes, and that some of those decisions will turn out to be sub-optimal.

So what in the world does that have to do with software development?  Perhaps this is a stretch, but as I reflected on that experience I started to realize that most development work is a similar never-ending series of decisions, each of which could potentially have a large long-term impact if it’s done wrong, or at least could require a large amount of effort to fix.  What do I name this method?  Where do I put this class?  Do I store this value as two booleans or a tri-state enum?  Do I add this extra field to the object, or should it logically be its own object/row in the database?  Do I mock this out in testing, or use the production version of the class?  Do I use a factory to build this or just a normal constructor?  Do I make this method public, protected, or private?  Is this class properly decomposed?  Should this class really be split into multiple pieces?  On and on it goes.  Even if all those decisions are relatively neutral in the sense that you can make any of them work right now, each one of those decisions can have a large downstream impact on your ability to test, maintain, extend, and understand the code base.  One bad decision could literally be the difference between making a future feature take 15 minutes and making it take 15 hours, or between making debugging easy or turning it into a multi-day expedition.

One interesting complication with software, though, is that the time to make the decision itself is a cost, and with software development time is usually everything (assuming you take correctness for granted, i.e. that you will spend whatever time is necessary to test and debug and ensure correctness).  So even though the name of a method is critical to people’s understanding of it, their ability to read code that calls it and their ability to discover it, how much time do you spend making the decision?  Is the name that you come up with after fifteen minutes of deliberation likely to, on average, save you fifteen minutes of development time compared to the name you came up with right away?  Well, it depends on how bad your initial name is, and how much better the other name could be.

As developers, we all make hundreds of those meta-decisions a day:  do I make a snap judgment and keep going, or do I spend a bunch of time thinking or talking or prototyping?  It’s almost like you need to draw a three-dimensional curve, plotting, for each amount of time that could be spent deliberating, what the the probability is for different kinds of time savings over, say, the next five years:  if we spend 15 minutes on this, there’s a 5% chance we’ll come up with a decision that’s worse by 1 hour over five years and a 50% chance we’ll find something that’s 1 hour better and a 45% chance we’ll come up with something exactly as good, but if we spend 30 minutes on it we now have a 60% chance of saving 1 hour . . . then all you have to do is do a little calculus to find the amount of deliberation time that maximizes the expected savings – that deliberation time . . . and then throw in some extra variables to account for the potential that your probability graph itself has a certain probability of being wrong in certain ways . . . not to mention a discount factor on the value of time now versus the value of your time five years from now to represent the importance of shipping on time . . .

Clearly, I’ve thought about this way too much.

The same reasoning applies to decisions beyond just the “do we spend time talking about this” decision.  Do I just hack this in?  How much will it cost us down the line?  Do I refactor this?  Will the time spent refactoring be paid back in the long run?

By now, you’re probably wondering where I’m going with this.  Honestly, as much as I wish I did, I don’t have an answer as to how to determine when to spend the time and when to just hack away.  My point is merely that it’s a subject worth thinking about.  Most of us have an internal dial that’s set one direction or the other, where we’re either biased to just hack away or we’re biased to deliberate a ton and refactor a ton and try to make everything perfect.  And I think much of the time we take for granted that our dial is set correctly, and that we’re making the right tradeoffs.  If we’re just hacking away, it’s obviously because we just need to get this done and this is probably close enough to optimal anyway, so why bother talking about it?  And if we’re refactoring or talking about how to do things, it’s because it’s critical to contain complexity and avoid technical debt so it’s really important to get things as right as we can.

Every so often, it’s good to do a little meta-cognition and ask yourself if you have the dial cranked to the appropriate place given your project and all the other constraints.  Are you bounding ahead just because you want to, when some extra time and discussion would lead to a better decision?  Or are you overthinking decisions that are so speculative or so inconsequential that you should just make the decision and move on?

I’ve been a bit hesitant to write something about this, since I think it’s something of a delicate subject . . . it’s just a bit odd to write about, knowing that people I work with will read it.  But it’s an important enough subject, and one that I feel strongly enough about, that I think it’s finally time to say something about it.

For those of you who are reading this and don’t know, my technical title is something like “PolicyCenter Architect.”  Since I think the word “architect” in a software context has taken on all sorts of horrible connotations (it conjures up images of people who draw boxes and arrows but don’t actually write any code themselves), I tend not to use it personally.  Instead, I tend to think of my role on the team as being the guy who should be blamed when things don’t work.  Not the guy who should be blamed when something he did doesn’t work, but just the guy to blame, period.

I know it’s something of an extreme stance to take, which is why I think it’s worth an explanation.  Coming at it from a different angle:  my job is to make sure that the project doesn’t fail for technical reasons.  If the code is buggy, or it doesn’t perform, or it’s not flexible enough or in the right ways, then I’ve failed to do my job.  It doesn’t really matter why, or who wrote the code:  I’m supposed to make sure that doesn’t happen, and to do whatever it takes to get there.  In some ways that’s fundamentally unfair, since I can’t possibly control everything that goes on, yet I’m personally responsible for it anyway.  I honestly think that’s still how it has to be.

Taking responsibility for things is a powerful thing, especially when they’re things that you don’t directly control.  Part of being a professional is certainly taking responsibility for your own work, and not passing the blame to someone else when things go wrong.  But I think it’s even more powerful to simply say I will not let this fail.  Full stop.  But I think the only way to really get to that mindset is to take responsibility for that failure; otherwise, there will always be the temptation to throw up your hands and pass the blame on to someone else, and at the moment you absolve yourself of responsibility for that failure you’re lessening your personal drive to do something about it.  The other thing that needs to happen is that you have to accept the fact that things could fail, that you have limited control over that, and if they do it’ll be your fault anyway.  Failure sucks, and it’s a hard thing to swallow:  on my current project, it took me about five months to really come to peace with the fact that I was going to have to make a lot of very, very hard yet speculative decisions and that my best efforts could still end in failure (and a very, very costly failure at that).  Once I finally did, though, I slept a lot easier, and it freed me up to do better work than I otherwise would have been able to do.  Whatever role I managed to play in making the project succeed, I think it was significantly aided by the fact that I got over being afraid of screwing up.

I think there are three primary reasons why taking responsibility and blame for things you don’t actually have control over works out.  First of all, it forces you to do whatever you can to minimize that chance of failure.  Since you don’t have complete control, you can’t get it down to zero, but you sure have an incentive to get it there.  And not only that, but you have an incentive to do what’s right for the project as a whole, not just for your part of it or for you personally.  That extra drive, extra motivation, and whole-project orientation will lead to you doing more to make the project succeed than you would otherwise.  Secondly, it lets you make hard decisions under uncertain conditions.  A lot of projects fail simply because no one is willing to make those decisions; it’s always less personally risky to leave things as they are, or to take the well-trod route, or to pass the decision off to someone else even if they’re not in the best position to make the decision.  Sometimes, you’re the best person to make the decision, and you just have to be willing to do it.  Sometimes you’re not, so you have to pass the analysis off to someone else, but in order for them to really make the best decision they can they have to know that you’re going to back them up and not blame them if it goes wrong for some reason.  Lastly, while I really have no idea what anyone who works with me really thinks, I like to think that having someone willing to take that level of responsibility lets everyone else on the team do better work too, since they know that someone has their back, they can focus on doing their best instead of worrying about trying to look good, and they have the freedom to do what they think is the right thing.

It’s worth mentioning that the flip side of taking the blame is not taking the credit.  I’m not the first one to point out that good leaders should accept blame and deflect responsibility, but I think it’s one of those things that it’s hard to repeat enough.  Never, ever take credit for other people’s work (even if you helped out), and when in doubt err on the side of taking too little (preferably way too little) credit.

My hatred of method overloading has become a running joke at Guidewire. My hatred is genuine, icy hot, and unquenchable. Let me explain why.

First Principals
First of all, just think about naming in the abstract. Things should have good names. A good name is unique and easy to understand. If you have method overloading, the name of a method is no longer unique. Instead, the real name of the method is the sane, human chosen name, plus the fully qualified name of each argument’s type. Doesn’t that just seem sort of insane? If you are writing a tool that needs to refer to methods, or if you are just trying to look up a method reflectively, you have to know the name, plus all the argument types. And you have to know this even if the method isn’t overloaded: you pay the price for this feature even when it isn’t used.

Maybe that strikes you as a bit philosophical. People use method overloading in java, so there must be some uses for it. I’ll grant that, but there are better tools to address those problems.

In the code I work with day to day, I see method overloading primarily used in two situations:

Telescoping Methods
You may have a function that takes some number of arguments. The last few arguments may not be all that important, and most users would be annoyed in having to figure out what to pass into them. So you create a few more methods with the same name and fewer arguments, which call through to the “master” method. I’ve seen cases where we have five different versions of a method with varying numbers of arguments.

So how do I propose people deal with this situation without overloading? It turns out to be a solved problem: default arguments. We are (probably) going to support these in the Diamond release of GScript:

  function emailSomeone( address:String, subject:String, body:String,
                         cc:String=null, logToServer:boolean=false,
                         html:boolean = false ) {
    // a very well done implementation
  }

A much cleaner solution. One method, with obvious syntax and, if your IDE is any good, it will let you know what the default values of the option arguments are (unlike method overloading.)

True Overloading
Sometimes you truly want a method to take two different types. A good example of this is the XMLNode.parse() method, which can take a String or a File or an InputStream.

I actually would probably argue with you on this one. I don’t think three separate parse methods named parseString(), parseFile() and parseInputStream() would be a bad thing. Code completion is going to make it obvious which one to pick and, really, picking a unique name isn’t going to kill you.

But fine, you insist that I’m a terrible API designer and you *must* have one method. OK, then use a union type (also probably available in the Diamond release of GScript):

  function parse( src:(String|File|IOStream) ) : XMLNode {
    if( src typeis String ) {
      // parse the string
    }
    ...
  }

A union type lets you say “this argument is this type or that type.” It’s then up to you to distinguish between them at runtime.

You will probably object that this syntax is moderately annoying, but I’d counter that it will end up being fewer lines of code than if you used method overloading and that, if you really want a single function to handle three different types, you should deal with the consequences. If it bothers you too much, just pick unique names for the methods!

So?
Let’s say you accept my alternatives to the above uses of method overloading. You might still wonder why I hate it. After all, it’s just a feature and a pretty common one at that. Why throw it out?

To understand why I’d like to throw it out, you have to understand a bit about how the GScript parser works. As you probably know, GScript makes heavy use of type inference to help developers avoid the boilerplate you find in most statically typed languages.

For example, you might have the following code:

  var lstOfNums = {1, 2, 3}
  var total = 0
  lstOfNums.each( \ i -> { total = total + i  } )

In the code above, we are passing a block into the each() method on List, and using it to sum up all the numbers in the list. ‘i‘ is the parameter to the block, and we infer it’s type to ‘int’ based on the type of the list.

This sort of inference is very useful, and it takes advantage of context sensitive parsing: we can parse the block expression because we know the type of argument that each() expects.

Now, it turns out that method overloading makes this context sensitive parsing difficult because it means that when you are parsing an expression there is no guarantee that there is a single context type. You have to accept that there may be multiple types in context when parsing any expression.

Let me explain that a bit more. Say you have two methods:

  function foo( i : int ) {
  }

  function foo( i : String ) {
  }

and you are attempting to parse this expression:

  foo( someVar )

What type can we infer that the context type is when we parse the expression someVar? Well, there isn’t any single context type. It might be an int or it might be a String. That isn’t a big deal here, but it becomes a big deal if the methods took blocks, or enums or any other place where GScript does context type sensitive parsing. You end up having lists of context types rather than a single context type in all of your expression parsing code. Ugly.

Furthermore, when you have method overloading, you have to score method invocations. If there is more than one version of a method, and you are parsing a method invocation, you don’t know which version of the method you are calling until after you’ve parsed all the arguments. So you’ve got to run through all the argument types and see which one is the “best” match. This ends up being some really complicated code.

Complexity Kills

Bitch, bitch, moan, moan. Just make it work, you say. If the java guys can do it, why can’t you? Well, we have made it work (for the most part.) But there’s a real price we pay for it.

I’m a Berkeley, worse-is-better sort of guy. I think that simplicity of design is the most important thing. I can’t tell you how much more complicated method overloading makes the implementation of the GScript parser. Parsing expressions, parsing arguments, assignability testing, etc. It bleeds throughout the entire parser, its little tentacles of complexity touching places you would never expect. If you come across a particularly nasty part of the parser, it’s a good bet that it’s there either because of method overloading or, at least, is made more complicated by it.

Oh, man up! you say. That’s the parser’s and tool developer’s problem, not yours.

Nope. It’s your problem too. Like Josh Bloch says, projects have a complexity budget. When we blow a big chunk of that budget on an idiotic feature like method overloading, that means we can’t spend it on other, better stuff.

Unfortunately, because GScript is Java compatible, we simply can’t remove support for method overloading. If we could though, GScript would have other, better features and, more importantly, fewer bugs.

That is why I am hate method overloading. And so can you.

George Bernard Shaw put it best when he remarked that all progress depends on the unreasonable man.  Progress in just about any area requires that you not be content with the status quo, since otherwise you wouldn’t bother trying to make it better.  But at the same time, constant discontent with the world is a rough way to live; if nothing is ever good enough, you’ll always be unhappy, and if you’re always going to be unhappy what exactly is the point of working so hard to make things better?  On a pretty fundamental level, isn’t that drive to make progress and improve things the result of a desire to turn some discontentment with the current state of affairs into contentment?  But once you reach that contented state, how do you keep going?  Shaw could just as easily have said that all progress depends on the discontented man, but who wants to spend their live discontented?

Dealing with that requires finding a balance between contentment and complacency on one hand and discontentment and action on the other hand.  Logically, they’re kind of two ends of one scale; they’re P and not-P.  You can only have one, right?  Thankfully (or not, depending on your perspective), the human brain isn’t actually constrained by the rules of logic.  Logic tells us “P or not-P” but your brain says, “Why can’t I have both again?”  Paradoxes are a part of life, and they’re not just the result of lazy thinking; as much as the analytic philosopher/computer scientist in me wants the world to be a rational, P or not-P sort of place, it’s not, and we can find ways to use that to our advantage.  

My solution is to take that one single axis of “am I happy with X” and treat it as if it’s actually two independent axes; I call them “contentment” and “satisfaction” for lack of better terms, such that I can be content while still not being satisfied with things.  If I’m running, I can simultaneously be contented that I ran what for me is a good time while still being dissatisfied that I didn’t set a new personal best.  And if I’m working, I can simultaneously be extremely happy with how much I’ve been able to do while being appalled that I haven’t done more, and I can be content with the state of our code and our tools and how advanced they are compared to what our competitors try to work with while still feeling like I won’t actually be happy until they’ve come miles further than where they are now.

This is just my opinion, of course, but I think a lot of people tend to fall into just one sort of thinking or the other:  either they’re happy enough with things that it reduces their drive to improve them, or they’re driven to improve things but they’re unhappy (and in many cases, that can lead to feelings of futility, which eventually just kills motivation altogether).  I’m not sure I really have the balance mastered myself, but I’m pretty convinced that it’s critical to doing the best work that I can do and avoiding complacency on one hand and burnout on the other.

There’s a more sinister form of the discontentment that afflicts software engineers as well, and that’s the gold-plating/no shipping afflication that leads to never deciding something is good enough to move on.  So if you’re an engineer, that’s yet another potential affliction to master; you need to find a way to be content and enjoy your work while not getting burned out, and to constantly improve everything while still being able to ship code and move on to new projects.

If there’s one thing I look for in engineering candidates these days, it’s probably that constant drive to improve things, and that recognition that things can always be better.  Because they can.  There’s no end-point in the game we’re playing (yes, even if you’re a Lisp guy, the answer is still out there); the language, the tools, the libraries, the techniques, the tests, the code, the process . . . no part of the stack is ever truly done.  To me one of the joys and truly unique things about software development is the fact that there is always progress to be made, and that every engineer has the opportunity to make that kind of progress.  I think that constant drive is something that all truly great engineers share.  Be unreasonable.

But even better than that constant drive to make things better is the ability to keep that drive alive over months and years and decades while staying happy, enjoying the work, avoiding burnout, and being able to priortize that against the need to ship and the need to move on to other projects.

It’s funny that Bruce Eckel posted on Artima yesterday about how programming is analogous to writing.  I’ve thought that for a long time, and Eric Roberts, who developed the introductory CS curriculum at Stanford (and is just an amazing teacher in general) liked to point out to incoming freshmen that success in those classes was more closely correlated with SAT verbal test scores than with SAT math scores.  But what I’d been meaning to write about was a bit of a tangent to that:  it wasn’t about how programming is like writing, but rather about how people have wildly different writing styles, which I think everyone accepts (no one expects Dan Brown to compose a novel the same way as Salmon Rushdie or Haruki Murakami), yet people often expect that there’s One True Way to write quality software.  Method X works for them, so if it doesn’t work as well for you, it must be because you’re doing something wrong.

But the truth of the matter is that programmers program differently, just like writers write differently.  One of my favorite discussions of writing comes from Kurt Vonnegut.  In his novel Timequake, he says there are two types of writers:

Tellers of stories with ink on paper, not that they matter any more, have been either swoopers or bashers. Swoopers write a story quickly, higgledy-piggledy, crinkum-crankum, any which way. Then they go over it again painstakingly, fixing everything that is just plain awful or doesn’t work. Bashers go one sentence at a time, getting it exactly right before they go on to the next one. When they’re done they’re done.

It’s an over-generalization, but as over-generalizations go it’s a pretty brilliant one.  Personally I fall into the swooper category; in fact, I’m even more extreme in that I’ll often rewrite from whole cloth large sections or even entire essays (or e-mails, or term papers).  I’ve found that I need to get something down in some roughly-complete form simply in order to collect my thoughts, so I simply couldn’t bash out something perfect one sentence at a time no matter how much I wanted to.  When English teachers in grade school made us write outlines for an essay, I’d write my outline after I was done with the paper, because that was the only way the two would line up.

Interestingly (to me, at least), I’ve found that I’m most productive when I program the same way.  For a long time, I felt kind of bad that I didn’t enjoy doing rigorous test-driven development; it seemed like TDD was some kind of ideal of well-constructed code and if I was just a little more disciplined, I’d fully TDD everything and my code would be flawless as a result.  But in reality, that style is just sub-optimal for me.  Just like when I write, I need to sketch something out in order to really see where I’m going.  So my development methodology these days is often to start with a brutal, hacked-up end-to-end spike of a feature, write some end-to-end tests, and then start building it sideways, back-filling more targeted tests as I go and always keeping it close to some known-good state.  If I try to throw that spike away and start over with TDD, or if I don’t do that spike at all, I go slower and produce code that’s harder to read or modify later.

My point is that there is no one right way to develop; what works well for one person won’t work so well for another person.  Most developers, I think, would admit that different methodologies and techniques and tools are appropriate for different problem spaces; one-person throwaway projects are obviously a different deal from 100-person projects where the code needs to live for decades.  But even within the same problem space, people are just different, and one programming methodology or style doesn’t fit all.

As a developer, it’s your responsibility to figure out what works for you, and that requires some experimentation and often the willingness to try out something that feels horribly awkward and unfamiliar at first.  Unfortunately, it’s often hard to know if you’re giving up too early or if something really just isn’t going to work for you.  So by all means, read all you can about TDD, BDD, pair programming, rapid prototyping, getting real, modeling, or just go out there and start hacking.  Try a method out, ask people who like it, take what works and leave what doesn’t, and find the style that leads to you doing your best work.

And if you ever happen to find yourself in a position of authority, one of the worst things you can do is require everyone to try to program the same way just because it’s the way that works for you or that works for some guy in a book you read.  Give people the freedom to do their best work and, surprise surprise, they will.

Technical debt, we all know, is hard to manage.  To fight against it, you have to (among other things) refactor and improve your code mercilessly.  But along the way, your attempts to make the code base a safer place can actually make them worse:  if you add in a new way to do something without actually removing the old way, you could end up with a code base that’s more cluttered and more inconsistent, making it even harder to understand than if you’d just never tried in the first place.

To take a contrived example, imagine that you’re writing a new test that needs to create a sample hierarchy of widgets and gizmos for your application, and you realize your existing architecture could use some improvements.  Over time you’ve accumulated a bunch of random helper methods that do various things, like WidgetUtil.createSampleWidgetWithOneGizmo(), but the methods therein are brittle, take too many arbitrary parameters, and are difficult to combine to create richer test data.  So, you decide to refactor the test utilities using the builder pattern to make things more fully-parameterizable and chainable.  Great idea, right?  So you create your nifty new WidgetBuilder and GizmoBuilder classes, use them to write your tests, and as predicted they make the data setup a lot clearer and more flexible.

The question, then, is what do you do next?  Do you just use them in for new tests, or do you go back and refactor the 158 existing tests that use WidgetUtil so that they use your new builder classes?  And do you stop there, or do you attempt to kill off all of your old data creation methods in favor of new builder patterns for everything?

When trying to improve the code, the real work is often not in the initial improvement itself, but rather in fully replacing the old way of doing something with the new, improved way.  Either option has some serious potential downside.  On the one hand, forcing the change through every part of the system is usually hugely labor intensive  and carries a high risk of breaking something that was already working, all for no immediate return whatsoever.  On the other hand, leaving things as is just adds to the complexity of the system:  now there are two ways to do X instead of just one, and every subsequent developer needs to understand both of those and know what the differences are.

So what’s the right thing to do?  Well, as with most programming tasks, it comes down to a judgment call about whether to attempt to push it through the system, whether to just add the new change but not attempt to refactor further, or whether to abandon the change and just do things the old way to avoid adding complexity to the system.  Here are a few questions to ask yourself:

• How localized is the change?  Is it likely that most other developers will need to be aware of both ways to do things, or will only a smaller subset have to deal with it?
• How bad is the status quo?  Is the improvement drastic, or merely incremental?
• How much work is the refactoring?  Are there five other places where a similar pattern is present or 500?
• How likely is the refactoring to break things?  Is it a fairly straightforward drop-in replacement or change, or is it something more involved that could turn out to have unanticipated interactions?
• Can automated tools do the refactoring, or is it something that has to be done by hand?
• How long is this system going to be around for?  Is this something relatively throwaway (though it always seems like every program lives longer than its creators expect), or is it something you know you’ll be dealing with five or ten years from now?
• How likely is the refactoring to uncover and fix latent bugs or to otherwise clean up buggy areas in the system?

In general, in my experience most people tend to err too far on the side of not finishing things off and truly eliminating all vestiges of an old way to do something, be it in the form of a method or merely a general approach to a problem.  Especially once a system gets to a certain size, the cost starts to seem prohibitive.  But unfortunately, those are often exactly the times that it’s important to keep the code clean and avoid technical debt; technical debt is much more of a killer on large projects than on small ones.

So next time you come up with a new way to do something, or decide that way X is better than way Y, ask yourself if you’re stopping too early or if you should really be following the refactoring through to the end.

Martin Fowler recently updated his article on technical debt, and we’ve been discussing it in-house lately as well (though isn’t that always a conversation at any company with long-lived products?), so I’ve been thinking about it a lot lately.  

Personally, I think it’s perhaps the most difficult engineering concept for non-engineers to internalize, because most things in the real world just don’t work that way; hence, the necessity of some analogy to a more common real-world concept.  The core feature of development that lets technical debt happen is that the input of one “operation” is always the output of the previous one, meaning that mistakes and shortcuts build up over time, progressively dragging you down.  Everyone has infrastructure that subtly impacts everything they do (if you’re a chef and your pots are poor quality or your kitchen is poorly laid out, it makes everything harder), and reputation can always make your business suffer (if you’re a sales guy and you offend a potential client, that can hurt you long term), but there are very few disciplines where you have to continuously build something up over the course of years.  Even if you’re in construction, once you’re done with a given building you move on to the next one.  But code bases are expected to essentially live forever, meaning that mistakes made during the beginning add up over time.  That just doesn’t happen if you’re a chef, or a salesman, or a doctor, or an artist, or almost any other job you can think of.  For engineers, the concept comes naturally:  everyone understands that the decisions they make now will affect how they do their job in the months and years ahead.  But for someone that’s never experienced that, I think it’s just a very difficult concept to internalize.

That said, it’s an important concept to grasp.  To me, the important part about technical debt isn’t the principal, as it were:  it’s the interest.  Realistically, though, it doesn’t work like real interest.  Rather, it’s more like a tax:  the amount you pay isn’t fixed based on the size of the “debt,” but rather is generally proportional to how much work you want to do, and the size of the “debt” really determines the tax rate rather than some fixed amount of overhead.  (One might argue that there is, in fact, some fixed amount in the form of ongoing maintenance, so there’s probably an argument to be made that the tax analogy isn’t really accurate either.)  But either way you think about it, the important part of the concept isn’t just that there’s a backlog of stuff to fix, but rather that prior decisions that were made affect your ability to work productively in the future.

There are two things that I think are less obvious about how insidious technical debt is.  The first one is that incurring the debt sets expectations artificially high about how much work the team can do; if you incur a ton of debt in the first version of a product in order to get it out the door, you’ve set the expectation that the team can do X amount of work in a 12-month release, when in reality you could only do X/2 without incurring debt . . . and because of that debt, it’s now more like X/2.5.  The second insidious thing is that paying it down requires a huge resource commitment and delivers very little short term benefit.  It often seems like a total black hole; if you’re paying 10% yearly interest on a $100,000 loan, paying back $50,000 on top of the interest only saves you $5,000 a year.  So paying off the debt often seems like a poor investment, which means it just builds up and slowly exerts more and more of a tax on development, making it even harder to do something about it.

Of course, technical debt isn’t exactly measurable, and neither is productivity, but just for fun let’s pretend that we can and do a little  math anyway, since I think it’s an interesting exercise.  Imagine we’re measuring both productivity and debt in feature-dollars, and that we have a team that can do $100k worth of feature-dollars in a given year.  The first version of the product, however, needs to be ready in one year and have $200k worth of features in it.  So to get over the hump, the team borrows $100k at 10% APR.  Of course, the technical debt lenders are cut throat, and in reality it’s always harder to fix things than it would have been do them right in the first place; we can imagine that as if the tech debt lenders charged back-breaking fees, say 30%.  So after year one, we’ve got $200k worth of features and $130k worth of debt costing us $13k a year.

The team realizes that it overextended in the first release, but no one can quite swallow a 50% cut in productivity; the team did $200k the first time around, right?  So instead, they shoot for $120k, thinking that’s a much more reasonable target.  But their original rate minutes the $13k in debt means that to get $120k of features out, they need to incur another $33k of debt, which with fees we’ll round up to $40k.

By the time the third year rolls around, the project is $170k in debt, and the team decides to do something about it.  They decide to scale their dev effort back in half and only deliver $60k worth of features, so they pay $17k in interest on the debt, do $60k worth of feature work, and have $23k leftover for technical debt.  By sacrificing about 1/4 of their total dev capacity for the release (and more like 40% of their actual feature-building capacity), the team manages to reduce the debt down to $147k, saving them all of $2.3k per year in debt.  So next time around, they’re basically in exactly the same boat.

As the debt gets ever higher, there becomes an inflection point where the debt is high enough to nearly bring development to a total halt, and yet so large that nothing can be done about it.  Imagine if the team instead tried to deliver $200k of features in each release.  In the second release, they’re paying $13k in interest, so they have to take out $113k in loans to hit their target, adding maybe $150k after fees.  In the third release, they’re paying $28k in interest, so they have to take out $128k in loans, adding $160k in debt.  So by the fourth release, they’ve got $440k in debt; if they take on no further loans (and after some point you really can’t), their dev capacity will around half of what it should be.  But the debt is also so large that there’s realistically no way to pay it down; it would take five years of no further feature work.  So at that point, it’s basically checkmate for the product . . . either you limp along with a product that doesn’t really evolve anymore and hope that a competitor doesn’t blow by you while you’re standing still, or you try to rewrite the whole thing and hope that doesn’t completely kill the project (which is by far the most likely outcome of a total rewrite effort).

You can play through that scenario with different perceived interest rates, or thinking of the debt as a tax instead of a constant amount, and over longer periods of time, but hopefully it illustrates the problems that I mentioned above, both around artificially increasing expectations for the team, leading to yet more debt, leading to yet more pressure to cut corners, and around the fact that paying down the debt often requires a Herculean effort for very little payoff.  Make of that what you will.  As with real debt, there’s a time and a place to incur it:  sometimes it’s important to hit a deadline, or to get a feature in for a key client, and the interest and fees are worth the cost.  But in the long run, technical debt can’t be allowed to build up to the point where it’s both too large to pay down and too large to allow for future development work, which requires walking a fine line between incurring debt when it’s necessary to get things done fast enough and holding it off or paying it down so that it doesn’t get out of control.