The State of Software Development

So I'm not sure how we continue to get by with it. I'm constantly and continually surprised by what passes as quality in our industry. In my particular neck of the woods - the City (financial district of London) - stories abound about all kinds of things. All the FSA, SEC, OCC, ... would need to shut most banks is a fairly competent senior dev or hands-on architect to join their audit teams.

Most of the stories shared amongst my friends and colleagues involve numbers that are astounding to us (even given our rates) but are barely a glitch to the businesses we support. Trading desk lost a few million because the software actually lost the trade? No problem (well, technically they could recreate it by looking at the logs). Sure some dev gets it in the neck, and has to change employers to maintain his bonus level, but they'll be laughing about it down the pub by the end of the month.

At least those governing bodies apply some pressure for us to have our act together at some level. In other domains, the situation is even worse. Who is making the broadband providers accountable for what they do? What kind of flimsy network do you have that I have to bounce my router (that sounds like a south jersey pick up line: "Hey baby, wanna ...") whenever there is a power outage? Why do I have to re-dial into a call center because there is no way for one department to transfer me or themselves to another area? Can I arrange a delivery time in increments smaller than "Tuesday". Without the headlines and correlation to the big £$£ amounts, its death by a thousand cuts for the rest of us.

I recently tried to see if I could get home earlier by switching my flight to land in Portland rather than Seattle, my ultimate destination being somewhere in between. Starting in San Francisco, the computer looked up the next flight leaving for Portland, and routed me from SF to Spokane to Las Vegas to Portland. So technically, the results were accurate. But I had asked for a flight that would get me to Portland as soon as possible, not leave as soon as possible. The system wasn't configured to search by arrival.

In most orgs, IT is considered a cost. And customers are external to the company and are easy to dismiss: "If you just do it the way we've designed it to work, then all will be well." Except they haven't designed it with any grand vision of how it all works. They put in the cheapest solution to take care of the happy day scenario and maybe a few rigid pathways to deal with contingencies. The value add that they all miss, the reason why I miss talking to a person with responsibility and authority, is for the edge cases.

There are sophisticated logistics systems that make it possible for deliveries to be specified in time increments smaller than "Tuesday". If you can transfer money in seconds, why does it take 2-5 days for a wire transfer to clear? These inefficiencies benefit the business so they are unlikely to change. Unless some bright spark wakes up to the idea that, if you stop pissing of your customers, then disruptive technologies and business models will have less of an impact on your business.

Internal systems can be even worse. This is where you really learn whether the company you works for both "gets" IT and cares about the people doing your job. At one previous employer, their internal systems were (and still are, by all accounts) hideous for the people doing the business of the firm. They would spend money on opening a new office in whatever corner of the third world the CEO thought he could pick up chicks, but wouldn't spend the pesos to develop a decent front end for time and expenses. This led to procrastination by the consultants, followed by regular brow beatings about getting your paperwork done so that bills could get generated and we could all get paid.

The connection was never made that if they just made it easier for us to do the right thing, we would. But by showing they didn't value it, we were happy to not value it as well. End result, bills being paid 90 or more days out from when the work was done. Not the best position to be if you are a growing professional services firm.

Financial systems get more scrutiny because of our capitalistic society. Were we more socialist, or at least humanitarian, we'd call out many more examples of software that, while technically capable of completing a task, makes one wish for a human and some paper.

What if OFT, OfCom and similar authorities stepped in and said, "Right, this is ridiculous and doesn't conform to any industry standard or best practice business process" and slapped people with a hefty fee. Or if Health and Safety looked at your internal systems and shut down them down: "I'm sorry sir, but that doesn't conform to any known UI guidelines; it's clearly not fit for purpose and is causing create stress amongst your workforce".

And that's just the visible wackiness inherent in the application code. The bigger design and architectural decisions, and policies put in place to protect an enterprise from itself, are much higher risk. I was on a plane recently that had to re-boot in an attempt to get the GPS system to kick in. Something had gone wrong during a recent upgrade and they never did get it working. We ended up flying without it. Wasn't sure if that was bemusement or concern in the pilot's voice.

So why does this happen? Why are we so bad at delivering software that adds to an experience?

Oh where to start...

Luck has nothing to do with it

As of 31 May, I am a free agent. I have been at my former employer for more than six years across 3 offices around the world, and it surprised more than a few that I would leave.

Announcements of leaving from others elicit a strange response in all of us. They appear random and arbitrary (much like Luck) and it sends ripples through our routine day to day world. Why is this person leaving? Is it for personal reasons? a better opportunity? do they know something I don't? am I missing out by staying? The seemingly arbitrary and random act, at odds with the messages we tell ourselves and receive from our organisation, creates an unease among those remaining.

Of the reactions sent my way, many included expressions of "best of luck to you." Which, while appreciated and taken in the spirit intended, got me to thinking that - wait a minute - luck has nothing to do with it! Personally, I'm with Seneca: "Luck is what happens when preparation meets opportunity."

My leaving was planned and executed purposefully. And one of the deciding factors in leaving was to better control my own destiny. Of course there needs to be demand for my services, but starting 1 June I control supply. And more importantly, I control what, when and how I am supplying. [read this article then read the book]. I used to work for organisations like those described in the article. I want to get back to that feeling of creativity, diversity and fulfillment that comes from those environments.

Some may argue (some have argued with me already) that if I can create that environment at my new gig - an IT department for an investment bank - I can create it anywhere. The difference is that by changing status from a consultant at a firm to an independent, I have much greater control over the roles I take and which compromises I'm willing to accept. In this case, I am joining a particular team with particular parameters. And in 4 months, I can decide whether to stay or go.

At first the thought of all this freedom was rather daunting. I've worked a long time under the auspices of an organisation and the security blanket it provides. But I recognise that as a false sense of security. In the words of Neil Peart: "If you choose not to decide, you still have made a choice."

So I'm grabbing destiny by the reins. And while she can still throw me off, at least she's in my hands.

Donkey work

So I reckon everyone has a physical age, a chronological age, and an emotional age.

I also think we have a core age. This is the age where our visceral reactions and prejudices come from. Along the same lines as a "All I Really Need to Know I Learned in Kindergarten". That author's core age is 5.

I think mine is 12. That summer, I spent working on a glorified commune picking vegetables with a bunch of college students, and one bored housewife. I learned about sex, drugs and how to drive a clutch. Hanging around a diverse bunch of people who successfully combined having a good time with high esteem and working hard has shaped everything I've done since. I struggled for most of my 20s trying to find a career that would match that experience.

I knew I wasn't fit for office work - I have an extremely low threshold for busy work, and let's face it, there is a lot of busy work in the corporate world. The closest I came was working for a sports and fitness magazine in Atlanta. That was fun, but the pay sucked and there are only so many things you can write about aerobics. So ironically, I spend my life automating away other people's busy work through the magic of programming.

And I think that's what a lot of people involved in developing business applications don't get. The stuff we write is meant to ease someone's burden, not to shift it to some other unsuspecting dupe. It's not about data integrity, or risk or governance. Companies have to do that regardless. And they are doing it already (or at least ought to be).

Why write an application that shifts the burden for this donkey work from the people who care about this stuff (back office staff and managers, usually) to the people who are creating the stuff that needs to be administered and managed? It's still donkey work. What you are telling people is that "I hold the managing and administering in higher regard than the creative output".

At least the original group cared about the work. Handing off work that even people who care about the results but still don't want to do, to people who don't even care about the results is fundamentally messed up. So what invariably happens is that a lot of energy is spent getting the new group to care about the work.

Ain't gonna happen.

As an example, I've done some work in the City, where traders are notorious for having a complete fuck-you attitude towards any busy work. And to them, busy work is anything that is not making them money. So they prefer the ad hoc, scribble it down approach. Some get enthused because spreadsheets and similar let them hook into all kinds of data and run scenarios - "Hey, I can make more money using this". But that's as far as it goes.

Now, their companies want them to input trade information into bespoke back office systems so that it can be automated and processed and governed. People on the floor don't care. They were hired to make money, for the bank and themselves. Using any system that is more onerous than what they already do just gets in the way of why they are there. End result? System gets ignored, or they find specious things to claim the system is unusable and go back to the old ways. £$€s spent for nada.

Given that computers are the donkeys of the information age, why not let the donkeys do all the donkey work? So when the donkey gets to me, all I have to do is load or unload it. I don't want to load the entire donkey. I want to add this bit. And I don't even want to do that. I want to hand it to the donkey and let the donkey sort it out, including how to add it to its pack. Computer age donkeys (ie. computers) are quite capable of taking something and munging it to whatever shape it needs to be in.

Forcing me to give the donkey my stuff in a way and by means that makes sense to the donkey, as opposed to the ways and means that makes sense to me is wasting both mine and the donkey's capabilities.

Funny thing is, I spent my 12th summer as a donkey, literally humping bushels and baskets around, and really enjoyed it. Largely, because we didn't have to do any donkey work.

Throwing myself in front of the train of opportunity

Something around "raining" and "pouring"....

In the last week or so, I've been selected to represent TW as a JCP expert for Java 6 (JSR-270), and Kane Mar and I have had our tutorial session "Wikis in Software Development" accepted for Wikisym at OOPSLA.

The java one is a little out of the blue, and I happened to luck into it, at least in the sense that a few people I work with could do it, but I happened to be the one asked. Kind of amusing, because at the moment I'm doing a lot of VB work. Compared to that, Java is near perfect. That being said, there are a few things I know I'd like to see in version next, but that's another post altogether.

The wiki/oopsla thing is more deliberate. I'm very interested in process and building successful delivery teams. This is my first foray into presenting at conferences. In fact we are both conference virgins. So of course we pick a small, backwater like OOPSLA to debut.

It will be a bit of a challenge to put this together, as Kane and I each have 40+ hour day jobs, and live on opposite sides of the world - I'm a Yank pretending to be British; he's a Kiwi pretending to be American. We've got a few months to sort it out. What could possibly go wrong.

Road trip

So early next year, I plan on driving from Fairfax, VA to Snohomish, WA. I've got a fast car, money in the bank and no deadlines. Who's with me?

As I've told a few people about this, I've gotten a lot of enthusiasm for joining in. Typical is Andy, who wants to know the dates for New Orleans. The most extreme reaction has been Mike, who is willing to pay to secure shotgun for the entire trip. I'm not sure even my by-then-wife would want to spend that much time with me. I'd prefer to be making this trip with her, but February is her high season. So instead she'll join me at points along the way.

My plan is to publish the itinerary, and welcome people to meet up at various places or come along for a leg or three. I'm taking the southern route; the only definites so far are New Orleans, Austin, Sante Fe and Vegas. And some time in the wilderness, probably Joshua Tree. I expect 3-4 weeks door to door.

This could get ugly, in a beautiful way.

Sore at the Apple Store

I'm getting married at the end of the year, and one of the seemingly infinite things that needs doing is registering.

I am sorely disappointed to discover that we (meaning I) can't register at the Apple Store.

Struggling a bit

I'm struggling a bit with my current project. Good people, agile process, new technology and new business domain.

I think it comes down to being hampered by lack of domain knowledge and working in new technologies. It's hard to lead from the front when you don't know enough to decide where to place the next step. And leading from the back can only get you so far. Especially on a software team where, more often than not, people are not naturally inclined to put themselves forward.

A few of us on the team are in this same boat, and no one is proficient in both the techs and the domain, and it's causing us to founder a bit. We spend too much time on process because it's something we can understand. We are still making are marks, but it gets more difficult as time goes on.

I'm naturally inclined to be decisive, and in the absence of anyone stepping up I continue to do so. But my reticence means I'm not firing on all cylinders, and I don't like that.