The art of software

I've long thought that development is much more of an art than many people realise. Even people within the industry treat development as something which needs to be specified and that developers are little more than typists who seem to make frequent spellig and grammer mistakes ;-)

It's a strange view, but lets face it - development is fantastically artistic. There are many, many different ways to create a CRM system, many ways to build an email component, thousands of ways to create a user interface page which captures information from a user.

And in the 'real' world, we accept that some tasks have an engineering part and an artistic part. Take Terminal 5 at Heathrow airport. It has some obvious (and substantial) engineering requirements - it needs to accommodate over 30 million passengers every year and all the accompanying baggage (literally). And yet we are quite prepared to accept the clearly artistic approach taken by the Richard Rogers Partnership who designed the building. It exposes many of the structural elements to the users of the terminal and, although very much in the eye of the beholder, is acclaimed for its artistic elements as well as the engineering which has gone into the construction. You can argue the same for many other buildings, bridges and even the odd consumer device of late. The 'how' is as important as the 'what' - not more important, not less, but equal.

So are we treating development the same way? I see little evidence - we continue to focus on the engineering and leave little room for artistic flair. And why should we? What is the point?

I wonder whether anyone ever asked da Vinci, Brunel, Richard Rogers or Norman Foster?

Planning & preparation versus agile

Many of the interesting agile discussions I have are with those whose job has an element of 'crystal ball' about it. No matter how crazy this might sound, this is what we try and do time and time again whilst delivering IT - we tell the customer how long it will take, we tell the customer how much it will cost. And in many cases we also attempt to document how systems will be structured in the years to come, the technologies they will use, the interactions they will have, etc. etc.

Of course, some of this is necessary - if you don't know where the ship is heading, you might as well not leave the port. So certainly we need an idea of whether a system is going to become legacy, whether it has a future, and what that future might look like. But it's important to control the amount of effort put into this kind of work, and the message which emerges.

An estimate given before work commences is just that - an estimate. It should be accurate enough to set expectations appropriately but at the end of the day, its only based upon the data we have at the time and our past experience. The same is true for any kind of strategic planning or design work - what is often overlooked is what we don't know. We don't often know the complexity which will arise out of the development of a particular component, we don't know that we'll find a tool on the web which can simplify our email template editor, we don't know that the business will cancel the project and complain that they've sunk £1.8M and received no benefit.

Lets get back to the primary goal of software development - the engineering of software which supports the business and allows it to operate efficiently. At its heart are the developers - the 'coalface' workers, and our responsibility is to minimise the work which isn't actually chipping coal from the coalface - everything else is less useful and unless tightly controlled is simply waste.

But we can't just go developing code randomly without a target in mind. And the customer is funding all of this and expecting some benefit, so they need to understand how long and how costly this will be - that's entirely understandable.

Agile can really help here - it is simply a matter of when and how to communicate. I think we would all agree that when a project kicks off, we need to get some brains in a room and discuss options. There's a lot of ground you can cover in a few hours with the right people. You can set the approximate direction, the tools you'll use, the objectives, etc. If you get the business involved we've seen huge leaps forward with gathering requirements (user stories) and success criteria, dependencies, etc.

So you have a couple of kickoff meetings. At this point you've spent next to nothing and should have a very rough idea of the size of the project (S-M-L-XL), the features it will deliver and the benefit. Enough to secure funding for a small team to execute a few sprints? If not, you're doing something wrong.

Once the team is up and running you'll have the experts on the case - developers will be able to get their hands dirty, the testers will be able to get to grips with any complexities, and we'll be deploying at the end of every 2 weeks so will understand what it takes to get the code released. In short, we'll touch every part of delivery and will build up experience.

After 3-4 sprints i'd recommend a major checkpoint to validate that the velocity is appropriate and burndown shows delivery at a suitable point. Now is the time to can the project, continue, or inject some investment (additional teams?) to potentially speed up delivery. The decision is in the hands of the customer but is based upon fact rather than crystal ball.

Strategic planning isn't irrelevant - it's the backbone on which the meat of development sits. But it supports development, rather than constraining it. Of more concern is the planning & prep work which we attempt on each project - we need to spend more time learning from development teams rather than second-guessing what they will do in the misguided hope that this somehow makes us more efficient. In many cases, the opposite is true.

Offshore development - why bother?

I've worked with many offshore companies over the years - some with significant operations in my home country, some based overseas with barely an account manager representative over here. Some a couple of hours flight away, some 6+ hours away in a different continent with only a few hours timezone overlap each day.

So why bother?

I suspect most initial interest (certainly from the UK) is around saving cost. The fact is that a typical Indian developer day rate is around 1/3 the cost of a typical UK developer. So you've got to ask the question - are we really saying that having a remote team is more than 3x less efficient than having a team on your doorstep? Surely we can save some money here?

Nobody really assesses efficiency in this way, so the question is actually quite difficult to answer. I'm not aware of any research which compares developer efficiency between an in-house developer and a developer next to him who has been provided by a third party. So lets take a developer on a journey ..

Step 1: Employ him through a third party

This one is relatively easy to calculate. From a line management point of view straight away i'm going to be employing contractors at a 30% premium over in-house permanent staff. Granted I might get some subject matter experts, but if I really want someone special, i'm going to pay more than 30% on top of permie rates, and it's unfair to paint permies as unskilled inexperienced resources - that's just not true.

Step 2: Add some layers of management

Ok, so we now have a more expensive resource, but we're going to give him or her a slightly different agenda to that which we would set ourselves. A percentage of his/her time will be spent filling in local timesheets, attending local team meetings, company briefings, training courses, or helping colleagues who have nothing to do with their primary client. I say 4 hours per week or 10% less efficiency. Of course we then have to pay for the management overhead - likely a share of a line manager - lets say 15% in total.

Step 3: Move him 4000 miles away

Go along the equator and this means for every day they work, they are disconnected from the rest of their team for around half of the working day. If I said you weren't allowed to discuss anything with anyone for 2.5 days per week, how less productive would you be? How much more time would you spend planning for those 2.5 days where you could communicate? Conservatively I reckon another 10% less efficiency - it could be more if there is a sizeable local team with whom he needs to communicate, but could be less if most of his communication is around him offshore.

On top of this, we now need to employ some local account managers and a small administrative team to keep the client happy - add another 5%.

Step 4: Change his culture

A tough one to assess, this one, but the impact ranges from language difficulty (ever agree something on a conference call only to discover that something else has been done a few days later?) to fundamental differences in behaviour to deal with the same situation. For example, the relatively hierarchical system in India naturally prevents a team from directly reporting failure if this is known to be undesirable. More likely, the team will do everything within its power to achieve, including enlisting more people to assist. On many occasions this will succeed (but will permanently affect the team behaviour in the future) but on other occasions it will fail, and fail seemingly without warning to the customer. In order to mitigate against this risk it would be reasonable to assume that at least 2 hours per week are dedicated to smoothing out these wrinkles, or 5% in real terms.

The total obvious inefficiency above? Around 65%.

That's not far off the 70% required to break even but you only need to get the rates wrong (lets make sure your offshore blended rate is < 30% of your onshore rate) or allow your offshore organisation to recruit candidates who don't actually compare like-for-like with your onshore staff, and you'll be the wrong side of the break-even line..

Introducing agile in a corporate environment

It's been a while now since we first started talking about agile in our organisation and we've come a long way since those first formative discussions around the number of sprints we'd require and how we'd clear this with the 'governance police' ! It's fair to say that a few things came together at that point which helped me push agile as a viable option -

1. The business were in the middle of a difficult and painful implementation process for a couple of large projects.
2. We had inadvertently employed a qualified scrum-master as one of our developers.
3. As a delivery organisation, we were considered expensive and slow.
4. An incoming CIO was a known fan of agile.

If so much as one of the above hadn't been the case, i'm not sure we'd have been in with a cat in hells chance of doing some work in an agile way. But as it happened, this created a 'sweet spot' where we were able to gather together some of the key stakeholders and gain approval for our first 'agile' delivery.

Some might say it wasn't agile at all - we already had business requirements and 75% complete high level/logical design documents, some of which would have included screen mockups. But we soldiered on - we organised a scrum team, planned a number of 2-week sprints (15 if I remember), created the product backlog and got started.

Unfortunately the project was canned after a few sprints due to funding issues, but in that short time we'd learnt an awful lot.

1. The business really enjoyed the regular demos  and were keen to attend.
2. We rattled through delivery at a fair old rate of knots.
3. We cost about the same as our existing part-offshore development.
4. Although we'd flattened the development team structure, everyone (without exception) loved working in this way.

It was a while before we got the opportunity to work in this way again but it was on a similar sized project - a single team of 6-9 people, anticipated to run for around 30 weeks. We are actually going live this weekend, so my enthusiasm may be a little premature but some interesting items of note:

• we are on time
• we are within budget
• we found and overcame some significant issues with the system which we were enhancing
• we have adapted to new requirements and discarded those which are no longer required

Ultimately we've seen the same sort of positive outputs from this project as with the previous and I see absolutely why this has happened. We have simply gone back to the basic 'get the right people together and let them get on with it' approach. And therein lies a little problem ..

If we find this approach works, what we're really saying is that all the huge governance and management structure around these teams, the layers of analysis, design and architecture which we put beforehand and the QA processes we stack aftwards are suddenly looking a bit overweight for this tiny little team in the middle who have just got on with things.

And if this team are so valuable why do we put them in a corner of the office and spend so little time listening to their complaints about the office environment, their lack of collaboration tools and the size of their monitors? Perhaps we ought to be putting these guys (and girls) first when we get some spare capex, perhaps we ought to be giving them the best desks in the office. And perhaps we should start to consider the value that us managers really add ..