This post is a summary of where I’m at with thinking about the title question. I’m running an internal discussion on this topic later in the month.
In many software projects, coding skills are not essential to success. Most systems I’ve worked on involve simple, well-known functionality – sharing messages, automating workflows, making payments. These are all things people have been working on for decades but somehow projects still fail – despite advances in technology and methodologies. It’s not code that makes the difference.
For a while, I thought the issue was project resources – people trying to do things with too few people and too little time. But now I’ve come to think that the problems of maintaining a large code base for an organisation over years require skills very different to simply writing code. My thoughts on this have been focussed by reading Sean Goedecke’s essays, in particular Pure vs impure software development.
…pure engineering – is interested in solving a technical problem as perfectly as possible… The second kind – impure engineering – is interested in solving a real-world problem as efficiently as possible.
impure engineering is a brawl: you’re fighting decades of previous technical decisions, competing political views about how the product ought to work, consensus among your colleagues or the company at large, and in general much more incidental complexity.
This has led to me thinking about three questions:
Why is enterprise software development so slow?
What skills does it demand?
How can we get better at it?
Why is enterprise software development so slow?
In a discussion about AI project success, Goedecke quoted the statistic that 84% of non-AI projects fail. Indeed, “The infamous 2015 CHAOS report found an 61% failure rate, going up to 98% for “large, complex projects”. Most IT projects fail.
One explanation Goedecke offers is the the wicked feature, the sort of addition to software that affects every other feature – for example, user types or different deployment options. As software grows, such changes accumulate making it harder to add new features. Companies can try to resist such changes but eventually give in to them to unlock new markets.
Incidentally, this is why the quality of big tech engineering is sometimes worse than you would expect. The cognitive load of operating in this environment is pretty intense. Delivering anything at all is really, really difficult. So really obvious balls sometimes get dropped because the engineer in question was trying to thread a path between ten other features.
What skills does it require?
One skill Goedecke has pointed out is the ability to explain large and subtle pieces of software, providing what he refers to as ‘technical clarity’: “In an organization, technical clarity is when non-technical decision makers have a good-enough practical understanding of what changes they can make to their software systems.”
Goedecke also sets out shipping as a distinct skill to writing code: “The default state of a project is to not ship: to be delayed indefinitely, cancelled, or to go out half-baked and burst into flames. Projects do not ship automatically once all the code has been written or all the Jira tickets closed. They ship because someone takes up the difficult and delicate job of shipping them.”
Some others I would add:
Being able to track the work that needs doing.
Effective asynchronous collaboration, particularly with external teams
Running effective and efficient meetings
How do we get better at it?
I’ve only started thinking about this third question. My main concern is that most of the skills listed above are non-technical. I still think technical skills are important but, they are not sufficient. I’m looking forward to exploring this further.
How many open tickets do you have in your tracking software? And how many tickets were closed in the last six months?
In a lot of places I’ve worked, the relative size of these two numbers meant it would take years to clear all the open tickets. When the company has been going a long time, there are even sometimes tickets relating to issues that were fixed years before, often as the side-effect of another change.
These other tickets turn up in searches and need to be considered, if only in a small way, when new work is prioritised. While it’s useful to have long-term plans and roadmaps, I am not convinced that the ticketing system is the right place.
The more tickets you have, the more temptation there is to throw things in the backlog that need doing one day. Things get lost, and it’s hard to be certain you’re working on the most important thing. If something is not going to get done in the next year, there’s no point tracking it.
What about bugs and live issues? The best platforms I’ve worked on have had a zero-defect policy. If something is worth fixing, it should be fixed as soon as possible. If you’re prepared to live with it, and you’re not going to get round to fixing it any time soon… why bother recording it?
What about technical debt? If your technical debt is causing problems, you’ll be able to figure the most important thing to focus on. The rest of those things that you might get onto next year… Still might not be a bug enough problem to actually solve next year.
Massive backlogs are a weight we don’t need to carry. Tickets you’d love to do but won’t are clutter. They’re self-delusion.
Do some new year tidying up: delete the tickets you’re not going to work on, and focus on what you actually can do.
I spoke at Brighton Lean Agile last week, stepping in at short notice to cover for a friend. The event was great fun and I enjoyed the discussions I had with everyone. Rather than share the slides, I thought I’d do a quick blog post, allowing me to add some context to the bullet points.
Photo by @timrabjohns
Introduction
The talk discussed how agile doesn’t always work, and what can be done about it.
Overview
As Churchill put it: “it has been said that agile is the worst form of project methodology except for all those other forms that have been tried from time to time.”
While I might sound cynical about agile in what follows, I still believe in it. I think that the principles it promotes, such as regular reviews, tailoring methods to the team etc. are incredibly important. At the same time, a good waterfall project beats a bad agile one.
Agile is:
Simple
Flexible
Popular
While agile is detailed and subtle, there are core principles that are easily applied. It is flexible – there are accounts of it being applied to many industries and sizes of team. And it is popular – a large number of people have experience of it.
However, most companies face problems with the implementation, and even fundamental aspects. Every IT job ad I see these days has agile as a requirement. But listing something in a set of job requirements is not the same as understanding or implementing it.
Issues with the daily meeting
I believe in the principle that ‘the way you do anything is the way you do everything’. I’ve seen a lot of issues with the daily meeting, and feel that many wider issues with agile could be alleviated by getting this right. The stand-up is particularly useful for experimentation since it happens daily, and only lasts 15 minutes (in theory!). Note how ‘briefing scenes’ turn up in movies and TV shows a lot – this meeting should also be compelling, informative and entertaining. Problems seen include:
The stand-up is boring
Too many people in the stand-up
The stand-up is not relevant to the entire team,
The stand-up is at the wrong time
The stand-up shifts with a manager’s schedule
The stand-up becomes a reporting tool
Some developers think agile is silly
Some things I’ve tried
Balancing co-ordination vs social vs reporting
Adding initial reports to slack (particularly useful across timezones)
Changing who runs the standup (making team members feel more invested in agile)
Enforcing promptness
Track who is talking and who isn’t
Asking what we gain from it
Be high energy, be entertaining
Most of the points above are self-explanatory. I think it’s important to look at what an organisation needs from the daily meeting, and to focus on providing that without distraction or fluff (for example, if you do need people to report to management, find a more effective way to do that). The energy is also important – attendees need to think about their contribution and how to make it effective.
Estimation Problems
The estimates are already set (by a team member or management) and planning is a sham
Low expectations – developers aren’t pushed
No theme to the sprint
Team members don’t understand some stories (specialisations)
Nobody cares about the estimates – the code is produced late without any investigation
Translation is made between story points and time (worse when compared between teams or the translation being used is not explicit)
Estimates aren’t believed by management
Pressure is applied to reduce estimates: “But how long if you try harder…”
Estimates are given as self-protection – Padding or straight-out refusal to play
Punishing bad news
The team are told how, not what and why, meaning they estimate that method rather than considering wider goals
Boring
In the discussion we talked a little about how teams refusing to give estimates is usually a symptom of a larger problem. Another issue is estimates as negotiation rather than a best-effort prediction (ie padding for safety). Punishment for failing to hit previous estimates often leads to this sort of padding.
I concluded this section by saying that estimation sessions were something I often wished to cancel, possibly in favour of the person who was most informed/influential doing the estimates themselves. One participant said they had abandoned estimates in favour or sizing stories in a predictable manner.
One important thing to consider is whether the information in the estimates will change anything. If not (for example, schedule and scope are already set) then there may be little point spending time working out how long something will actually take.
Don’t want to do the role (the product owner is the most senior person who can’t avoid doing it)
Product owner has no power (proxy product owners)
Too much long-term planning – looking months ahead rather than supporting the team’s current work
Committing to futures that might not happen
Disconnect between stories and business
No idea how much a story costs to produce
Team composition shifting so you can’t learn anything
Technical debt is ever-growing
Multiple developer commitments outside project
Agile not adopted/understood/respected above the team level
The project is scheduled with no slack for developers being called away by emergencies.
Planning based on what people would like to happen, rather than taking note of what will happen
An example of this is ‘schedule compression’ – the deadline doesn’t move, but everyone resolves to try harder and be more efficient.
Not warning people about coming problems in the hope that someone else announces a serious issue first
Why aren’t Things Better?
I made two suggestions – the first was a reference to Sturgeon’s Law, which claims that “90% of everything is crap”. Maybe agile is difficult, everyone is doing their best, and we just have to be happy that things are working better than they would without agile.
My other suggestion was looking at cargo cults. This is a pejorative term in agile, but this betrays a lack of sympathy/understanding of the actual cargo cults.
Cargo cults were seen on a number of Polynesian islands, particularly after world war 2, where the residents of those islands built planes and airstrips from local materials, supposedly in the hope of cargo planes landing on the island. It’s given as an example of people imitating something without understanding the principles.
However, this idea that cargo cults are foolish is a racist and limited one. Indeed, the idea that they are about the arrival of cargo and that people actually expect planes to land are probably misapprehensions. For those involved, there are other reasons for setting out these airstrips. Western anthropologists have potentially had a blind-spot in their own obsession with monetary goods and missed these deeper motives.
In fact, these cargo cults are a success, if you understand why they have been set up. They are also effective simply on the terms for which they have mocked, since the cults have brought anthropologists and TV documentary crews to the islands in question. But to see why they succeed, you need to have sympathy/empathy with why cargo cults exist.
Sabotage
A lot of agile implementations seem to feature a sort of sabotage, where people involved do things that are directly against their best interests.
I closed by suggesting that, when an organisation is doing things that are obviously wrong and harmful to agile, there is often a deeper, more compelling system in place. This then formed the basis of a final discussion section.
It’s been a while since my last post here! Most of 2017 was taken up project managing a massive mobile project across ten countries – which came in on time and on budget. More recently, I’ve started a new company called Impaladev with Brad Legge, providing managed mobile and Java development teams.
As part of this, we’ve set up a workshop on Mobile Happiness. This condenses all the things we’ve learned about mobile delivery into a single hour (although many of the points are relevant to web and desktop applications). And we reveal the innermost secret of Project Management.
We ran a test of the workshop last night for a variety of ex-colleagues and it went very well – not least because we finished in time for that evening’s football match. Everyone came out of the workshop activities with new ideas for shaking up there projects.
We’re now looking for new venues and audiences for the talk. If your group or company might be interested, get in touch! Below is the obligatory corporate pointing-at-a-screen photo.
