Month: January 2019

Categories
programming

The TDD Lie

The requirements for pretty-much every developer job I see these days includes Test Driven Development. Which is exciting – everyone recognises the importance of unit tests.

In my previous post I talked about how often companies claim to do agile, but don’t succeed in practise. And I see the same thing happening with test-driven development.

It’s easy to tell which companies want to use TDD but don’t in practise: you ask what percentage coverage they have for their code.

In its purest sense, TDD demands that tests are written before any new code is written. If you’re coding in this way (as suggested in  Kent Beck’s Test Driven Development book) your coverage should be close to 100%. Of course, there are other ways one might approach TDD, such as writing business tests before the code. This is definitely better than adding a few tests at the end of a project.

But, if you’re tracking the coverage and have chosen a figure less than 100%, then there is a choice being made about which code is not covered. I’ve never seen a standard coverage threshold working out well. The obvious indication  of this is how often the coverage of projects is just over the required threshold.

I’m a great believer in high coverage (while acknowledging the limitations of that). And if you don’t grow your code with the  tests then you tend to end up with code where it’s difficult to add code at the end.

If you’re listing  TDD in job requirements and not practising it, then it’s worth asking why not. A lot of times it’s a fundamental limitation to the company – we know we should have tests but we’re rushing. It’s the same problem that many companies have with agile – these things are obviously a good idea, but difficult to do in practise.

As Bob Martin pointed out,”You know what you believe by observing yourself in a crisis. If in a crisis you follow your disciplines, then you truly believe in those disciplines. On the other hand, if you change your behaviour in a crisis, then you don’t truly believe in your normal behaviour” (from Chapter 11 in the Clean Coder).

If you know you should be using TDD but aren’t, then you need to ask why not. And, if you can’t fix that, stop lying that you’re doing TDD – admit what you’re actually doing and focus on improving that.

Categories
projectmanagement

“We’re agile, but…” talk notes

I spoke at Brighton Lean Agile last week, stepping in at short notice to cover for a friend. The event was great fun and I enjoyed the discussions I had with everyone. Rather than share the slides, I thought I’d do a quick blog post, allowing me to add some context to the bullet points.

Photo by @timrabjohns

Introduction

The talk discussed how agile doesn’t always work, and what can be done about it.

Overview

As Churchill put it: “it has been said that agile is the worst form of project methodology except for all those other forms that have been tried from time to time.”

While I might sound cynical about agile in what follows, I still believe in it. I think that the principles it promotes, such as regular reviews, tailoring methods to the team etc. are incredibly important. At the same time, a good waterfall project beats a bad agile one.

Agile is:

  • Simple
  • Flexible
  • Popular

While agile is detailed and subtle, there are core principles that are easily applied. It is flexible – there are accounts of it being applied to many industries and sizes of team. And it is popular – a large number of people have experience of it.

However, most companies face problems with the implementation, and even fundamental aspects. Every IT job ad I see these days has agile as a requirement. But listing something in a set of job requirements is not the same as understanding or implementing it.

Issues with the daily meeting

I believe in the principle that ‘the way you do anything is the way you do everything’. I’ve seen a lot of issues with the daily meeting, and feel that many wider issues with agile could be alleviated by getting this right. The stand-up is particularly useful for experimentation since it happens daily, and only lasts 15 minutes (in theory!). Note how ‘briefing scenes’ turn up in movies and TV shows a lot – this meeting should also be compelling, informative and entertaining. Problems seen include:

  • The stand-up is boring
  • Too many people in the stand-up
  • The stand-up is not relevant to the entire team,
  • The stand-up is at the wrong time
  • The stand-up shifts with a manager’s schedule
  • The stand-up becomes a reporting tool
  • Some developers think agile is silly

Some things I’ve tried

  • Balancing co-ordination vs social vs reporting
  • Adding initial reports to slack (particularly useful across timezones)
  • Changing who runs the standup (making team members feel more invested in agile)
  • Enforcing promptness
  • Track who is talking and who isn’t
  • Asking what we gain from it
  • Be high energy, be entertaining

Most of the points above are self-explanatory. I think it’s important to look at what an organisation needs from the daily meeting, and to focus on providing that without distraction or fluff (for example, if you do need people to report to management, find a more effective way to do that). The energy is also important – attendees need to think about their contribution and how to make it effective.

Estimation Problems

  • The estimates are already set (by a team member or management) and planning is a sham
  • Low expectations – developers aren’t pushed
  • No theme to the sprint
  • Team members don’t understand some stories (specialisations)
  • Nobody cares about the estimates – the code is produced late without any investigation
  • Translation is made between story points and time (worse when compared between teams or the translation being used is not explicit)
  • Estimates aren’t believed by management
  • Pressure is applied to reduce estimates: “But how long if you try harder…”
  • Estimates are given as self-protection – Padding or straight-out refusal to play
  • Punishing bad news
  • The team are told how, not what and why, meaning they estimate that method rather than considering wider goals
  • Boring

In the discussion we talked a little about how teams refusing to give estimates is usually a symptom of a larger problem. Another issue is estimates as negotiation rather than a best-effort prediction (ie padding for safety). Punishment for failing to hit previous estimates often leads to this sort of padding.

I concluded this section by saying that estimation sessions were something I often wished to cancel, possibly in favour of the person who was most informed/influential doing the estimates themselves. One participant said they had abandoned estimates in favour or sizing stories in a predictable manner.

One important thing to consider is whether the information in the estimates will change anything. If not (for example, schedule and scope are already set) then there may be little point spending time working out how long something will actually take.

When agile doesn’t fit the business

  • External factors (deadlines, releases, tradition) restrict agile
  • Product owner issues:
    • No time to do the role
    • Don’t want to do the role (the product owner is the most senior person who can’t avoid doing it)
    • Product owner has no power (proxy product owners)
  • Too much long-term planning – looking months ahead rather than supporting the team’s current work
  • Committing to futures that might not happen
  • Disconnect between stories and business
  • No idea how much a story costs to produce
  • Team composition shifting so you can’t learn anything
  • Technical debt is ever-growing
  • Multiple developer commitments outside project
  • Agile not adopted/understood/respected above the team level
  • The project is scheduled with no slack for developers being called away by emergencies.
  • Planning based on what people would like to happen, rather than taking note of what will happen
    • An example of this is ‘schedule compression’ – the deadline doesn’t move, but everyone resolves to try harder and be more efficient.
  • Not warning people about coming problems in the hope that someone else announces a serious issue first

Why aren’t Things Better?

I made two suggestions – the first was a reference to Sturgeon’s Law, which claims that “90% of everything is crap”. Maybe agile is difficult, everyone is doing their best, and we just have to be happy that things are working better than they would without agile.

My other suggestion was looking at cargo cults. This is a pejorative term in agile, but this betrays a lack of sympathy/understanding of the actual cargo cults.

Cargo cults were seen on a number of Polynesian islands, particularly after world war 2, where the residents of those islands built planes and airstrips from local materials, supposedly in the hope of cargo planes landing on the island. It’s given as an example of people imitating something without understanding the principles.

However, this idea that cargo cults are foolish is a racist and limited one. Indeed, the idea that they are about the arrival of cargo and that people actually expect planes to land are probably misapprehensions. For those involved, there are other reasons for setting out these airstrips. Western anthropologists have potentially had a blind-spot in their own obsession with monetary goods and missed these deeper motives.

In fact, these cargo cults are a success, if you understand why they have been set up. They are also effective simply on the terms for which they have mocked, since the cults have brought anthropologists and TV documentary crews to the islands in question. But to see why they succeed, you need to have sympathy/empathy with why cargo cults exist.

Sabotage

A lot of agile implementations seem to feature a sort of sabotage, where people involved do things that are directly against their best interests.

I closed by suggesting that, when an organisation is doing things that are obviously wrong and harmful to agile, there is often a deeper, more compelling system in place. This then formed the basis of a final discussion section.

Categories
programming

How to Build a Webapp in Java

tldr; building a webapp is complicated – particularly if you’re doing it right. JHipster offers a promising solution.

I was a little shocked recently when I realised that I didn’t know how to make a web-application.

It’s not like I’m inexperienced. I’ve been working on the web since 2000; but my job involves focussing on specific areas. Even when helping with strategy, I have very little input into the front-end. Being responsible for an application going live and doing it all myself are very different things.

Part of the problem is how intimidating front end development has become, with flurries of Javascript frameworks, UI libraries and strange pre-compilers. Angular has two versions that are so different as to be incompatible, but with similar names. Tutorials date quickly and beginners can’t tell if they are current. There is a huge amount of knowledge needed to build a web application.

Now, obviously, as the project website says, Spring boot “makes it easy to create stand-alone, production-grade Spring based applications that you can ‘just run’“.  My initial impulse to cobble something together from Spring and Thymeleaf would allow me to get something live. But this is an unsatisfying solution. Server-generated template pages don’t feel particularly powerful even with a good MVC framework. They have several problems:

  1. They tend to have code for the view tangled up with the application, with no clear boundary
  2. Coupling the templates to the server-side code misses the separation of architectures like REST. Adding, say, a mobile client would involve a lot of hassle to get the two different front ends working.
  3. More modern architectures are available for front-end. Using REST and angular allows rapid development of flexible front ends, less coupled to the backend architecture.

Another question that arose was user management and security. This is one of those things that every site needs, but tends to be tedious to code (as well as inviting errors). Any decent user management system needs account creation, password resetting, different roles etc. For most projects, these are going to be the same. My basic assumption would be that Spring security would be useful for that, but that is really a framework for creating a security framework, and there are no obvious mature implementations. Setting up email account confirmation is a drag.

I weighed up a few options for authentication, particularly since I wanted one that supported a stateless web-server. After speaking to a friend, JWT seemed to be the way to go, but this still seemed like a lot of work. And I’d not even started grappling with the whole front-end stack.

I started working in a new office recently and a colleague suggested JHipster as a solution to my problems. I’d heard about the technology via InfoQ but hadn’t paid it a lot of attention. Part of this was the name, which sounded insincere and jokey. And my cursory look suggested it was ‘just’ angular with spring boot. I’d missed something very interesting.

According to the JHipster mini-book, JHipster allows Java devs to “be hip again”. The knowing irony of the framework’s name is irritating, but this idea of being ‘hip’ is good. Web development has changed a great deal, and JHipster provides a simple way to get started with up-to-date technologies, combining bootstrap, angular (or react) and Spring boot. The project started back in October 2013, with the first release in December of that year.

JHipster is similar to Ruby on Rails, providing rapid development of a stack based upon entity definitions. The ‘Hipster’ part of the name is referring to the need to keep up with the modern tools. JHipster produces the plumbing and boilerplate for a whole stack including a variety of best of breed tools. It also adds in a basic user-management system for free. JHipster prepares multiple files for each entity:

  • Database tables
  • Liquibase changes
  • JPA entity definitions
  • Spring data repository
  • Spring rest controller
  • Angular route, controller and service
  • HTML page

The best way to get a flavour of how JHipster works it to look at Matt Raible’s 16-minute demo of building a blog application, get started with JHipster 5.

Full-stack development is hard work, even with JHipster – you need to be familiar with both Spring and angular. But this does put you a lot closer than ever before to putting live a full-stack application using best of breed technology.

Categories
Uncategorized

speaking at Brighton Lean Agile on 9th jan 2019

Due to a last-minute drop-out, I’m speaking at Brighton Lean Agile on January 9th. The title of the talk is “We’re agile, but…” and looks at some of the ways I’ve seen agile going astray in implementations.

Despite the simplicity and popularity of Agile, it often goes wrong in practise. Drawing on 15 years’ experience of Agile as developer and PM, James Burt will lead a discussion on why agile sometimes fails, and what can be done about it.

The session is currently full but there is a waiting list, as people sometimes drop out as the event approaches. Refreshments are provided by the sponsor, Bright Interactive.

Categories
books programming

Book review: Beyond Java by Bruce A Tate

Tldr; things looked bleak for Java in the mid-noughties, but the language and eco-system have undergone massive changes, allowing Java to thrive and survive.

My new job has a daily commute, which means time to read. I’m focussing on tech books and recently picked Bruce A Tate’s Beyond Java. I originally read this in July 2006, the year after it was published. The book described why Java was doomed, with Tate saying in the introduction, “Java’s reign will end. It’s not a question of if but when.” (pviii). While the predictions have turned out to be wrong it is interesting to re-read the book and see how Java escaped its fate.

Revisiting the mid-noughties was fascinating in itself: a time when AJAX was just emerging, Web 2.0 was the next big thing and folksonomies would change everything. It was an environment that caused obvious problems for Java. Its flaky text-processing was just no good for large XML files, and Java was forced to deal with a lot of XML. In addition, Java tended to be used for the middleware on top of databases, which it was not particularly suited for (p52).

Even with early versions of Spring and hibernate, making Java applications was a chore, with overwhelming amounts of boilerplate and XML config. Even with ORMs, Java required you to work with both entities and database code, and to keep them consistent.

Tate talked about this difficulty in getting software running, and all the tools that were needed. Compared with PHP, Java felt slow to develop in, and its advantages weren’t sufficient compensation. It did seem that Java was on the ropes. I remember dropping out of Java to do simple text tasks. Ruby-on-rails and similar frameworks promised the chance to get applications running quicker.

The book is a pessimistic one. Tate saw the problems as unsolvable and, given Java’s history and trajectory, this was a fair assumption to make. Tate looked at Java critically and suggested other platforms developers should investigate. Reading the book with hindsight, one can see Tate’s predictions have failed, but the discussion is still interesting.

In the 12-or-so years since Tate was writing, Java has managed a comeback. Decent dependency management from maven and gradle has eradicated the nightmare of setting classpaths. The language has been made more expressive with new features, such as the functional constructs in Java 8. And the growing fashion for opinionated configuration has also made things easier – getting Spring Boot running is incredibly easy. The culture of the language has changed for the better too. In the past, checked exceptions proliferated everywhere, and now people tend more towards throwing runtime exceptions and handling everything at a single sensible point.

One particularly strong improvement is the Spring framework. At the time Tate was writing, Spring was a mess of XML configuration and cryptic errors. With annotations, it has become easy to use and unobtrusive. New platforms such as Spring Boot allow developers to release production-ready features out quickly.  At the same time, Java/Spring offers a mature platform, where code can be taken to production and supported at massive scale. Ruby-on-rail’s promises have faltered, with Twitter famously moving to Java because of scaling issues. The XML problem has disappeared as JSON takes over as a less robust, easier to use standard. I don’t miss the days of choosing between SAX and DOM parsers for XML.

Around the time Tate was writing, a debate raged about variable typing: strong vs weak, static vs dynamic. Java’s static and strong typing felt too rigid for many people. This debate seems to have reached an interesting conclusion: Javascript has been restricted by typescript while Java 10 has introduced the var keyword.

(I’ve always seen the inflexibility of Java as a positive. While it’s less expressive than languages such as python or, um, perl, the code it produces tends to look very similar. It’s a good language for non-rock stars – who massively outnumber the sort of people capable of producing cryptic code given the slightest chance).

At the time of writing, Tate saw Java’s fate as connected with that of Sun. Tate wondered what might happen if Sun disappeared, wondering who might take over Java. The resulting situation, with Oracle running Java (and MySQL!) was hard to predict but has worked out well – even with the occasional fake news scare that Oracle would start charging for Java.

Tate discussed a number of contenders for Java’s throne. Perl has lost popularity over the last decade, with Python offering a more structured (and readable) alternative. Perl, like PHP, was a great tool for its time but ultimately a ladder to better things. The book makes a quick mention of functional languages: Clojure seems to have come and gone, but  Scala and Kotlin (first released in 2011) seem to be gaining traction. Ruby itself is much less popular. Java seems to have been saved by the flexibility of the JVM, which offers more powerful options for those that need them.

For me, the most interesting discussion was around state in web-servers, and the problems of that. Tate talks about continuation servers as an upcoming opportunity. This is another debate that has been settled, with REST winning out over stateful alternatives. I worked with Seam for a time – it was an interesting idea, but so alien to the way most developers worked, that its subtleties tended to be fought with rather being  made use of.

One problem that Tate raised which is still present is the approachability of Java. Even in the innocent days of 2005, the simplest useful Java application required use of a large number of tools. This issue is present in a lot of fields, including front-end dev work, where increasing sophistication has made things much more complicated. There are solutions to this, with JHipster being a promising one, but these work to hide the complexity. Just because the tools are set up for you doesn’t mean they aren’t there and can’t go wrong in strange and unpredictable ways.

EDIT: Tom Hume pointed out  that the discussion above doesn’t include Android, which was originally released in 2008 and has become hugely popular, with over 2 billion users. As someone who  focuses on backend dev, I didn’t engage with this, but Android has been contributed significantly to Java’s survival. Having  said that, Kotlin is becoming more popular in Android, threatening Java’s dominance on that platform. Interestingly, Kotlin is failing to see significant uptake in backend dev, despite recently becoming compatible with Spring.

If I had more time, I’d also look at the JVM platform as distinct from Java. While Groovy and Clojure have faltered, Scala (launched 2004) and Kotlin (launched 2011) are both popular JVM languages