Every system I’ve worked on has had scheduled jobs. Regular tasks need to be automated, particularly if they need to run in the middle of the night. Often these jobs are used for billing. For example, a job might need to:
Find all customers who need subscription renewals
Bill each customer
Update that customer’s records, allowing them to continue accessing a service
Email a billing confirmation
Writing a job like that is easy. The problem comes with remembering all the different things that might go wrong. Some developers are good at this, but others are optimistic, happy-go-lucky souls that never consider all the terrible things that might happen. I thought it would be useful to make a list of the sort of things I ask myself when thinking about a scheduled job:
What happens if it fails to run? How do I find out it has failed (ie, where do we see the effects of the job not running?)
What happens if there is a problem? Who gets notified? How do they know what to do next?
If there is a problem with processing one of the records, does the job continue? What if every record is failing? Do we give up or keep going?
If a run of the job doesn’t happen, does the job run as normal on the next execution? Does it catch up on the previously missed work? Should it?
Can I run the job manually if I need it to? How should this be done? Who should be allowed to do it? How do they know when they should do this?
What happens if the job runs more often than it should? What if it’s running once a minute rather than once a day?
What happens if the job is taking too long? How can I tell if the job has failed, is paused, or is just taking a really long time?
What if an execution of the job is still going on when it’s time for the next execution to begin? Can the different instances of the job interact safely? Does the job check whether it is already running?
What happens if two instances of the job are running simultaneously?
What happens if the job fails halfway? Can it be restarted safely?
What happens if the data used by the job is changed by another process? For example if a user cancels their subscription after the job has started?
What happens if one of the steps fails? Email and Billing are often third party systems. Which order should the events happen to ensure the safest failure?
What is the worst possible thing this job could do if it were to go wrong?
Not all these steps are relevant to all jobs. Writing code to handle every eventuality can sometimes be more expensive than clearing up the mess when things go wrong. And, obviously, a lot of the issues are handled by frameworks. But it’s worth running through these sorts of questions before writing any code. Thinking about terrible things and knowing what will happen produces more robust code.
I followed up my two days at µCon with the LJC Open Conference. The third early start in a row was made a little more difficult by a late night – I gave a talk in Eastbourne the night before – but the sessions kept me alert. Rather than µCon’s deep focus on one area, the Openconf was an opportunity to learn about many different things, with introductions to Spock and Cassandra, as well as a good discussion about Akka, actors and other methods of distributed computing.
I don’t have a use for Cassandra yet, but it was good to learn more. In his talk, Chris Batey made an interesting point that abstracting out underlying data storage architecture can be a bad thing, since it hides the particular nature of that layer. For example, in Cassandra it is important to be aware of the partition key’s meaning. The slides are online.
The lightning talks were excellent, including a spoof on devops and consulting by Daniel Bryant, who coined the term QAOps, as well as a report on mob programming. I also caught Daniel Bryant’s presentation on microservices, which I’d missed at MuCon, having chosen the other track.
It was a good day out and excellent value. I’m very glad that I was able to go.
The second day of µCon proved to be as interesting as the first. I have another huge list of things to investigate, and there are some things I need to get underway at work tomorrow. As I said in my post about day 1, the most important thing about microservices is, for me, the discussion of the problems and patterns of modern web-development. Pretending that monolithic applications work is no longer an option.
The relationship of microservices to SOA came up several times, with people questioning whether this was just a re-branding or rehabilitation of SOA.
Another good question emerging from microservices is why we still can’t produce satisfactory architecture after all this time (Conway’s Law dates back to 1968). We have a cycle of new techniques becoming common, then standardised, then growing bloated and cluttered until someone takes a clean-sheet approach and we start again.
The concept of cohesion is one that I haven’t paid enough attention to; (micro)services should be ‘loosely coupled, highly cohesive’. It was also good to be reminded that logical and physical architecture do not need to be the same. I don’t think we pay enough attention to things as simple as package structure.
Jeppe Cramon pointed out that “a service that only deals with data is a database,” and we already have APIs for that. Why wrap that in REST?
One thing that’s difficult for me personally is getting out of the database transaction mindset. Cross-service transactions are a bad thing, but are likely to become prevalent as large software providers make it easier for people to do things the wrong way.
Jakub Korab had an important lesson for developers working with monoliths: if you touch any external system then you’re working with distributed computing. Monolithic architecture is an assumption that has worked (more-or-less) for a long time, but it’s not really true. All developers should have an awareness of the CAP framework and the Fallacies of Distributed Computing.
The only major problem I had with µCon was the lack of female presenters. If a subject has no female practitioners with something to say, then the topic may not be ready for a conference. Otherwise it was an excellent two days. I’ve already booked a ticket for the 2015 event.
I spent today in London, attending the first day of µCon, a conference dedicated to microservices. A lot of delegates seemed to be in the same position as me – working with a monolithic application that needs separating. Microservices are certainly not the answer to my problems, but many of the techniques and architectures will help me. The most relevant session was Chipping Away at a Monolith by Mark Landeryou and Steve McDonald from SagePay, which gave me a lot to think about.
Some other immediate responses:
There was a great deal of pragmatism. As enthusiastic as people were about microservices, there was an air of caution rather than triumphalism. This is a good solution in some situations but certainly not a silver bullet.
Many microservice architectures have focussed on HTTP/REST, but some of the other options (pub/sub messaging, thrift) are interesting.
Conway’s Law is a constant reference point, and I need to sit down and read the original paper. I find myself wondering recently why we even organise programmers into IT departments rather than place them in the business where they are needed.
The Unix Philosophy came up a lot too, with one speaker describing Unix pipes as the most perfect API ever invented.
My main concern with microservices is that they are an ideal. Most people have good intentions but still somehow find themselves dealing with difficult codebases. Greenfield development is rare and temporary. Maybe we need conferences focussed on monolith programming – but I expect that would be a hard sell
Another early start tomorrow for day 2. It’s going to take me a few days to read through the notes I made properly (21 pages so far), but I’m looking forward to another day of talks.
I’ve been writing drafts of technical posts recently. I was going to put them on my personal site, but they didn’t seem to fit there. So I decided to start a whole new site for my technical writing.
Over the last year or so I’ve been playing with a lot of fascinating technology: microservices, Spring boot, AWS, Jenkins, REST and so on. For a while I’d been bored by programming but I now feel enthusiasm again. I’m full of ideas for things to learn and services that I want to build, and wanted a space where I could write about what I’d learned and what I was thinking.
I’m off to µCon on Thursday, then the LJC Open Conference on Saturday, both of which I’m sure will give me a lot to write about. There are various new technologies I want to play with, like Vagrant. I’ve been thinking about how the lessons from microservices apply to the world of monoliths. Programming has become massively more complicated over the years I’ve been doing it, but that is starting to feel exciting.