Every system I’ve worked on has had scheduled jobs. Regular tasks need to be automated, particularly if they need to run in the middle of the night. Often these jobs are used for billing. For example, a job might need to:
- Find all customers who need subscription renewals
- Bill each customer
- Update that customer’s records, allowing them to continue accessing a service
- Email a billing confirmation
Writing a job like that is easy. The problem comes with remembering all the different things that might go wrong. Some developers are good at this, but others are optimistic, happy-go-lucky souls that never consider all the terrible things that might happen. I thought it would be useful to make a list of the sort of things I ask myself when thinking about a scheduled job:
- What happens if it fails to run? How do I find out it has failed (ie, where do we see the effects of the job not running?)
- What happens if there is a problem? Who gets notified? How do they know what to do next?
- If there is a problem with processing one of the records, does the job continue? What if every record is failing? Do we give up or keep going?
- If a run of the job doesn’t happen, does the job run as normal on the next execution? Does it catch up on the previously missed work? Should it?
- Can I run the job manually if I need it to? How should this be done? Who should be allowed to do it? How do they know when they should do this?
- What happens if the job runs more often than it should? What if it’s running once a minute rather than once a day?
- What happens if the job is taking too long? How can I tell if the job has failed, is paused, or is just taking a really long time?
- What if an execution of the job is still going on when it’s time for the next execution to begin? Can the different instances of the job interact safely? Does the job check whether it is already running?
- What happens if two instances of the job are running simultaneously?
- What happens if the job fails halfway? Can it be restarted safely?
- What happens if the data used by the job is changed by another process? For example if a user cancels their subscription after the job has started?
- What happens if one of the steps fails? Email and Billing are often third party systems. Which order should the events happen to ensure the safest failure?
- What is the worst possible thing this job could do if it were to go wrong?
Not all these steps are relevant to all jobs. Writing code to handle every eventuality can sometimes be more expensive than clearing up the mess when things go wrong. And, obviously, a lot of the issues are handled by frameworks. But it’s worth running through these sorts of questions before writing any code. Thinking about terrible things and knowing what will happen produces more robust code.