Would we be better off without test environments?
I try not to recommend too many talks, but I loved this one from Nicky Wrightson of Skyscanner about living without pre-production environments. It provides an interesting solution to a lot of problems with performance environments I’ve been thinking through. Trying to accurately reproduce live environments seems like a wasteful, quixotic endeavour and I kept wondering about was just not using them. This video is an encouragement to that thinking.
Talks are often very different to the reality in companies, but there are some great questions for anyone about what test environments are for. “Historically, we had lower environments that were like production so that we could test releases that we couldn’t confidently reason about the effect of the changes.”
The only performance test that actually matters is how the live environment. ”At the end of the day, just make it a lot easier to track issues in production with really well instrumented code rather than try to replicate those issues in lower environments”.
There will always be inherent differences between environments, which limits the ability to draw conclusions from them. Yes, H2 is great for integration testing, but it has subtle differences with Oracle and MySQL which need managing. In performance tests, replicating load is hard – data and traffic shapes are as important as infrastructure and code. And then there are all the environment specific configs to manage, and the bugs from that.
Obviously, doing away with testing environments is the sort of decision that gets people fired. But! Just imagine how much things would improve if it worked!