Author: admin

Categories
openstreetmap

Using postgis on docker to Find Ley-Lines

Back in 2015, I wrote a blog post about finding ley-lines with postgis, based on some work by Steven Kay.

I described the outline of how to produce as leylines as a How-to-Draw-An-Owl tutorial, since I skipped over a lot of the complex points. I was setting up my temporary postgis instances on AWS and there was a certain amount of faff involved in sorting out database access and structure. This post is an improvement on those methods.

(There are still bits I’m not explaining in full – basically, how to display the ley line – but if you need more details, please let a comment and I’ll get in touch. But, basically, you need a little bit of HTML/Javascript to take this all the way through)

I’ve returned to this project for some recent writing, and the technical aspects are much easier using docker. In fact, the process can be broken down to seven steps, which I’ve listed below. You’ll need to be comfortable with a command line, have docker installed, and be willing to play about a little. But the process is relatively straightforward:

  1. Start up a postgis instance from a computer with docker installed:
docker run --name some-postgis -e POSTGRES_PASSWORD=mysecret -d postgis/postgis

2. Open a terminal in the docker instance:

docker exec -it some-postgis /bin/bash

3. From inside the docker instance, install the OSM tools

apt update
apt install osm2pgsql

4. From the host computer, download some OSM data and copy the file into the container. There are various options for downloading this data but I tend to use the Geofabrik server. 

docker cp ~/east-sussex-latest.osm.pbf some-postgis:postgres

Inside the container, the OSM file can then be loaded into postGIS using the command

osm2pgsql merseyside-latest.osm.pbf -U postgres

5. Back to our terminal inside the host, start up postgres

psql -U postgres

6. Now, here is the bit where I send you to another tutorial to get the ley lines SQL. These have been made available as a github gist by Steven Kay. Cutting and pasting the commands will get a fairly decent result, but I recommend having a play to get the best results.

7. You can now use another query to get the KML (Keyhole Markup Representation) format of the lines. This format is fairly portable, and I tend to plonk it in a simple HTML page. The larger the “ct” value, the more items are connected by the ley.

SELECT (ST_AsKML(geom)) as geom from leys where ct = 7;

8. And this is the hand-waving step, which is to display your KML lines on an actual map. Basically, you need to amend a simple OpenLayers example. Basically, if that leaves you stuck, leave a comment and I’ll get back to you (commenter email addresses are not be visible on the actual site).

The script from Steven Kay is great for producing candidate lines. The only problem is that it tends to follow clusters of pubs on the same street. The restriction that the lines must begin and end at a site while being at least 20km long prevents short local lines being plotted. But it would also be good to add an extra predicate to judge lines by how spaced-out the locations are. That’s something I need to work on in future.

Categories
silicon-brighton

The Story Of Silicon Roundabout

I’ve written before about the failure to create a Silicon Brighton, despite both a vibrant community and significant investment. While the whole country has tried for years to create technology innovation clusters, the most successful example is “Silicon Roundabout”.

The name Silicon Roundabout came from a jokey tweet by Matt Biddulph (who formerly worked at Dopplr and Moo), back in July 2008: “Silicon Roundabout: the ever-growing community of fun startups in London’s Old Street area“. This off-hand tweet led to a small flurry of press, as well as an article in Wired UK.

Enough time has passed that there are in depth reports on Silicon Roundabout (from the LSE’s Center for Economic Performance), and Wired UK has even published an obituary.

As Max Nathan, one of the LSE report writers, summarised on Twitter, “David Cameron launched the Tech City programme in 2010 to ‘accelerate’ the Shoreditch cluster – using place branding, business support, networking, tax breaks and a new delivery agency. By 2014, Mayor of London Boris Johnson was hailing it a huge success.”

That Tech City programme occurred significantly after the Silicon Roundabout Cluster had been identified. Nathan questioned whether the Cameron intervention had any significant effect: “Instead of catalysing the cluster, policy generally rode the wave.” (ref). The discussion on twitter concludes:

Where does this leave us? None of this takes away from the real and impressive growth in London’s overall tech ecosystem… However ~ this work challenges the backstory. Many of the big claims made for the Tech City programme in East London don’t really stack up. And in the paper I find some evidence of negative side effects – for example, spatial disruption and movement within cluster space…. On the one hand, it’s impressive that a tactical, light-touch, cheap programme like this had any impact. Though I’m not sure this approach would work outside London.

Wired’s obituary also looks at the way in which the Silicon Roundabout ended up as “a real estate nightmare”. Low rents were one fo the attractions of the area, but the branding had an effect on this. The article admits that :

the Silicon Roundabout cluster is still there – and it is bigger than ever. Ironically fulfilling Cameron’s prophecy, it has crawled all the way east, as people ran away from rising property prices. In merely spatial terms – how big the cluster has grown – the policy has been a success

But:

With many more firms scrambling to move near the famed Old Street roundabout and rents duly skyrocketing – the initial benefits of the cluster vanished. Many companies, especially pure tech startups ended up being overwhelmed and closing shop; others left very quickly; the environment overheated and churn accelerated, destroying any semblance of a stable network of cooperation. “For digital tech firm productivity, I find suggestive evidence of a negative policy effect,” Nathan’s paper reads. “This suggests that policy weakened the net benefits of cluster location.”

What does this mean for Brighton? We’ve had several Silicon Initiatives and the Digital catapult. Why are we not a ‘Tech City’? The story of Silicon Roundabout suggests that policy interventions are not going to help as much as we think.

Categories
brightonjava

An Introduction to FXGL

Last week, Brighton Java had a session from Almas Baimagambetov of Brighton University, who introduced the FXGL game library. The session featured three introductory videos and some discussions . I’ve embedded the three videos at the bottom of this post.

What was most exciting to me was how quickly results could be produced. One of the problems with modern Java is how hard it is for beginners to produce tangible outputs. There are also a host of examples available that can be customised.

The format, with three videos and discussion in between worked well. And, while it wasn’t the focus of the talk, it was also useful to see some of the native packaging options for Java.

I’m passionate about the importance of Brighton Java as a local group, enabling people to communicate with nearby colleagues. However, moving online has enabled people from further afield to attend, and we had guests from Honduras and Senegal. Even as we move back to in-person events, I hope we can maintain access to the group for people outside Brighton.

Almas is scheduled to give a longer version of this talk at at FOSDEM21

https://www.youtube.com/watch?v=8alLIUHnJC0

Categories
programming

The Seven States of a Boolean Object

Everyone understands Boolean variables – they’re either true or they’re false. Right? Except, in languages, where the Boolean variable might be null, which makes comparing two Boolean values a little little trickier. And the DailyWTF once gave an example of a Boolean ENUM that could be true, false or file-not-found.

Long ago, when I worked for Tom Hume at Future Platforms, one of the developers suggested something more ambitious. After all, a three-state Boolean leaves too much room for ambiguity. By the time he was done, Thom Hopper had suggested 6 or 7 states for a Boolean variable. Tom Hume and I recently tried to remember as many as we could, but only got 5 or 6.

We tweeted Thom Hopper to see if he remembered, but the states of this variable are lost in time, like tears in rain. But, for the sake of posterity, I want to record some of the values that a proper Boolean type might hold:

• true
• false (obviously)
• null
• uninitialised (similar null – but this means a true/false has never been set)
• unknown (we’re currently not certain of the value)
• indeterminate (there’s no way of knowing this value)

I’ve love to be able to declare that this is stupid, but I don’t know. I mean, if you accept null as a Boolean state, where do you stop? Maybe Java needs a ‘true’ boolean that can take all of these states?

Categories
infrastructure programming

Living Without Pre-Production Environments

Would we be better off without test environments?

I try not to recommend too many talks, but I loved this one from Nicky Wrightson of Skyscanner about living without pre-production environments. It provides an interesting solution to a lot of problems with performance environments I’ve been thinking through. Trying to accurately reproduce live environments seems like a wasteful, quixotic endeavour and I kept wondering about was just not using them. This video is an encouragement to that thinking.

Talks are often very different to the reality in companies, but there are some great questions for anyone about what test environments are for. “Historically, we had lower environments that were like production so that we could test releases that we couldn’t confidently reason about the effect of the changes.”

The only performance test that actually matters is how the live environment. ”At the end of the day, just make it a lot easier to track issues in production with really well instrumented code rather than try to replicate those issues in lower environments”.

There will always be inherent differences between environments, which limits the ability to draw conclusions from them. Yes, H2 is great for integration testing, but it has subtle differences with Oracle and MySQL which need managing. In performance tests, replicating load is hard – data and traffic shapes are as important as infrastructure and code. And then there are all the environment specific configs to manage, and the bugs from that.

Obviously, doing away with testing environments is the sort of decision that gets people fired. But! Just imagine how much things would improve if it worked!

Categories
infrastructure programming

What’s So Special About Cloud Software?

One of the more interesting interview question I’ve been asked in the past few years was about the differences between cloud development and monoliths. I don’t think I gave the expected answer when I said that they’re not all that different

Yes, cloud environments are complicated but good cloud development relies on the sort of fundamentals that sometimes go wrong in monolith development.

Co-ordinating subscription renewals, credit card billing and confirmation emails is an example of working with distributed systems on a monolith. Do this in the wrong way and you billed a customer multiple times, or spammed their email. Trade-offs, failures and retries needed to be carefully considered.

A lot of applications take external systems for granted, rather than considering that they do sometimes go wrong. One place I worked coupled their login process to Salesforce. When Salesforce had an outage, their monolith shared it. When local filesystems have issues, applications will fail dramatically.

There are obvious differences between cloud and monolith development (not least the operational complexity), but both require an attention to the principles of software development. You don’t need a large system to be hit by the fallacies of distributed computing.

Categories
programming

My Most Useful Technical Interview Question

One interview question that I’ve been using for about ten years seems to filter out more candidates than any other. It’s not a trick, and I still don’t understand how come it catches so many people. Sometimes I worry that there is something wrong with what I’m asking.

The question is this – using a text editor and not an IDE, write a simple method to take an integer as an argument and return the factorial. I make sure to explain what a factorial is and wait.

The people I’m interviewing are rarely novices. I’ve asked this from people with years of banking experience. Some of them had exciting CVs, with successful projects and all the skills I was looking for; they could talk fluently about complex technology. Yet they did not seem that familiar with code. I’ve senior developers struggle with writing a simple loop.

I ask a lot of other things in interviews. I try to be open-minded, searching for strengths rather than weaknesses. I don’t bother with tricky algorithm questions that people rehearse for interviews and forget once they get an offer. With everything else I ask, if a candidate doesn’t know the initial answer, I follow-up to find what they do know.

But I expect anyone going for a technical position to be comfortable writing a simple piece of code, to be familiar with what code looks like. Can you write a loop and check it? I try to account for the fact that the candidate might feel nervous, and might find the lack of an IDE challenging. Sometimes, I tell them not to worry too much about syntax, to use pseudocode if they like.

I’ve been interviewing developers for years, and that question is essential. The piece of code I ask for is trivial. I’ve heard of interviewers getting the same results with FizzBuzz. The example that I use is listed across the web as an interview cliche, something a prepared candidate would expect. A good candidate disposes of this quickly and moves to the next question; but some people struggle. The question shouldn’t work, but it does.

Categories
programming

Some things learned in 20 Years of Software Development

I’ve now been a software developer for over 20 years. I started out thinking this would be a temporary diversion, but it’s grown to be something I love. I’ve been lucky enough have a wide experience of the industry, from mobile to microservices, and from three-person companies to multi-nationals. So, I decided to compress some of what I’ve learned into some short points:

  1. If a bug isn’t getting fixed this month, then you might as well not track it as you’ll never touch it. (or, to put it more positively – use a zero-bug strategy!)
  2. TDD is never going to take off. Everyone has automated tests, which is great, but I’ve never worked anywhere that used the proper TDD cycle in practise.
  3. Good project management is more important than methodology. Projects are just as messed up under agile as they were under waterfall, but we now have more meetings.
  4. The DRY principle is overrated. Too many people go for this ease of change rather than ease of reading. This is especially problematic in test code, which is mostly write-only.
  5. Focus on the data – I’ve always considered any computer system as a data-store with some code attached, and this works pretty well. If you get the transactions right, everything else will follow.
  6. The best code is simple. If it can’t be followed by junior colleagues, it’s too complicated.
  7. Projects rarely fail for technical reasons. Unless you’re doing something cutting-edge, the failure is due to something within your control. Software development is one of the least important parts of being a developer.
  8. Performance testing is hard.
  9. New technologies get less exciting as you compare them to things you already know. Like, gRPC is just fancy SOAP.
Categories
brightonjava

Brighton Java and the Pandemic

When the pandemic first struck, I put Brighton Java into hibernation. I expected the pandemic to be resolved after maybe six months, and thought I would wait it out. Since that’s not what has happened, it’s time to consider what a local Java User Group should be when we can’t meet together.

The obvious response to the situation is to move events online. Back in March I decided against this. My life already involved a lot of screens, and I couldn’t imagine many other people being excited about watching someone discuss programming on a videoconference call after doing a similar thing all day. For me, the attraction of the events are meeting people in person: it’s sharing of food before the talks, and the conversations afterwards. We’ve had some fantastic talks at Brighton Java, and I’m grateful to all of our speakers, but talks are only part of the reason for running these events.

I do think local user groups are valuable. They provide community at a scale wider than the individual companies in an area. They help companies, by exposing people to ideas and techniques that might not be encountered within a single company. For developers, the group provides a continuity beyond that of a single job.

Over the next few months we’re going to be restarting Brighton Java. Yes, virtual talks will be a part of this. But we’re going to work hard to make these relaxed and enjoyable. We’re going to take advantage of the virtualisation of other groups, and the rich programmes they’ve been running over the past few months. We’re also going to look into novel formats, particularly asynchronous ones.

As ever, we’re interested in hearing from any local developers who are interested in participating or running events. And some announcements from us will be coming soon.

Categories
brightonjava

The Art of Running Meetings

I love this spoof advert, which I first saw back when you had to share your memes by email:

We are social animals, so meetings are an important part of our work. But they can also get in the way of work. I’ve known too many people who claimed to be more effective when they worked from home, or who worked early or late to ‘get things done’. Given how important meetings are, it’s worth spending time getting them right. Running effective meetings is a skill, and too often people don’t bother.

For me the following is essential for any meeting:

  • A clear summary of the meeting, so people can quickly work out if they need to attend (particularly if they see it on a shared calendar).
  • A goal – what do we want to get out of this meeting?
  • An agenda – what specific points are we discussing?
  • Pre-requisites – what should people read or do to prepare for the session?
  • Follow-up – after the meeting, a clear summary should be produced and preferably posted to a shared wiki. There should also be a list of action items, with each assigned to a specific individual.

A lot of work is needed around a meeting, but this is worth doing because meetings are expensive. Sometimes I think there should be a clock in the room, showing how much it has cost so far. A one hour meeting for seven people is taking up the equivalent of a person’s work for a day.

In his most recent video for Brigton Java/Silicon Brighton, James Stanier talked about running effective remote meetings:

James provides a great primer on meetings, as well as some useful points that I’d not considered before:

  • Recurring meetings should have a rolling agenda in a shared document which anyone can add to during the week.
  • The best meeting is one that does not need to happen, where people can resolve the issues without the need for synchronous time.
  • James also suggests ‘silent meetings’, allowing time for people to read relevant documents at the start of a meeting.
  • As well as a chair and a scribe, larger meetings can also use ‘spotters’ to identify people who have not been able to speak or to contribute.
  • James also suggests recording meetings to reduce the need for them to be synchronous. I don’t like listening to meeting recaps personally, preferring written summaries, but maybe I should try making space for other people’s preferences in meetings that I’m running.

As James says in his presentation, “synchronous time is special”. It’s worth investing some preparation to make sure it is used as well as possible.