Category: openstreetmap

Categories
openstreetmap

Using postgis on docker to Find Ley-Lines

Back in 2015, I wrote a blog post about finding ley-lines with postgis, based on some work by Steven Kay.

I described the outline of how to produce as leylines as a How-to-Draw-An-Owl tutorial, since I skipped over a lot of the complex points. I was setting up my temporary postgis instances on AWS and there was a certain amount of faff involved in sorting out database access and structure. This post is an improvement on those methods.

(There are still bits I’m not explaining in full – basically, how to display the ley line – but if you need more details, please let a comment and I’ll get in touch. But, basically, you need a little bit of HTML/Javascript to take this all the way through)

I’ve returned to this project for some recent writing, and the technical aspects are much easier using docker. In fact, the process can be broken down to seven steps, which I’ve listed below. You’ll need to be comfortable with a command line, have docker installed, and be willing to play about a little. But the process is relatively straightforward:

  1. Start up a postgis instance from a computer with docker installed:
docker run --name some-postgis -e POSTGRES_PASSWORD=mysecret -d postgis/postgis

2. Open a terminal in the docker instance:

docker exec -it some-postgis /bin/bash

3. From inside the docker instance, install the OSM tools

apt update
apt install osm2pgsql

4. From the host computer, download some OSM data and copy the file into the container. There are various options for downloading this data but I tend to use the Geofabrik server. 

docker cp ~/east-sussex-latest.osm.pbf some-postgis:postgres

Inside the container, the OSM file can then be loaded into postGIS using the command

osm2pgsql merseyside-latest.osm.pbf -U postgres

5. Back to our terminal inside the host, start up postgres

psql -U postgres

6. Now, here is the bit where I send you to another tutorial to get the ley lines SQL. These have been made available as a github gist by Steven Kay. Cutting and pasting the commands will get a fairly decent result, but I recommend having a play to get the best results.

7. You can now use another query to get the KML (Keyhole Markup Representation) format of the lines. This format is fairly portable, and I tend to plonk it in a simple HTML page. The larger the “ct” value, the more items are connected by the ley.

SELECT (ST_AsKML(geom)) as geom from leys where ct = 7;

8. And this is the hand-waving step, which is to display your KML lines on an actual map. Basically, you need to amend a simple OpenLayers example. Basically, if that leaves you stuck, leave a comment and I’ll get back to you (commenter email addresses are not be visible on the actual site).

The script from Steven Kay is great for producing candidate lines. The only problem is that it tends to follow clusters of pubs on the same street. The restriction that the lines must begin and end at a site while being at least 20km long prevents short local lines being plotted. But it would also be good to add an extra predicate to judge lines by how spaced-out the locations are. That’s something I need to work on in future.

Categories
openstreetmap

Openlayers and Angular

I recently set up a simple Openlayers page, which proved to be an interesting experience. As mentioned in that post, one problem I had was working with so many unfamiliar technologies. It’s a long time since my first tinkerings with Javascript, and I’ve paid little attention to it. Since then, front-end development has exploded into a variety of frameworks and libraries that interact in strange and spooky ways. Node! Typescript! Less! Angular! Jquery!

In the long term, I want to get back into full-stack development; but in the short term, I just want to put a mapping site live. I’ve settled on JHipster as a platform, since it makes managing the database and middle-tier incredible easy, as well as setting up a lot of the Angular infrastructure. The problem is that almost all the Openlayers examples that I can find are written with JQuery.

I found a few clues on what to do, such as a Stackoverflow post on How do I dynamically add popups to markers using OpenLayers 5 in Angular 4?, but my meagre experience of Angular meant I couldn’t get the Popper library this used to work. But the answer gave me enough of a clue to get the syntax and code set up for a very basic implementation of Openlayers popups within angular. The work is in done within the controller, and builds and executes correctly.

It’s far from perfect – because the component is embedded within another page, the pop-up doesn’t appear with the feature, rather it is placed a short distance above. There is also a problem with the click detection being flakey – the window has to be resized for this to work reliably.

But the good thing is, I now have a proof of concept for Openlayers working within JHipster. As scrappy as this is, I do seem to have them working together now. It’s going to be ugly, but I have enough to build a basic application with.

Categories
openstreetmap programming

First steps with Openlayers

tldr; first steps in setting up an Openlayers map with markers loaded from a GeoJSON file

I spent a chunk of this weekend wrestling with writing software. Which was a good way to relax after a late night dancing at Is that all there is to a Disco? Saturday was a slow day: writing code, watching Escape Plan 2 with Rosy Carrick (not a good film) and getting to grips with Openlayers.

Most of the time I learn about programming through toying with examples. While this is easy in a subject I know well, it can prove more difficult in new areas. Working with Javascript libraries can be particularly difficult since it’s hard to tell which versions particular examples are for – with Openlayers there are several mutually incompatible versions with very different APIs. They say you should only do one new thing at a time, and learning Openlayers was held back by my poor Javascript skills. I also needed to get to grips with the particular way the mapping geometries worked.

But I managed to beat it all into submission and produce a working demo. I wanted to load some data from GeoJSON and display location markers on an Openstreetmap map. It’s not very much, but it is something I can later add to a larger project.

Introduction

According to Wikipedia , Openlayers is “an open source JavaScript library for displaying map data in web browsers as slippy maps”. It supports multiple map vendors and different data formats. The main project site  says that version 5.2 is the latest. There are links on the site to

Basic Example

The quickstart page shows how to set up a simple Openlayers map. CSS and HTML are used to create a page-wide div that can hold the map. The Javascript to create and display the map is simple:

var map = new ol.Map({
target: 'map',
layers: [
new ol.layer.Tile({
source: new ol.source.OSM()
})
],
view: new ol.View({
center: ol.proj.fromLonLat([37.41, 8.82]),
zoom: 4
})
});

This creates a map object and sets it into the map object. A single layer is set up with OSM (openstreetmap) as the source. The view allows the map’s centre, resolution and rotation to be set. Note that zoom level 0 is zoomed out fully.

  • As the tutorial explains:
    A layer is a visual representation of data from a source. OpenLayers has four basic types of layers:
    Tile – Renders sources that provide tiled images in grids that are organized by zoom levels for specific resolutions.
  • Image – Renders sources that provide map images at arbitrary extents and resolutions.
  • Vector – Renders vector data client-side.
  • VectorTile – Renders data that is provided as vector tiles.

The center of the view is set using  a conversion from longitude and latitude. This is one of the confusions with open streetmap. It uses a co-ordinate system called Spherical Mercator projection (known as EPSG:3857),  and the usual longitude/latitude co-ordinates are held in a projection called EPSG:4326.

Adding Markers

The next stage was finding a way to add markers to the map. A lot of the examples for this were using different versions of Openlayers. The one that I ended up adapting was a recent tutorial on mediarealm.

This piece of code gave me enough of a clue to be able to set up a layer with a marker:

function add_map_point(lat, lng) {
var vectorLayer = new ol.layer.Vector({
source:new ol.source.Vector({
features: [new ol.Feature({
geometry: new ol.geom.Point(ol.proj.transform([parseFloat(lng), parseFloat(lat)], 'EPSG:4326', 'EPSG:3857')),
})]
}),
style: new ol.style.Style({
image: new ol.style.Icon({
anchor: [0.5, 0.5],
anchorXUnits: "fraction",
anchorYUnits: "fraction",
src: "https://upload.wikimedia.org/wikipedia/commons/e/ec/RedDot.svg"
})
})
});
map.addLayer(vectorLayer);
}

We create a new vectorlayer which contains  another vector with all of the features. This could then display markers with a small red dot.

GeoJSON

Ultimately, I wanted to use some GeoJSON  to provide a set of features. This proved to be tricky, as I had features in longitude/latitude co-ordinates, and couldn’t see how to make the import with that taken into account. Fortunately, it’s easy to iterate over objects in javascript, and I simply looped over the GeoJSON, making the conversion with one object at a time before adding it to the layer.

var featureArray;
for (feature of placedata){
        featureArray.push(new ol.Feature({
 geometry: new ol.geom.Point(ol.proj.transform([parseFloat(feature. longitude), parseFloat(feature.latitude)], 'EPSG:4326', 'EPSG:3857')),
name: feature.name,
description:feature.description
 }))
}

Once loaded, this feature array can then be dropped easily into a vector layer and added to the map. The source of the placedata array can be changed – I started out with a hardcoded version, and then added in one from a REST endpoint. I created a quick REST server using the Spring Boot REST example as a basis, and changing it to return place data.

Loading the data from GeoJSON turned out to be the fiddliest part of this because of the conversions between different projections.

Pop-ups

The last thing I wanted to do was add pop-ups to the markers. There was an example of this in the Openlayers workshop. The way this works is simple – an onClick event for the map is added, checking whether there is a feature at the point in question. There’s a lot of code to the example, but it was easy enough to amend and get working.

Summary

It took a little longer than planned, but I managed to set up a simple HTML page that displayed openstreetmaps tiles, with markers provided by a REST endpoint.

Categories
openstreetmap

Finding Ley-Lines with PostGIS

I’ve recently been playing with PostGIS. This post will summarise a simple attempt to manipulate data and draw it on top of Open Streetmap. I wanted to produce a Brighton version of Steven Kay’s Pub Ley Lines. As you will see, the outcome was less interesting than the process.

What follows is, basically, a How-to-Draw-An-Owl tutorial. It summarises the steps I usually look up and is intended to share with a few specific people. If you find yourself here via Google and want more information, leave a comment and I will add more detail.

1 – Download data from OSM

The first thing I needed was the data from Open Streetmap, which contains pub locations among the points-of-interest. There are a number of options. Downloading direct from OSM failed when I last tried it, but I had an older version of the data available.

2 – Set up Postgis and osm2pgsql

I’d previously installed Postgresql and Postgis on my laptop but somehow the installation has become broken and won’t be easily repaired or uninstalled. I should fix this, but I wanted to get on with this experiment. I’ve been meaning to set up AWS for some time, and using a micro instance on Amazon allowed me to get a version of PostGIS running very quickly.

I created a new EC2 micro instance based on the basic ubuntu instance, ran ‘sudo apt-get update’ followed by ‘sudo apt-get install osm2pgsql’ and I had everything I needed.

AWS is awesome, and I love being able to run up an instance for a small task and throw it away once I’m done.

3 – Create postGIS user and table

Setting up a new user on postgres is certainly less of a hassle than doing it on MySQL, but there are a few gotchas – such as needing to add a line to the /etc/postgresql/9.3/main/pg_hba.conf file. Also, when creating a new database, remember to enable PostGIS with the command “CREATE EXTENSION POSTGIS”

4 – Load the OSM data in Postgres

Having transferred my PBF file to the AWS instance using scp, I could then load the data. The command here is a little different to the one I used previously because of this server’s limitations:

osm2pgsql -U ley -d ley  --slim --cache-strategy sparse --number-processes 4 brighton.pbf

5 – Create a table containing all of the ley lines

I pretty-much followed the recipe given by Stephen Kay here.

6 -Extract the data

This time, rather than a tab-separated format, I selected the WKT (well-known text) format for all the ley-lines with more than 8 pubs. I’m sure there are better ways to extract this. The query used was:

psql -U ley -w ley -c "COPY (select st_astext(st_transform(geom,4674)) from leys where ct> 8) TO STDOUT WITH CSV" > lines.txt

This may not be the most portable format for the data but I can bully it into something Openlayers can use.

7 – Create a page to display the data

Since the data is constant and will be low-traffic, I am using OSM via static files to display the results.  The linestring WKT representations of the leys have been copied into the javascript file rather than being loaded from a file. All quick-and-dirty, but it has worked. The source is on github and the results are online.

Categories
openstreetmap

Getting Open Streetmap working on Ubuntu

It’s taken me quite a bit of time but I’ve finally got an open streetmap tileserver working on my laptop. The main issue appears to have been that I was following the manual Ubuntu set-up instructions rather than building a tile server from packages. Which isn’t to say that using packages was plain sailing. This post will outline some of the problems I ran into and how I solved them.

(Bear in mind that I’d already tried to install OSM using the manual method on the laptop and fragments of these installs were still on the system. Installation might be a lot easier on a clean OS. However, it’s worth documenting the issues in case anyone runs into the same issues and is googling for them)

The main point of this post is to say that I have managed the open streetmap set-up and now have a map of Brighton working on my laptop:

Screenshot from 2015-03-03 08:11:32
I can see my house from here!

The installed slippymap.html page is great as the radio buttons on the top right allow you to compare a remote instance against the local one – useful for finding out where you are when things aren’t working, or you only have a small area set up. Now I’ve fixed this, I can get on with actually using the server.

The main problems I had to solve with the packaging method were:

Mod-tile dpkg installtion hangs

The command to install mod-tile hung. Googling around, I found a patch for this  – a line was missing from the libapache2-mod-tile.postinst script. Editing the script and re-running the installation fixed this issue.

osm2psql didn’t work

When I ran the command osm2pgsql I received an error:

Osm2pgsql failed due to ERROR: SELECT AddGeometryColumn('planet_osm_point', 'way', 900913, 'POINT', 2 ) failed: ERROR:  AddGeometryColumns() - invalid SRID

This seemed to have been caused by not having set up data in the spatial_ref_sys table. To check this see if there are any lines from the SQL query

SELECT * FROM spatial_ref_sys;

If not, then this is easily fixed:

psql --username=postgres --dbname=gis --file=/usr/share/postgresql/9.1/contrib/postgis-1.5/spatial_ref_sys.sql

After re-running this script, the osm2psql script worked. Note that this may have been a problem due to me incorrectly removing previous installation attempts.

Tiles not rendering

This is more of a generic issue. Basically, when the slippymap page is working, but the images are not being generated, a lot of missing images will be displayed:

Pink tiles

You can open the blank tiles in a new page and see there where they are being generated from. Some of the help online suggested that it could be down to the slippymap.html pointing to the wrong OSM server (it defaults to localhost). In my case it was a permissions issue:

Database permissions not set correctly

There seems to be a minor issue with the installation scripts. This was fixed by explicitly setting up the postgis user.

I was able to diagnose this issue by running renderd as a foreground process (ie renderd -f) and seeing permissions errors such as the following:

renderd[26466]: An error occurred while loading the map layer 'default': :
FATAL: role "www-data" does not exist (encountered during parsing of layer 'landcover' in map '/etc/mapnik-osm-data/osm.xml')
renderd[26466]: An error occurred while loading the map layer 'default': :
FATAL: role "www-data" does not exist (encountered during parsing of layer 'landcover' in map '/etc/mapnik-osm-data/osm.xml')
Categories
openstreetmap

First steps with OpenStreetMap

(3/3/15 EDIT – it turns out that installing OSM is simpler than I thought. I’ve published an update to this post which details the steps taken to a successful installation)

Over the past few weeks I’ve spent a lot of time wrestling with installing an OpenStreetMap server. In theory this should be simple. The tutorial for the version of Ubuntu that I use makes it look easy. Instead this is the most difficult install I worked with since the days when Oracle released their shrinkwrap products with incorrect documentation.

The OSM install involves five different products, some of which need to be compiled from source, and one which doesn’t work in the documented version. Fortunately, a lot of the problems I faced could be resolved via Google. Although using make only increases my love for maven. I don’t miss old-school build systems.

I’ve had multiple stabs at installing OSM, which seems about par for the course. I’ve not succeeded yet, but I seem to be making progress. I thought I would document what I’ve done here – partly for my own benefit and partly for anyone who comes by on Google and is as lost as I am. I’m determined to succeed at this – it’s a challenge and a puzzle and will probably seem quite simple once I’ve cracked it.

The Stack

The OpenStreetMap stack consists of four elements:

  1. mod_tile, an apache module that serves and caches map tiles
  2. renderd – a “priority queueing system” to manage the load from mod_tile’s rendering requests.
  3. mapnik – the software library that does the actual rendering of the map tiles.

  4. postgresql/postgis – the datastore containing the mapping elements

OSM also needs osm2pgsql to load the OSM data into postgis. So far, I have mapnik working and mod_tile passing requests to renderd, but I cannot link renderd to mapnik. The only thing I’ve served from mod_tile is this world map, which is produced outside the main mapnik process:

0

Installation

The OSM installation is fiddly, which means Googling for the issues. There are also lots of fileswhere a small mistake can need debugging, for example replacing variables in DTD entities. Things like these would be easier if the installation files were opinionated – this works well in Spring Boot and Maven.

The main issue I’ve had is with versions of software (which is where I think I’m currently stuck). One bug had to be fixed by using a version of mod_tile from github.com:springmeyer/mod_tile.git. However, I am not convinced this is playing nicely with some versions of mapnik.

I had decided to use a limited area of the world, rather than the full-on 29GB download. To work out what tile co-ordinates are needed by mod_tile requires a mapping from long/lat to the OSM cordinates (I used the python scripts on that page). These co-ordinates can be checked against the OSM public servers. For example, Brighton is http://a.tile.openstreetmap.org/12/2046/1374.png.

As I said above, I still can’t get renderd to talk to mapnik, although mapnik is working, shown by this attempt to render Brighton:

out2

Apparently, the East Sussex data I had didn’t include Brighton. But a query in East Sussex produced a decent map:

out

What Next?

Despite my issues, I’m still excited about OSM and have a couple of ideas for using it. And I’m convinced that I can figure out what’s going wrong, although I will need to sit down and actually read the mod_tile documentation rather than trusting to installation guides and online discussions.

I regret not attempting the original installation on a Vagrant VM, as this would have enabled me to start from scratch. I may have to do a little work to remove the different versions of libraries on my laptop before I can continue the installation.

Hopefully, another few hour’s work should see me with a working version of OSM. Once I’ve done that, I will write up a step-by-step list of the steps required.