2014_03_04 progress re LA colours, calculating LA extrema and switching off unwanted polygons for markerClusterGroups

We’re into March! After meeting with the client, a few changes are needed.

Colouration

Working from the chloropleth example, I’ve chosen 5 colours (cyan = #0080ff, red = #ff0000′, yellow = #ffff00, blue = #0000ff’ and green = #00ff00). This choice was inspired by Education Scotland’s map.)

In the geoJSON file, each feature (i.e. each LA) has an extra line in the ‘properties’ piece: “colour_code”:3 //BMR 2014_02_21.

The code to decide colours is
function getLAcolour(d) {
  return d == 1  ? ‘#0080ff’:
            d == 2  ? ‘#ff0000’:
           d == 3  ? ‘#ffff00’:
           d == 4  ? ‘#0000ff’:
           ‘#00ff00’;
} // end getLAcolour()

Then there’s a function to create a style object:
function laStyle(feature) {
    return {
        fillColor: getLAcolour(feature.properties.colour_code),
       weight: 2,
       opacity: 1,
       color: ‘white’,
       dashArray: ‘3’,
       fillOpacity: 0.7,
       fillOpacity: 0.4
   };
} //end laStyle(feature)

Then the geoJSON function does the heavy lifting:
L.geoJson(laBoundaryData, {style: laStyle}).addTo(laLayerGroup);

So the whole effect looks like this:Screen Shot 2014-03-04 at 16.28.40It’s deliberately similar in look to this map of constituencies and MSP activities:Screen Shot 2014-03-04 at 16.29.32I’ve also moved the link to © data and acknowledgements to a second line in the acks piece at the bottom of the map, so far less screen area is taken up by non-map things. The instructions will go onto a pop-out pane access from a big ? on an edge of the map. Currently they clutter things:Screen Shot 2014-03-04 at 16.34.14

Calculating LA boundaries and extrema: how I did it

I do not claim this was the most efficient way! The following is an expanded write-up of a previous blog entry.

  1. Obtain up to date shape files from the Ordnance Survey (BoundaryLine is the relevant product.)
  2. Start a new project in QGIS.
  3. Do Layer > Add vector layer and select district_borough_unitary region.
  4. Make the layer editable.
  5. Select and cut out the non-Scottish features.
  6. Save the project as Scotland.
  7. Select each LA in turn, then do Vector > Geometry tools > Simplify geometries. The options I used were:Screen Shot 2014-03-04 at 17.07.58 but with a different file for each LA. For example, Shetland’s 160,883 vertices were reduced to 5416.
  8. Select the newly-created object, then do Layer > Save selection as vector file, with optionsScreen Shot 2014-03-04 at 17.09.30

So now I had 32 separate geoJSON files. To combine them, I used the process described here to make a single laBoundaryData.js file. That’s my geoJSON file on which the above styling magic works.

Finding LA extrema

In future versions of this map, I’ll want to zoom into individual LAs. The easy way to do this will be (I think) to find the furthest north, south, east, west points of each LA. So I adapted my code to find Scotland’s extrema to work on each LA:
function drawMap() {
    //BMR 2014_02_25
    //find extreme points
    var north = 0;
    var south = 90;
    var west = 90;
    var east = -90;
 
    for (var i = 0; i < coordinates.length; i++) {
        var laDatum = coordinates[i];
        // the other way round from the getScotlandBounds because geoJSON latlongs are in the opposite order to leaflet latlogns
        var longitude = laDatum[0];
        var latitude = laDatum[1];

        if (latitude > north) {
            north = latitude;
        //end if

        if (latitude < south) {
            south = latitude;
        //end if

        if (longitude < west) {
            west = longitude;
        //end if

        if (longitude > east) {
            east = longitude;
        //end if
    //end for

    document.write(“north = ” + north + “, south = ” + south + “, west = ” + west + “, east = ” + east);

}// end drawMap

It’s a complete hack but I copied, pasted and edited the results into my laData.js file to get line such as
    [“Angus”, “http://www.angus.gov.uk/commcouncil&#8221;, 
        56.986816427679120, 56.46164866362316, -3.407021822671358, -2.420365421269425],

If I’d been clever, I’d have made the script write the extrema data to the file. But this is a one-off and so it would have taken longer to write code to do it than to do it myself.

Switching off unwanted polygons for markerClusterGroups

Awe and respect to the leaflet programming folk. I feared I might need to dig into their actual code to switch off these unwanted polygons. But it’s as simple as adding an option object to the otherwise unoptioned code. That is, from this

L.markerClusterGroup();

to this

L.markerClusterGroup({showCoverageOnHover: false});

2014_02_21: even more LA boundary progress

So a good conversation with Napier’s visualisation expert and I’m now more aware of some of the ways to make my code more bombproof – too be implemented this weekend, if paper-writing and having a life allow.

Meanwhile I’ve completed colouring the LA boundaries. Here’s some pretty pictures:

Neither LAs nor CCs switched on

Neither LAs nor CCs switched on

CCs switched on

CCs switched on

LAs switched on

LAs switched on

both switched on

both switched on

I’m not too bothered that this uses 6 colours when there should be a four-colour solution. I’d be more picky about the actual colours used. Changing the colours is easy – just changing up to 6 values in a function in my main script. Changing which LAs each colour is applied to involves opening the huge geoJSON file full of LA data, then finding, say, East Ayrshire, then changing the value of colour_code immediately below it.

2014_02_21: more LA boundary progress

Manually copying and pasting 32 sets of bits of file is fraught with difficulty, as any fule kno.  So there had to be an easier way to assemble the individual LA geoJSON files into one javascript file. I did a lot of comparison of the brackets and guts of the working-so-far leaflet example and my geoJSON files. My file began with 

{
“type”: “FeatureCollection”,
“crs”: { “type”: “name”, “properties”: { “name”: “urn:ogc:def:crs:OGC:1.3:CRS84” } },
                                                                                

We only want one FeatureCollection, with 32 features for the 32 LA boundaries. So in a new laBoundaryData.js file, containing var laBoundaryData = { }; from the first LA file insert

      “type”: “FeatureCollection”,
      “crs”: {
            “type”: “name”,
            “properties”: {
                  “name”: “urn:ogc:def:crs:OGC:1.3:CRS84” } },
 
      “features”: [{
            “type”: “Feature”,
            “properties”: {
                  “NAME”: “Aberdeen City”,
                  “AREA_CODE”: “UTA”,
                  “DESCRIPTIO”: “Unitary Authority”,
                  “FILE_NAME”: “ABERDEEN_CITY”,
                  “NUMBER”: 7.0,
                  “NUMBER0”: 34.0,
                  “POLYGON_ID”: 122136.0,
                  “UNIT_ID”: 30421.0,
                  “CODE”: “S12000033”,
                  “HECTARES”: 20561.013,
                  “AREA”: 1990.394,
                  “TYPE_CODE”: “AA”,
                  “DESCRIPT0”: “CIVIL ADMINISTRATION AREA”,
                  “TYPE_COD0”: null,
                  “DESCRIPT1”: null,
                  “colour_code”:2         //BMR 2014_02_21
            },
            “geometry”: {
                  “type”: “MultiPolygon”,
                                    “coordinates”: [ [ [ [ -2.360870189456887, 57.10919031061777  ….]]]] } },

Note the comma. After it, paste { “type”: “Feature”, “properties”: ….]]]] } } from the next LA file, then add a comma. Then rinse and repeat.

Far from elegant but much easier than isolating wee bits from each file.

2014_02_20 more LA boundary progress

I spent a while today trying to drop a marker at the user’s entered location. It’s easy to do on(zoomed) add marker but removing it at the next zoom is so far beyond me. Back to fighting with LA borders – the theatre of battle being styling them

I knew from leaflet’s example that leaflet does style geoJSON – so the issue must have been with my geoJSON files. When in doubt, copy: I copied leaflets’ code for colouring US state geoJSON data, substituted in the Scottish LA names and a piece to colour them by code rather than population density, and lo and behold it works.

Truth – it took a while to isolate the co-ordinate lines, so I’m only a wee way into substituting in the LA co-ordinates. But here’s the proof:

Screen Shot 2014-02-20 at 22.51.16

2014_02_18: progress and success

Remember the problem with geocoding? It may just be fixed.

My main script has a function to set the bounds of the displayed map. If you try to drag the map so that areas outside these bounds are displayed, the map will snap back to the area I want it to display.

I had a look through the geocoder script – it has an options object, one line of which sets bounds to null. So I made this line call the setScotlandBounds function from my main script. After a bit of futzing to understand how one script can call a function in another script (the secret is to make the web page call them in the right order!), it works. EH10 5DT is where it should be. Searching for London gets London in Orkney.

I am so relieved – hacking the geocoder script or playing with Nominatim looked fearsome.

Next step is to hack geocoder or my main script so it doesn’t zoom in ridiculously. And that’s quite simple – just set a maxZoom in the map options object.

And the final MUST do – adding a pin at the user’s entered location sort of works. Because the map will definitely zoom when displaying the desired location, adding an onzoom event to put a marker works. But if you zoom in again, another marker will be placed.

The solution to that will be to place the marker into a layer. The next zoom would remove that layer.

2014_02_16: progress

So after ac couple of days of literature reviewing, and spending Saturday being social, I was back at the code-face today.

Task 1: understand – and maybe cure – the failing geocoder

Entering single-initial letter postcodes such as EH3 7PX works – the map zooms to the right place. Entering textual places such as Merchiston, Leith Links, Macmerry and Dalry also works. But EH10 5DT still appears to be in Walthamstow, instead of Merchiston. EH12 6AP appears to be slightly east of Walthamstow, instead of near Murrayfield.

I noticed that EH10 5DT and E10 5DT both give the same place (in east London), as do EH12 6AP and E12 6AP. So Nominatim is omitting the H in some cases. Interestingly, the geocoder works with Macmerry’s postcode (EH33 1PL).

To see whether it was my fault, I created a stripped-down version from scratch: the javascript contains just the map declaration, the code to fill it with tiles and the geocoder. Same result – so it’s the geocoder, not my code. I feel far less guilty but I’ll need to fix this.

Task 2: understand how leaflet.js handles geoJSON files

I worked my way through http://leafletjs.com/examples/geojson.html. I got most of it to work, except for styling the output. I then got a trial LA boundary to work – this trial had 4 co-ordinates. If it will work for four, it will work for many, many more – but will take longer.

Task 3: make geoJSON files of LA boundaries

With thanks to a Leaflet mailing list member for some sage advice, I’ve installed QGIS. This could open the 18-month old UK local authorities file I obtained from https://www.sharegeo.ac.uk/handle/10672/305. (It couldn’t handle the .sld files I obtained from the Ordnance Survey.) The way I worked – there may well be better ways – was to select an LA, then do Layer > Save selection as a vector file with options

  • format = geoJSON
  • CRS =  WGS84/EPSG4326

It took a while to work out the correct CRS option. I was surprised it wasn’t WGS84/PseudoMercator/EPSG3857 because that appears to be Leaflet’s default CRD. Then again, what do I know about converting points on am irregular 3D surface to points on a flat plane?

Anyway, that gave me 32 nice geoJSON files in a little less than 2 hours.

Task 4: display LA boundaries

Using my stripped-down version of the code, I added in 32 routines to display the LA boundaries. They were still in the default blue – remember I’d not cracked styling them. There may be an advantage to separate routines and data files. If an LA ever changes, I’d just need to make new geoJSON files for that LA and its neighbours, and each can be coloured (if this ever works!) separately. Anyway, by adding var = <LA_name> [ to the beginning of each file and ]; to the end of each, then appropriately commenting off all routines except for the LA I was testing, I made sure each displayed OK.

I then tried a few methods to programatically assemble the files into one huge array of the form var laBoundaryData = [ [content of first geojson file], [content of second geojson file], ….[content of last geojson file] ];  Another hour down the pan and no joy, so I did it the hard way, copying and pasting the contents of the geoJSON files into
var laBoundaryData = [
],
[
], 
[
]
];

It almost worked straight off – all the LAs apart from Shetland and Scottish Borders displayed as they should. Shetland and Scottish Borders had some weird horizontal lines, but removing them from the huge file, then replacing their code from their individual geoJSON files got rid of these nasty artefacts. I guess I had pasted the code in slightly the wrong place first time round.

Then the code for displaying the LAs is just a simple traversal of the array of LA boundary data. And all this without going anywhere near the command line.

So with my map displaying all LA boundaries, it was time for…

Task 5: putting it all back together

I’d thought this would be easy – just replace the code lumps for calculating the map centre, its bounds and displaying the CCs. Not a bit of it! Failure points included not calling the data, scripts and even the css in the right order, the functions for calculating centres and bounds not being called properly – aarrgh. Anyway after a couple of hours, it’s all there. It’s slow (I guess need to simplify the LA shapes before writing them to geoJSON format), the geocoder is still a mystery but it works. Yeehah!

The initial result

The initial result

zoomed into the West End of Edinburgh - see the markers for individual CCs

zoomed into the West End of Edinburgh – see the markers for individual CCs

 

My reward for all that – 2 more hours doing stuff to a real CC’s website!

 

2014_02_12

Got my MSc dissertation mark today. Passed!

Code had 32 sets of ‘place markers’ – one for each LA, all basically identical except that each LA’s markers were added to a different cluster group. Nightmare to maintain – if I change one thing, I have to change it 31 more times (exactly the same change). Chances for mistakes very high.

So spent today hacking this to a routine. Not helped by wasting hours forgetting that ‘=’ sets a variable’s value, while ‘==’ gets the value. GRR!

I now have a forest of markers over Scotland.forest

Next step is to bring back the clustering.

2014_02_11: to do list

Peter: CC location finder
Edit Bruce’s draft IIDI item, then tweet it IN PROGRESS RIGHT NOW
Sort permissions on cngn.co.uk (MAY NOT BE NECESSARY – the webserver bit works)
Bruce: CC location finder
Retweet PAC’s tweet
Get task trackers up to date. DO THIS EVERY DAY! DONE
Update MoSCoW list for CC location finder (include bounds to pan/scroll and search) DONE
Refactor code using array for list of LAs (not 32 repeated lumps of code)
Next most important task: limiting pan/scroll and search
White background for site
Find a way of batchconverting postcodes to LatLong
Testrun it on 2011 data
Bruce: CeDEM and CC website survey
Draft tasklist for CeDEM paper revisions. Concentrate on stuff that would help survey project
Bruce: funding
Implement PAC thoughts (does Aims bit include appropriate words, dewhiteman text)
Get Hazel advice on language
Get Hakanpaa, Nina N.Hakanpaa@napier.ac.uk advice on costs
Bruce: Commission on Strengthening Local Democracy
Respond I will make it, PAC would like to come but has teaching commitments
Set date with Peter for discussion of what we might say

2014_02_10

Not hugely productive today. Some chat with a colleague about CCs in her LA being, er, less effective than she’d like.

Attempts to go back to unhacked versions of leaflet, etc, so i could properly document where I changed locations of the image files they call.

Some work on a funding application – I’m useless at those!

Knocked together all the things supervisor and I could talk about at tomorrow’s management meeting.

Er, that’s it

2014_02_09

So I’ve given up on the polygons for now. In fact, I’ve received some advice in response to an online plea for LatLong shape files for Scottish LAs but I won’t be able to look into this until Tuesday afternoon.

So today on with the show of drawing a map with markers for all CCs for which I have LatLong data. This should have been easy – I’d had something very like this working for 2 CCs already, with different coloured markers for whether the CC had both website and email address, just an email address, no electronic contact details or no actual functionality.

I started by knocking up some css and a disclaimer/copyright page and some custom markers for functional and non-functional CCs – easy, if not the prettiest thing ever. Then I followed the recipe for displaying a couple of LAs’ CCs (without differentiating between functional and non-functional), with the markers for each LA coalescing when zooming out from the map.

All fine, until I noticed the numbers in the coalescences were not correct – there should have been 19 CCs in Aberdeen and 71 in Aberdeenshire, but while Aberdeen was correct, Aberdeenshire had 90 CCs, so the markers for Aberdeen were presumable adding to the Aberdeenshire set.

I then added in Angus CCs, and got even more confusing numbers. I battled with this for several hours, even stripping the code right down to for each CC in the whole dataset, document.write(“rude word”); No joy.

So this is where my personal life affects my work life – I went to my regular sunday evening spin class. I started again about 8:30 and within 30 minutes, I had it properly working numbers for 3 CCs. 29 fairly tedious additions to a case statement and checks that the right numbers appeared, all 32 LAs’ CCs were appearing correctly.

A quick hack to one LA got me green marked for functional CCs and red marked for non-functional ones, with each LAs’ own CCs coalescing into one lump no matter what colour they were.

Then half an hour’s web-trawling got each LA’s CCs’ markers displaying popups linking to the relevant LA web pages. Job done!

So entry to the page draws a map of Scotland, with all CCs for which we have LatLong data coalesced into lumps that show the number of CCs in each coalescence. Zooming in, or clicking on a coalescence de-coalescesces so that markers for individual CCs display. A marker is green if the CC is functional and red if it isn’t. Each marker has a popup with the name of the CC and a link to the relevant LA’s webpages about its CCs. Entering a postcode or address in the search box zooms to that place, so you can see the CCs around it. (You may need to zoom out a little.) We will draw a discrete veil over the question of how good the LatLong data I’m currently working with is – not my problem!

Still to do

  • display LA boundaries
  • get the search box to drop a marker
  • optionally, calculate the distance to the nearest CC
  • optionally, calculate the distance to the nearest functional CC
  • be able to switch on and off each LA’s markers, using leaflet.js’s layers functionality
  • optionally, to calculate distances to the nearest (functional) CC in an LA if only one is switched one
  • write a hacker-guide
  • style links in <h1>s

Getting there!