2014_02_20 more LA boundary progress

I spent a while today trying to drop a marker at the user’s entered location. It’s easy to do on(zoomed) add marker but removing it at the next zoom is so far beyond me. Back to fighting with LA borders – the theatre of battle being styling them

I knew from leaflet’s example that leaflet does style geoJSON – so the issue must have been with my geoJSON files. When in doubt, copy: I copied leaflets’ code for colouring US state geoJSON data, substituted in the Scottish LA names and a piece to colour them by code rather than population density, and lo and behold it works.

Truth – it took a while to isolate the co-ordinate lines, so I’m only a wee way into substituting in the LA co-ordinates. But here’s the proof:

Screen Shot 2014-02-20 at 22.51.16

2014_02_18: progress and success

Remember the problem with geocoding? It may just be fixed.

My main script has a function to set the bounds of the displayed map. If you try to drag the map so that areas outside these bounds are displayed, the map will snap back to the area I want it to display.

I had a look through the geocoder script – it has an options object, one line of which sets bounds to null. So I made this line call the setScotlandBounds function from my main script. After a bit of futzing to understand how one script can call a function in another script (the secret is to make the web page call them in the right order!), it works. EH10 5DT is where it should be. Searching for London gets London in Orkney.

I am so relieved – hacking the geocoder script or playing with Nominatim looked fearsome.

Next step is to hack geocoder or my main script so it doesn’t zoom in ridiculously. And that’s quite simple – just set a maxZoom in the map options object.

And the final MUST do – adding a pin at the user’s entered location sort of works. Because the map will definitely zoom when displaying the desired location, adding an onzoom event to put a marker works. But if you zoom in again, another marker will be placed.

The solution to that will be to place the marker into a layer. The next zoom would remove that layer.

2014_02_16: progress

So after ac couple of days of literature reviewing, and spending Saturday being social, I was back at the code-face today.

Task 1: understand – and maybe cure – the failing geocoder

Entering single-initial letter postcodes such as EH3 7PX works – the map zooms to the right place. Entering textual places such as Merchiston, Leith Links, Macmerry and Dalry also works. But EH10 5DT still appears to be in Walthamstow, instead of Merchiston. EH12 6AP appears to be slightly east of Walthamstow, instead of near Murrayfield.

I noticed that EH10 5DT and E10 5DT both give the same place (in east London), as do EH12 6AP and E12 6AP. So Nominatim is omitting the H in some cases. Interestingly, the geocoder works with Macmerry’s postcode (EH33 1PL).

To see whether it was my fault, I created a stripped-down version from scratch: the javascript contains just the map declaration, the code to fill it with tiles and the geocoder. Same result – so it’s the geocoder, not my code. I feel far less guilty but I’ll need to fix this.

Task 2: understand how leaflet.js handles geoJSON files

I worked my way through http://leafletjs.com/examples/geojson.html. I got most of it to work, except for styling the output. I then got a trial LA boundary to work – this trial had 4 co-ordinates. If it will work for four, it will work for many, many more – but will take longer.

Task 3: make geoJSON files of LA boundaries

With thanks to a Leaflet mailing list member for some sage advice, I’ve installed QGIS. This could open the 18-month old UK local authorities file I obtained from https://www.sharegeo.ac.uk/handle/10672/305. (It couldn’t handle the .sld files I obtained from the Ordnance Survey.) The way I worked – there may well be better ways – was to select an LA, then do Layer > Save selection as a vector file with options

  • format = geoJSON
  • CRS =  WGS84/EPSG4326

It took a while to work out the correct CRS option. I was surprised it wasn’t WGS84/PseudoMercator/EPSG3857 because that appears to be Leaflet’s default CRD. Then again, what do I know about converting points on am irregular 3D surface to points on a flat plane?

Anyway, that gave me 32 nice geoJSON files in a little less than 2 hours.

Task 4: display LA boundaries

Using my stripped-down version of the code, I added in 32 routines to display the LA boundaries. They were still in the default blue – remember I’d not cracked styling them. There may be an advantage to separate routines and data files. If an LA ever changes, I’d just need to make new geoJSON files for that LA and its neighbours, and each can be coloured (if this ever works!) separately. Anyway, by adding var = <LA_name> [ to the beginning of each file and ]; to the end of each, then appropriately commenting off all routines except for the LA I was testing, I made sure each displayed OK.

I then tried a few methods to programatically assemble the files into one huge array of the form var laBoundaryData = [ [content of first geojson file], [content of second geojson file], ….[content of last geojson file] ];  Another hour down the pan and no joy, so I did it the hard way, copying and pasting the contents of the geoJSON files into
var laBoundaryData = [
],
[
], 
[
]
];

It almost worked straight off – all the LAs apart from Shetland and Scottish Borders displayed as they should. Shetland and Scottish Borders had some weird horizontal lines, but removing them from the huge file, then replacing their code from their individual geoJSON files got rid of these nasty artefacts. I guess I had pasted the code in slightly the wrong place first time round.

Then the code for displaying the LAs is just a simple traversal of the array of LA boundary data. And all this without going anywhere near the command line.

So with my map displaying all LA boundaries, it was time for…

Task 5: putting it all back together

I’d thought this would be easy – just replace the code lumps for calculating the map centre, its bounds and displaying the CCs. Not a bit of it! Failure points included not calling the data, scripts and even the css in the right order, the functions for calculating centres and bounds not being called properly – aarrgh. Anyway after a couple of hours, it’s all there. It’s slow (I guess need to simplify the LA shapes before writing them to geoJSON format), the geocoder is still a mystery but it works. Yeehah!

The initial result

The initial result

zoomed into the West End of Edinburgh - see the markers for individual CCs

zoomed into the West End of Edinburgh – see the markers for individual CCs

 

My reward for all that – 2 more hours doing stuff to a real CC’s website!

 

2014_02_12

Got my MSc dissertation mark today. Passed!

Code had 32 sets of ‘place markers’ – one for each LA, all basically identical except that each LA’s markers were added to a different cluster group. Nightmare to maintain – if I change one thing, I have to change it 31 more times (exactly the same change). Chances for mistakes very high.

So spent today hacking this to a routine. Not helped by wasting hours forgetting that ‘=’ sets a variable’s value, while ‘==’ gets the value. GRR!

I now have a forest of markers over Scotland.forest

Next step is to bring back the clustering.

2014_02_11: to do list

Peter: CC location finder
Edit Bruce’s draft IIDI item, then tweet it IN PROGRESS RIGHT NOW
Sort permissions on cngn.co.uk (MAY NOT BE NECESSARY – the webserver bit works)
Bruce: CC location finder
Retweet PAC’s tweet
Get task trackers up to date. DO THIS EVERY DAY! DONE
Update MoSCoW list for CC location finder (include bounds to pan/scroll and search) DONE
Refactor code using array for list of LAs (not 32 repeated lumps of code)
Next most important task: limiting pan/scroll and search
White background for site
Find a way of batchconverting postcodes to LatLong
Testrun it on 2011 data
Bruce: CeDEM and CC website survey
Draft tasklist for CeDEM paper revisions. Concentrate on stuff that would help survey project
Bruce: funding
Implement PAC thoughts (does Aims bit include appropriate words, dewhiteman text)
Get Hazel advice on language
Get Hakanpaa, Nina N.Hakanpaa@napier.ac.uk advice on costs
Bruce: Commission on Strengthening Local Democracy
Respond I will make it, PAC would like to come but has teaching commitments
Set date with Peter for discussion of what we might say

2014_02_10

Not hugely productive today. Some chat with a colleague about CCs in her LA being, er, less effective than she’d like.

Attempts to go back to unhacked versions of leaflet, etc, so i could properly document where I changed locations of the image files they call.

Some work on a funding application – I’m useless at those!

Knocked together all the things supervisor and I could talk about at tomorrow’s management meeting.

Er, that’s it

2014_02_09

So I’ve given up on the polygons for now. In fact, I’ve received some advice in response to an online plea for LatLong shape files for Scottish LAs but I won’t be able to look into this until Tuesday afternoon.

So today on with the show of drawing a map with markers for all CCs for which I have LatLong data. This should have been easy – I’d had something very like this working for 2 CCs already, with different coloured markers for whether the CC had both website and email address, just an email address, no electronic contact details or no actual functionality.

I started by knocking up some css and a disclaimer/copyright page and some custom markers for functional and non-functional CCs – easy, if not the prettiest thing ever. Then I followed the recipe for displaying a couple of LAs’ CCs (without differentiating between functional and non-functional), with the markers for each LA coalescing when zooming out from the map.

All fine, until I noticed the numbers in the coalescences were not correct – there should have been 19 CCs in Aberdeen and 71 in Aberdeenshire, but while Aberdeen was correct, Aberdeenshire had 90 CCs, so the markers for Aberdeen were presumable adding to the Aberdeenshire set.

I then added in Angus CCs, and got even more confusing numbers. I battled with this for several hours, even stripping the code right down to for each CC in the whole dataset, document.write(“rude word”); No joy.

So this is where my personal life affects my work life – I went to my regular sunday evening spin class. I started again about 8:30 and within 30 minutes, I had it properly working numbers for 3 CCs. 29 fairly tedious additions to a case statement and checks that the right numbers appeared, all 32 LAs’ CCs were appearing correctly.

A quick hack to one LA got me green marked for functional CCs and red marked for non-functional ones, with each LAs’ own CCs coalescing into one lump no matter what colour they were.

Then half an hour’s web-trawling got each LA’s CCs’ markers displaying popups linking to the relevant LA web pages. Job done!

So entry to the page draws a map of Scotland, with all CCs for which we have LatLong data coalesced into lumps that show the number of CCs in each coalescence. Zooming in, or clicking on a coalescence de-coalescesces so that markers for individual CCs display. A marker is green if the CC is functional and red if it isn’t. Each marker has a popup with the name of the CC and a link to the relevant LA’s webpages about its CCs. Entering a postcode or address in the search box zooms to that place, so you can see the CCs around it. (You may need to zoom out a little.) We will draw a discrete veil over the question of how good the LatLong data I’m currently working with is – not my problem!

Still to do

  • display LA boundaries
  • get the search box to drop a marker
  • optionally, calculate the distance to the nearest CC
  • optionally, calculate the distance to the nearest functional CC
  • be able to switch on and off each LA’s markers, using leaflet.js’s layers functionality
  • optionally, to calculate distances to the nearest (functional) CC in an LA if only one is switched one
  • write a hacker-guide
  • style links in <h1>s

Getting there!

2014_02_08

Just started processing Shetland shape file. There are 549 polygons. There are 201,158 co-ordinates. Updated process:

  1. Prepare a Word text file called Shetland.js which contains only var shetland = [ ];
  2. In a copy of the file containing the data for Shetland, replace all ] ], [ [ with two <new line> in Word. Save.
  3. In the same file, replace all ], [ with <new line> in Word. Save.
  4. Get rid of dross at beginning and end of file. Save. You now have just co-ordinates, with a double <new line> between the set for each island.
  5. Find first double <new line>.
  6. Cut all co-ordinates preceding this. Save – you don’t want to risk doing this set twice.
  7. Paste the clipboard into the online batch converter, then convert.
  8. Copy the converted data into TextMate. Save.
  9. Do the first RegEx to get rid of the dross from the conversion step. Save.
  10. Do the second RegEx to replace all <new line>s with ], [, then save.
  11. Add , [ [ to the beginning of the Textmate file, and ] ] at the end. Save.
  12. Copy the whole of the TextMate file and paste it into shetland.js just before ]; (For the first set, you don’t need the initial comma.) Save.
  13. Refresh the map webpage – another island will have been highlighted.

2014_02_06 encore

I couldn’t leave it alone this evening – 5 hours after I got home and I’ve done the 6 LAs on Scotland’s mid-east coast. The wee bit that was missing from Aberdeen has been restored. (I processed that dataset again and the missing points magically reappeared. So here it is in glorious technicolour:

Screen Shot 2014-02-07 at 04.46.30

The points are very close to the LA boundaries on OpenStreetMap tiles, which is quite encouraging. The dashed line is OpenStreetMap and the solid line is Leaflet’s rendering of the converted data.

Screen Shot 2014-02-07 at 04.46.12

Fife is a 4-part multipolygon, that is there are four polygons in the bits that represent Fife. I guess these are the Isle of May, Inchkeith and Inchcolm. Each part of a multi polygon requires separate processing via the hero of the hour(s), http://gridreferencefinder.com/batchConvert/batchConvert.php.

So I’m dreading doing Shetland, Orkney and the Western Isles. But perhaps they don’t need to be marked or coloured because they are clearly separate from the mainland.

Less heroic have been Mac TextEdit (no RegEx), TextMate (one successful RegEx, then fell over and won’t stop trying to process a long finished-with Angus file) and Word (it has done most of the heavy kiting but has fallen over at least 10 times). I’ll forgive Eclipse for chugging away with RegExes very slowly but I won’t forgive it for completing the RegEx then hanging so I can’t save the result.

The optimal process seems to be

  1. Open the original data file in Word.
  2. Find any sub-polygons, by replacing  ] ], [ [ with multiple <new line>s.
  3. Save.
  4. Grep ], [ with ,<new line>
  5. Save.
  6. Get rid of the dross at the start and end of the of the file, so that I’m left with just sets of [number,number],<new line>[number,number], …[number,number],<new line>[number,number].
  7. Save.
  8. Manually copy and paste the first set into the heroic web page, and tell it to convert. If it moans, there is more than one set in what I’ve copied, so go back and sort it out. At this point, Word will fall over – hence the multiple saves,
  9. When a clean data set is in the hero, it will convert the data to lines containing some unwanted blurb and the data in LatLong format.
  10. Copy that bit to a new TextEdit file called <LA-name>.js, then leave some spaces.
  11. Convert the next data set, i.e the next polygon in the multipolygon.
  12. Copy and paste that data into the TextEdit file.
  13. Repeat steps 8-12 until all polygons are in the TextEdit file. Save and close it.
  14. Open it in Eclipse. Do the RegEx to remove the unwanted blurb. (To play it safe, do a few hundred lines at a time, saving each time.
  15. Save and close the file – Eclipse is about to fall over.
  16. Open the file in Word again, then replace all multiple new lines with a suitable swearword.
  17. Replace all new lines with ], [
  18. Replace the swearwords with ], [
  19. Top and tail the file so that it has the format var <LAname>  = [ [ [number, number], … [number, number] ],  [ [number, number], … [number, number] ], …  [ [number, number], … [number, number] ] ]; 
  20. Save it in the right place in the website data structure, add the necessary call to index.html’s head and the necessary Leaflet call to drawMap.js.
  21. Open the web page and draw the map.
  22. Fix the colours in drawMap.js
  23. Move on to the next LA!

2014_02_06

  1. got UK local authority data from https://www.sharegeo.ac.uk/handle/10672/305
  2. converted .shx file to .js using ogr2ogr -f “GEOJSON” newfilename.js sourcefilename.shx
  3. open resulting 88MB file in Word – around 10,000 pages.
  4. Find first mention of Scotland – it’s about 7000 pages in. It’s to do with Angus.
  5. Find second mention of Scotland. This is in the code marking the start of the data for Clackmannanshire.
  6. Select all the data for Angus, then cut it out and paste it to a new document. Save that as Angus.txt.
  7. Repeat steps 4 to 6 to get data files for all 32 Scottish LAs.
  8. Such a shame they are in National Grid format, not lat/long.
  9. proj4leaflet claims to handle other projections. Trying it but the huge size of the clackmannashire data file seems to be killing Eclipse. So have abandoned that, going back to what we have so far and trying to convert NG co-ords to latlong.
  10. http://gridreferencefinder.com/batchConvert/batchConvert.php batch converts but there’s a lot of Word/RegEx jiggery pokery to prepare the stuff for batch-conversion, then convert the results to array format.
  11. It almost works for Aberdeen, but a few data-points seem to be missing.Screen Shot 2014-02-06 at 19.00.04Screen Shot 2014-02-06 at 18.57.41
  12.  (Ignore the blue line and shading in the first screenshot – it’s part of a very rough outline of the UK, from when I was trying to get to grips with geoJSON use in Leaflet.Screen Shot 2014-02-06 at 19.01.54
  13. The conversion step is choking my mac, so time to ask the internet if anyone has LatLong shapefiles for the Scottish LAs.
  14. Here endeth today’s lesson.