2014_02_08

Posted on February 8, 2014 by Bruce Ryan

Just started processing Shetland shape file. There are 549 polygons. There are 201,158 co-ordinates. Updated process:

Prepare a Word text file called Shetland.js which contains only var shetland = [ ];
In a copy of the file containing the data for Shetland, replace all ] ], [ [ with two <new line> in Word. Save.
In the same file, replace all ], [ with <new line> in Word. Save.
Get rid of dross at beginning and end of file. Save. You now have just co-ordinates, with a double <new line> between the set for each island.
Find first double <new line>.
Cut all co-ordinates preceding this. Save – you don’t want to risk doing this set twice.
Paste the clipboard into the online batch converter, then convert.
Copy the converted data into TextMate. Save.
Do the first RegEx to get rid of the dross from the conversion step. Save.
Do the second RegEx to replace all <new line>s with ], [, then save.
Add , [ [ to the beginning of the Textmate file, and ] ] at the end. Save.
Copy the whole of the TextMate file and paste it into shetland.js just before ]; (For the first set, you don’t need the initial comma.) Save.
Refresh the map webpage – another island will have been highlighted.

2014_02_06 encore

Posted on February 7, 2014 by Bruce Ryan

I couldn’t leave it alone this evening – 5 hours after I got home and I’ve done the 6 LAs on Scotland’s mid-east coast. The wee bit that was missing from Aberdeen has been restored. (I processed that dataset again and the missing points magically reappeared. So here it is in glorious technicolour:

The points are very close to the LA boundaries on OpenStreetMap tiles, which is quite encouraging. The dashed line is OpenStreetMap and the solid line is Leaflet’s rendering of the converted data.

Fife is a 4-part multipolygon, that is there are four polygons in the bits that represent Fife. I guess these are the Isle of May, Inchkeith and Inchcolm. Each part of a multi polygon requires separate processing via the hero of the hour(s), http://gridreferencefinder.com/batchConvert/batchConvert.php.

So I’m dreading doing Shetland, Orkney and the Western Isles. But perhaps they don’t need to be marked or coloured because they are clearly separate from the mainland.

Less heroic have been Mac TextEdit (no RegEx), TextMate (one successful RegEx, then fell over and won’t stop trying to process a long finished-with Angus file) and Word (it has done most of the heavy kiting but has fallen over at least 10 times). I’ll forgive Eclipse for chugging away with RegExes very slowly but I won’t forgive it for completing the RegEx then hanging so I can’t save the result.

The optimal process seems to be

Open the original data file in Word.
Find any sub-polygons, by replacing ] ], [ [ with multiple <new line>s.
Save.
Grep ], [ with ,<new line>
Save.
Get rid of the dross at the start and end of the of the file, so that I’m left with just sets of [number,number],<new line>[number,number], …[number,number],<new line>[number,number].
Save.
Manually copy and paste the first set into the heroic web page, and tell it to convert. If it moans, there is more than one set in what I’ve copied, so go back and sort it out. At this point, Word will fall over – hence the multiple saves,
When a clean data set is in the hero, it will convert the data to lines containing some unwanted blurb and the data in LatLong format.
Copy that bit to a new TextEdit file called <LA-name>.js, then leave some spaces.
Convert the next data set, i.e the next polygon in the multipolygon.
Copy and paste that data into the TextEdit file.
Repeat steps 8-12 until all polygons are in the TextEdit file. Save and close it.
Open it in Eclipse. Do the RegEx to remove the unwanted blurb. (To play it safe, do a few hundred lines at a time, saving each time.
Save and close the file – Eclipse is about to fall over.
Open the file in Word again, then replace all multiple new lines with a suitable swearword.
Replace all new lines with ], [
Replace the swearwords with ], [
Top and tail the file so that it has the format var <LAname> = [ [ [number, number], … [number, number] ], [ [number, number], … [number, number] ], … [ [number, number], … [number, number] ] ];
Save it in the right place in the website data structure, add the necessary call to index.html’s head and the necessary Leaflet call to drawMap.js.
Open the web page and draw the map.
Fix the colours in drawMap.js
Move on to the next LA!

2014_02_06

Posted on February 6, 2014 by Bruce Ryan

got UK local authority data from https://www.sharegeo.ac.uk/handle/10672/305
converted .shx file to .js using ogr2ogr -f “GEOJSON” newfilename.js sourcefilename.shx
open resulting 88MB file in Word – around 10,000 pages.
Find first mention of Scotland – it’s about 7000 pages in. It’s to do with Angus.
Find second mention of Scotland. This is in the code marking the start of the data for Clackmannanshire.
Select all the data for Angus, then cut it out and paste it to a new document. Save that as Angus.txt.
Repeat steps 4 to 6 to get data files for all 32 Scottish LAs.
Such a shame they are in National Grid format, not lat/long.
proj4leaflet claims to handle other projections. Trying it but the huge size of the clackmannashire data file seems to be killing Eclipse. So have abandoned that, going back to what we have so far and trying to convert NG co-ords to latlong.
http://gridreferencefinder.com/batchConvert/batchConvert.php batch converts but there’s a lot of Word/RegEx jiggery pokery to prepare the stuff for batch-conversion, then convert the results to array format.
It almost works for Aberdeen, but a few data-points seem to be missing.
(Ignore the blue line and shading in the first screenshot – it’s part of a very rough outline of the UK, from when I was trying to get to grips with geoJSON use in Leaflet.
The conversion step is choking my mac, so time to ask the internet if anyone has LatLong shapefiles for the Scottish LAs.
Here endeth today’s lesson.

2014_02_05 Progress

Posted on February 5, 2014 by Bruce Ryan

I can’t remember much of last week. My diary tells me I was occupied every weekday evening. Monday, Wednesday and Thursday evenings were at gyms, with a brief visit to a social media surgery before Thursday evening’s gym session. Tuesday and Friday evenings were spent helping a community council set up its new WordPress blog.

I know I spent Thursday and Friday creating management tools for the two community council projects. This also took up some of Monday (3rd). Tuesday saw a very brief planning meeting, then I jumped on a train to Glasgow to meet the ever-delightful Heather Burns, then go to Ofcom’s Digital Participation round table, then come back from Glasgow to help the CC with its website again. (There isn’t any real problem with it – it’s a standard WordPress.com website but linked to a .co.uk domain. I haven’t yet set up email addresses using the domain – I suspect that might be a headache.)

Today I’ve sent data to the visualisation client so they can verify it. (The data is from 2011-12 and so is out of date – the client’s contacts should clean it up so the project’s website accurately portrays where things are.)

The latest version of the code shows I can display a polygon based on a geoJSON file (tutorial).

I also know I can obtain open shapefiles for LAs from the Ordnance Survey. The intervening step is to convert shapefiles to geoJon. Supposedly GDAL does that but I’ve not yet persuaded my mac to run it. Following this tutorial may help – but that’s for tomorrow!

Update

This tutorial shows how to actually use geoJSON files!

Monkey magic

Posted on January 26, 2014 by Bruce Ryan

Code Monkey

As of Friday morning, there is a verbal agreement between a client and Edinburgh Napier University that the university will create some code for the client. And I’m to be the main code-monkey! I’ll even be paid. Not much, which is fitting because I am a very new code-monkey and this will be my first ever paid coding project, and because the project is really quite small – just proof that the concept can work and that Napier can do things the client wants. Nevertheless I’m very pleased.

Domain devilry

It’s also pleasing that 1&1, the providers of the domain to which I’ve mapped this blog, have finally fixed the fault which was preventing this domain mapping. I bought the domain on 12 January and it’s taken until today to get the issue sorted. There’s also an email address (bruce AT bruceryan.info) but it’s simply a forward to my main personal address.

Community Council news 1

I’ve had another meeting with the community councillor who wanted to set up a website for his CC. There appeared to be two copies of http://<name of CC>cc.wordpress.com and one copy of http://<name of CC>.wordpress.com. Each of these had bits of the most up-to-date material but none had all of it. So we copied the most recent content to separate text files on his laptop, deleted the errant blogs from WordPress, started a new blog at http://<variant of name of CC>.wordpress.com, reset the them, remade the pages, tags and categories, then remade the posts. Setting up widgets went quite quickly. He was very pleased to see that WordPress blogs can include Twitter widgets. While his CC doesn’t yet have a Twitter account, he has a personal Twitter account and a personal blog. Adding the widget for the former to the latter took about 2 minutes.

He now has to take his efforts back to his CC for their approval, comments etc. We’ll then implement any desirable changes, set up and train a co-editor. Then I’ll keep a watching/mentor brief as the site develops. I’ve also interviewed the community councillor about his aspirations for the website, his level of ability and similar. In a few month’s time, I’ll interview him about what actually materialised – and that should be the basis of a research paper. It will also be of interest to others who need proof that the CC system is worthy of further investment in training.

Community Council news 2

I’ve been asked to take minutes for another Edinburgh CC. It will pay a small amount but will knock out one Wednesday spinning session a month, but that’s not a problem – I could replace it with a spin on Tuesday or thursday evenings. This CC also wants a website similar to the one I help run for Leith Central CC, so that should be easy enough to start off.

Research news

On Tuesday I’ll meet with my supervisor to plan some further research into community council websites. This will be an update of research we did in summer 2012. We’ve won a small grant for this research, enough to pay me for about 8 weeks. This should lead to a couple of research papers, I believe.

Physical space

My old G4/800 now has a new home way out west. It’s good to keep old silicon running.

Not yet started

a good practice guide for community council websites
writing papers based on my MSc dissertation
a serious attempt at setting up a domestic server

Watch this space.

Progress update

Posted on January 15, 2014 by Bruce Ryan

Monday saw a few more baby steps into JavaScript and leaflet, then I met with a CC chair who wants to set up a website for his CC. He now has a functional site he can show his colleagues and understands more about making posts, adding static pages, tags and categories. I need to write him a guide to the menu items in WordPress – this could be the basis of a section for the good practice guide I intend to create.

Continue reading →

Faltering baby steps

Posted on January 13, 2014 by Bruce Ryan

Working through the leaflet tutorial as my first step in mapping community councils. What we want is a website where you can enter your address and see the location (ideally the area) of your nearest community council.

Took me a while to remember how to call a javascript file from a webpage. This is going to be slow and tedious.

Bruce's IT-ish world

Category Archives: Code