Tag Archive for 'mapping'

Following Tweeters


The world never stops amazing me, and this time (again) its the QGis community with a new Timeline tool to map geo-coded information over time – very cool but what to do with it?

With a very basic understanding of the Titter API and scripts openly available (remember, I’m no programmer) I captured a morning’s worth of Twitter data and plotted the geo-coded tweets as a time-lapse sequence.

httpv://www.youtube.com/watch?v=PWMkT5y0uhw

I’ll admit its not the most interesting video ever but it does show the power of opensource tools. Not a penny was spent on any of the tools and the base maps; all were Opensource or open-data.

 

 

Visualising NGA broadband


I was recently drawn to the Ordinance Survey’s blog where they had tweeted on image they had received which visualised the postcodes of Great Britain is a rather artistic way. Wondering if this approach was easy to replicate and if other data could be used I had a little play with the DCLG model I’ve used before to see if it was possible to create a short animated sequence which could show the spread of faster .

NGA broadband growth

Click on the image to see it run – it loops back at the end to highlight the gulf between where we’re starting from to where we need to get to.

Its not as polished as the OS image but I think it kind of works. The DCLG data is modelled on the ONS’s super output areas, so I resolved them to postcode level to give more points of light, and the colour simply matches the model’s traffic lights.

At the moment the image only covers England and Wales – Scotland and Northern Ireland have their own statistical output area systems which individually need resolving to postcode level. If I get a quite moment I’ll run additional areas to make the map complete.

People, politics and technology


MIT recently published a fascinating piece of research, looking at how social interactions can help to define regions based on a massive sample of land-line telephone calls across Great Britain . They used the anonymised information on the 12 billions phone calls made in Britain in a typical month to see if there were any patterns which could describe natural regions based on human interactions. The results are quire extraordinary!

A quick glance at the resulting maps will tell you that Scots only really talk to other Scots (left-most map below), similarly Londoners but to a lesser extent. The rest of the country tells a rather different story, challenging some of our traditional assumptions about .

Three iterations of partitioning

The right-hand map shows an optimised partitioning of the call data, showing that, for example, Welsh people fall into three regions, only one of  which is solely in Wales. North Wales communicates most strongly with Manchester and the southern part of what we normally associate with the North West; while Mid-Wales links most naturally with the West Midlands. A fair conclusion from this is that if the UK were to be fully devolved, it would make little difference to the day to day communications of the Scottish population, but it would have a profound impact on the Welsh population.

The Yorkshire-Lancashire rivalry also takes a bit of a knock, with West Yorkshire more likely to communicate with the people of Lancashire than their White Rose brethren; and the more rural Cumbrians are perhaps a mini region of their own.

In checking the validity of their approach, the researches aggregated a number of alternative partitioning models, and this generated additional insight into regional identity.

NUTS regions overlaid on the aggregated partitioning models To the west of London the team identified what they consider to be a new region in the making – a Western Crescent formed of , Berkshire and Buckinghamshire. This is the heart of England’s high-tech industries with the Silicon Corridor along the M4 and ’s Science Vale with Oxford University and Harwell. What it interesting is the ambiguity of the areas communication patterns – rather than having a very clear and arguably insular regional identity, this Western Crescent is something of a communications hub, reaching out to much of central England.

Why is this important?

The traditional regional boundaries being largely consigned to civic roles with political and economic control being passed to new Local Enterprise Partnerships. What these maps suggest is that the regional identities were already being challenged and that perhaps the more fluid LEP structure would be more able to mould itself to our day to day lives. While the South England region, which spanned Kent and Oxfordshire, meant very little to anyone except central , an Oxfordshire LEP able to partner with a Thames Valley LEP may be more successful.

And from my own personal perspective such an approach also means its possible to map telecommunications networks to human interactions. The formative signs of a new high-technology region around Oxfordshire sure deserves a commensurate infrastructure? And its role as a natural communications hub surely makes it the place to start building the future? The research should also have a big impact on Cumbria’s “vanguard”, and ought to shape Herefordshire’s thinking as they develop their pilot.

As I start to work with the new Oxfordshire LEP on their approach to broadband I’m sure this research will become something we refer back to.

How successful would Finland’s broadband policy be here?


At the NextGen Road-show event in Edinburgh this week, Professor Michael Fourman gave a fascinating talk on the special challenges for delivering in Scotland. At the heart of his work were some maps which very effectively demonstrated the impact the Finnish ’s policy might have on some of the more remote areas of Scotland as well as a -based estimate of how much it might cost to deliver it.

Heavily summarised, Finland’s policy says that there should be a  back-haul connection within 2km of any community; and they define a community as an area containing at least 70 people per square kilometre.

I was left wondering how effective this policy might be across England and Wales, as well as Scotland.  I don’t have to hand the core network details that Prof. Fourman used to calculate the costs of delivering the policy nor the time just at the moment to build the shortest-distance spanning tree model he used, so I’ve restricted myself to simply looking at where Finland’s policy might reach that the market won’t.

Finland's broadband policy applied to England & Wales The map (click on it to see it life-size) depicts in green the areas which the policy would deliver a fibre to, and the black is the extent of market-led next generation broadband according to DCLG’s 65% model. A first glance says “so what – doesn’t seem very impressive”. However this is where maps have the power to overstate a problem. Using the 2001 census, there would be 11,946,819 (don’t you love computer precision!) English and Welsh people who remained without broadband when 65% of the UK was already enjoying it. Applying the Finnish policy reduces this figure to just 275,451 – or in other words, increases the reach of from 65% to 94% of the population.

The Finnish broadband policy would reach 94% of the English and Welsh population

Of course this is academic without the costs that Prof. Fourman generated, but it is a powerful example of how the village pump model that Rory Stewart MP is advocating. So how many of these green areas are close to a Primary School, Library or GP whose existing broadband connections could be upgraded and converted into a Village Pump?

Data is king – part 2


In the first part I looked the importance of using empirical data to support a local narrative when developing a policy by looking at some of the pitfalls of opinion polls on the state of a communities infrastructure. In this second part I start to look at the kind of data and exercises that can help target interventions.

The first exercise has to be to understand the state of now – what does the broadband landscape look like today? For me there is only one resource for this kind of information – Samknows.

Their data is all from primary resources – BT, Virgin, and the unbundlers – and is carefully cross-referenced against their own checks. Their data forms the backbone of the price comparison sites and their services are the basis of Ofcom broadband quality studies.

Over the last year or so I have done a number of projects with , mapping and analyzing their data. Each time it has thrown up new local insights and shone new light on the national picture. And each time it has helped to create a middle ground on which a less emotive debate can develop; when faced with the data its no longer really possible to simply blame BT or the regional , but equally it provides policy makers with an image of the broadband landscape around which its possible to plan.

Such a rich image is only possible with very high quality data. There have been many attempts to model broadband speeds based on paper models of ADSL performance curves from manufacturers and tools that calculate the radial, as the crow flies distance from the BT exchange.

Some of these have tried to build in factors for guessing the true cable length, quality and so on – but at the end of the day they are just increasingly smart guesses – and this becomes very clear when you try to map the result. Estimating broadband speeds based on the radial distance from the exchange will create nice, uniform shapes on a map from which one glance tell you its not a reality.

Mapping quality empirical data provides a more organic image of broadband performance which starts to mould itself to the geography and topography of the area. From this its possible to build a narrative which links the cold data with the tales of broadband woe from communities – and to slay a few myths along the way.

This can perhaps be best demonstrated by a piece of work I recently completed for County Council. This is my home county and I wanted to see if I could create a map able  to highlight some of the broadband stories I knew to be true, and from that to seek out new detail which may have lain slumbering for the reasons highlighted in part 1.

So as well as the nice clean maps which blanket fill a postcode polygon with traffic light colours to represent poor, mean and good broadband speeds , I ran the data through a series of GIS modules to create a contoured heat map of the county – while its harder to say precisely what the speed is at a given location, it does provide a much richer analogue from which the broadband landscape can be described.

Trialling broadband speed contours

Its clear that the underlying data is not based on a simple model and is far from producing the neat conical contours of a radial guesstimate. A quick glance shows how broadband speeds are affected by the contours of hills and valleys, and by man-made features like railways.

From the generalities I wanted to test some established broadband legends. There were long rumours that broadband in parts of central Oxford were slow, and the reasons given seemed perfectly plausible but unproven. The story was always that BT had centralised its various telephone exchanges in the city, which increased the cable lengths, and that in this part of the city the cables had to additionally skirt around the old Morris car plant making them too long to support a good broadband service, even though some of the homes and businesses affected my be just a few hundred metres from the newer telephone exchange.

A glance at the new map clearly shows a glacial valley of poorer broadband to the North East of Oxford’s broadband summit. So while the now BMW car plant is much more compact, the data appears to support tales of  the ghost of the city’s industrial past still be haunting of one of the world’s most important knowledge centres.

Once I’d calibrated my eye using a few known problem spots, it became easier to start hunting new broadband stories. Slightly to the north of the city is Oxford Airport. Following the northern perimeter fence the map predicts the existence of an ox-bow lake of poor broadband coverage – perfectly obvious when you think about it. Would you really feel happy allowing Openreach engineers exercising the code powers to dig a trench across the runway?

The granularity of Samknows data again helps to highlights what is a small, localised broadband issue which would be completely missed by lesser datasets.

So now when I work with the good people at Samknows, I still produce the coloured political maps of broadband speed using traffic light colours to highlight potential problem areas but now I include the contoured explorers map which helps to link the data with the local legends.

In the final part of this series, I’ll start to look at some of the other data models which help to add further colour to impact of poor broadband.

Data is king – part 1


I was recently involved in a study of services in rural areas involving both a study and a survey of businesses. The whole process threw up a fascinating insight into the problem of developing a useful and targeted broadband policy.

The received wisdom in this area was that a number of small towns were poorly served by broadband and the survey of local businesses largely supported this view. However it was strongly contradicted by the mapping exercise – this suggested quite the opposite. In an attempt to reconcile the difference, it was much easier to check the cold, hard data than to suggest to businesses that they might be mistaken, so we looked at the logic of the data – a town in question was a tight, nuclear market town and had its own telephone exchange at its centre which supported the data in pointing to a good broadband service. A line check on each of the business lines further supported the data, and finally a software speed checker corroborated the data. There remained little scope to support the business communities belief that they were poorly served by broadband.

So what was going on? A theory supported up by a conversation I had with a professional pollster ran along these lines:

Defined market towns tend to build up their own support structures which can lead to the community becoming reliant on a narrow and possibly isolated pool of expert advice; the more esoteric and scarce the skill, the greater the scope for that advice to be of less than the highest quality. Here a respected opinion can become the received wisdom and if this wisdom is proffered by a local IT company that isn’t, shall we say, as technically adept as perhaps it might be, then a local mythology can easily develop.

This mythology can then be readily communicated using the more effective mechanisms of a tight, well structured community giving a wider voice to the views of the town.

Contrast this with more sparsely populated areas where people tend to travel further to plug into support networks and different people may seek support in different directions. This is more likely to create a richer, more diverse advice network where myths are more readily challenged.

More sparsely populated communities are perhaps also more used to poorer infrastructure, and may have less effective communication channels. As a result, sparsely populated rural areas – relative to small towns – may under report their broadband problems.

I think the lesson from this is that while the narrative of communities is critical to developing a broadband policy in rural areas, it should be used to add colour and to personalise cold, empirical data. That the nature of the problem should be based on facts, while the narrative gives voice to the impact of that problem.

As the shape of the digital divide hardens, with urban areas set for “superfast” broadband while rural area stay pretty much as they are, the debate is increasingly becoming emotive – and rightly so. During this time its critical that the interventions, however, can be targeted on those of greatest need and not necessarily those with the greatest voice.

In part two I’ll start to look at how the data can be used to support a local narrative – to keep the problem defined technically, but giving a stronger voice to the people and business suffering from poor infrastructure.

A final third parliament


As we enter the final hours of the general , I thought I’d have a bit of fun and see what a Final Third Parliament might look like.

Seats are electable if they are within the 33% of constituencies in Great Britain (apologies to NI – I didn’t have the data) least likely to see investment in next generation according to the DCLG model. To make it partisan, I’ve used the results from the 2005 general election as, like eveyone else, I’ve no idea what might happen on Thursday.

While the Conservatives hold power over the Final Third Parliament, they fall short of an overall majority. On the ground, Scotland and Wales feature heavily: (click here for an interactive map – click on a constituency to see the % of red areas not likely to see investment)

The Constituencies without hope of NGA


The release of the data allowed me to play around with the Department for Communities & Local (DCLG) data on where they believe next generation is most likely to appear. Here’s my league table of the top 25 Parliamentary Constituencies least likely to see investment in :

ConstituencyRedAmberGreen
Dwyfor Meirionnydd Co Const100.00%0.00%0.00%
Kingston upon Hull East Boro Const100.00%0.00%0.00%
Kingston upon Hull West and Hessle Boro Const100.00%0.00%0.00%
Na h-Eileanan an Iar Co Const100.00%0.00%0.00%
Haltemprice and Howden Co Const96.77%3.23%0.00%
Beverley and Holderness Co Const95.74%4.26%0.00%
Central Devon Co Const95.24%4.76%0.00%
Montgomeryshire Co Const95.00%5.00%0.00%
Kingston upon Hull North Boro Const94.29%5.71%0.00%
Berwick-upon-Tweed Co Const93.55%6.45%0.00%
North Cornwall Co Const89.66%10.34%0.00%
North Norfolk Co Const89.47%10.53%0.00%
Derbyshire Dales Co Const88.89%11.11%0.00%
Harwich and North Essex Co Const88.24%11.76%0.00%
Thirsk and Malton Co Const88.24%11.76%0.00%
Dumfriesshire, Clydesdale and Tweeddale Co Const88.00%12.00%0.00%
Carmarthen East and Dinefwr Co Const87.50%12.50%0.00%
Delyn Co Const87.50%12.50%0.00%
Ynys Mon Co Const87.50%12.50%0.00%
North Herefordshire Co Const87.10%12.90%0.00%
Ross, Skye and Lochaber Co Const86.96%13.04%0.00%
Forest of Dean Co Const86.84%13.16%0.00%
Louth and Horncastle Co Const86.84%10.53%2.63%
Ludlow Co Const85.71%14.29%0.00%
Carmarthen West and South Pembrokeshire Co Const84.38%15.63%0.00%

The Red, Amber, Green columns are the percentage of each constituency unlikely, possibly, or probably a target for investment in “superfast” broadband.

There are, according to the DCLG model, 56 constituencies where they don’t expect any investment including the Speaker’s constituency in Buckingham. The leaders of the three main parties do better:

  • Gordon Brown – 50% of his Kirkcaldy constituency is likely to see investment
  • David Cameron – 55% of his Witney constituency is likely to see investment but the largely rural areas outside the market town won’t
  • Nick Clegg – 90% of his Sheffield constituency is likely to see investment, not least through the Digital Region project

As for the candidates who may be responsible for digital policy after the :

  • Stephen Timms serves one the hottest broadband constituencies in the UK – East Ham is 100% Green
  • Jeremy Hunt and Ed Vaizey’s Surrey and constituencies are both good in parts but rural areas mean broadband have’s and have not’s
  • Lynne Featherstone, chair of the Lib Dems Technology Board, serves another hot broadband area – Wood Green in London is 100% Green

Other names of note on the list include:

  • Social media convert John Presctott and Home Secretary, Alan Johnson appear at 2 and 3 on the list of notspots due to the peculiarity of Hull
  • East Yorkshire Tory MP and civil liberties campaigner, David Davis is 5th on the list with just 3% of his constituency possibly a target of investment
  • Liberal cheeky-boy Lembit Opik and past leader Charles Kennedy are high on the list because they serve rural areas of Wales and Scotland
  • Alex Salmond, leader of the SNP’s (who don’t mention broadband in their ) serves the 27th least popular target for broadband investment
  • And leader of Plaid in Westminster, Elfyn Llwyd tops the list of NGA notpsots in the DCLG model as the constituency least likely to see broadband investment

To see how your constituency fairs, download the full list here in CSV format.

Think broadband’s important for the next Government? Click here for a quick breakdown of the main parties views on technology.

Dwyfor Meirionnydd Co Const100.00%0.00%0.00%
Kingston upon Hull East Boro Const100.00%0.00%0.00%
Kingston upon Hull West and Hessle Boro Const100.00%0.00%0.00%
Na h-Eileanan an Iar Co Const100.00%0.00%0.00%
Haltemprice and Howden Co Const96.77%3.23%0.00%
Beverley and Holderness Co Const95.74%4.26%0.00%
Central Devon Co Const95.24%4.76%0.00%
Montgomeryshire Co Const95.00%5.00%0.00%
Kingston upon Hull North Boro Const94.29%5.71%0.00%
Berwick-upon-Tweed Co Const93.55%6.45%0.00%
North Cornwall Co Const89.66%10.34%0.00%
North Norfolk Co Const89.47%10.53%0.00%
Derbyshire Dales Co Const88.89%11.11%0.00%
Harwich and North Essex Co Const88.24%11.76%0.00%
Thirsk and Malton Co Const88.24%11.76%0.00%
Dumfriesshire, Clydesdale and Tweeddale Co Const88.00%12.00%0.00%
Carmarthen East and Dinefwr Co Const87.50%12.50%0.00%
Delyn Co Const87.50%12.50%0.00%
Ynys Mon Co Const87.50%12.50%0.00%
North Herefordshire Co Const87.10%12.90%0.00%
Ross, Skye and Lochaber Co Const86.96%13.04%0.00%
Forest of Dean Co Const86.84%13.16%0.00%
Louth and Horncastle Co Const86.84%10.53%2.63%
Ludlow Co Const85.71%14.29%0.00%
Carmarthen West and South Pembrokeshire Co Const84.38%15.63%0.00%

Wow!!


I’ve been dabbling in systems for a while now, looking at how and social data can be combined to better understand the nature of the digital divide, and to just simply understand what the landscape really looks like.

For broadband information there is only one source of reliable primary data – .

For reliable social data the ONS is pretty good although finding exactly what you want can be a bit of chore  but  with the new data.gov.uk website this is only going to get better.

But at some point you need to combine all this information on a map. The UK and Ireland are fairly unique here in having a service that produces fantastically accurate and useful maps but the down side of this is that for most applications today this level of detail is rarely necessary but and the process is mind bogglingly expensive. The impact has been that I could find perfectly adequate maps of almost anywhere in the world to model my data but for a long time really struggled to find an affordable compromise in the UK.

I then started to use Openstreetmap – an /creative commons mapping project which has through leaps and bounds got better and better. With the entire world held in a database on my machine I’m able to produce perfectly reasonable maps for most of the work I do – except in the most important areas I need to understand – rural areas. Openstreetmap relies on the goodwill of its supporters to trace using GPS the areas it maps – fewer people live in rural areas so naturally less of it is mapped well.

Its felt like the OS have fought to retain their right to charge very large sums to anyone wanting access to their data, regardless of the use and need, so it was with a sense of cynicism I decided to take a peek at the ’s Opendata website launched last week – the place the OS have released some of their map information to the public.

There is only one word which really captured what I found:

WOW!!

There are geo-coded maps of various scales in glorious detail and superb quality ready to be loaded straight into my tools.

There is a variety of GIS files which provide any number of other locational resources including parliamentary constituency boundaries, councils, the lot.

And there is a file which says where each and every postcode is

So like a child in a sweetshop I delved in, downloading all the files which I’d wanted for so long but had to cobble together from secondary sources – now I had them in the original form from the most respected map makers in the world.

But then a problem – one of the files, the one I really wanted containing postcode data, didn’t download. So I dropped the OS a line expecting an automated reply in a day or two, leading to some perfunctory reply in a few days to say it was really my fault and to try again.

How wrong – within a few minutes I got an apologetic mail from Jamie, one of their developers on their help desk, asking a few sensible questions and we exchanged a few more emails before his shift finished, when I got a new thread from Dr Paul who picked the problem up until it was fixed. It seems a bit of test data in their Goliath system had refused to be flushed from a cache somewhere which given the scale of their launch is a pretty minor problem.

The launch of the data is absolutely fantastic – their support during the launch is something else!

Hats off to the Ordinance Survey – I’m off to do some mapping!



Get Adobe Flash playerPlugin by wpburn.com wordpress themes

Switch to our mobile site

Page optimized by WP Minify WordPress Plugin