What public data is already available?

To give the more technically minded some raw material and a headstart on getting any prototypes up and running, we have brought together here a huge quantity of information for reuse.

Several exciting new datasets are being make widely available for the first time, including  the Royal Mail's Postcode PAF file, Neighbourhood Statistics from the ONS, Health care information, from NHS Choices, a list of all schools in England and Wales from the DCSF and the Official Notices from the London Gazette.

Below you'll also find listed a wide range of other sources of public data, as well as some other useful APIs.

Please leave a comment if you know of any more sources of useful public data, and we'll add them to the list below. Have fun, and good luck...

---

NEW! Neighbourhood Statistics API from the Office of National Statistics

What’s there?
Neighbourhood statistics about almost everything.

More specifically, Neighbourhood Statistics (NeSS) data including 2001Census, Access to Services, Community wellbeing/Social Environment, Crime and Safety, Economic Deprivation, Education, Skills and Training, Health and care, Housing, Indicators, Indices of Deprivation, People and Society, Physical Environment, Work Deprivation.

Where do I find it?
http://neighbourhood.statistics.gov.uk/dissemination/Info.do?page=nde.htm

You may want to look at http://hublog.hubmed.org/archives/001689.html for advice on what's in there and how to use it.

What do I need?
Contact the ONS for an API key at better.info@ons.gov.uk
Obtain a free Click-Use PSI Licence to re-use Crown copyright information from http://www.opsi.gov.uk/click-use/system/online/pLogin.asp

---

NEW! Transport information from Transport Direct

What’s there?
Location of bus stops, stations etc and descriptions of the locations
The National Car Park Register
Journey planning information
A carbon calculator
The National Public Transport Data Repository feed

Where do I find it?
Access to the data is via a secure FTP service or API. To support the competition, Transport Direct have agreed to help users access the data on an individual basis. Some restrictions apply as not all the data available is owned by the government.

What do I need?

Email Nick Illsley at Transport Direct to access the data, e.g. for the FTP address and password or to access the API.

Given some of the restrictions, it will help Nick if you can say what you're planning to do.

Obtain a free Click-Use PSI Licence to re-use Crown copyright information from http://www.opsi.gov.uk/click-use/system/online/pLogin.asp

---

NEW! Health care services and information from the NHS

What’s there?
Information about health care service providers, and “live well” health information from the NHS.

Where do I find it?
http://www.nhs.uk/nhscwebservices/Pages/Webservices.aspx

What do I need?
Fill in the Web Services Registration Form and agree to the terms of use.

{Warning: it may take a few days for the NHS syndication team to get back to you. We're working on reducing this lag.}

Obtain a free Click-Use PSI Licence to re-use Crown copyright information from http://www.opsi.gov.uk/click-use/system/online/pLogin.asp

---

NEW! Notices from the London Gazette

What’s there?
All notices published in the London Gazette, the Government’s Official Journal and newspaper of record, in XML from February 2007 to May 2008. Notices types covered include State, Parliament, Ecclesiastical, Public Finance, Transport, Planning, Health, Environment, Water, Agriculture & Fisheries, Energy, Post & Telecom, Competition, Corporate Insolvency, Companies & Financial Regulations, Partnerships, Societies Regulation.

Where do I find it?
http://www.londongazette.co.uk/mashup/gazettesrdf.htm

What do I need?
Obtain a free Click-Use PSI Licence to re-use Crown copyright information from: http://www.opsi.gov.uk/click-use/system/online/pLogin.asp

---

NEW! Address Data from the Royal Mail

What’s there?
Sample raw data from the Postcode Address File. This will give you a comprehensive overview of the type, structure and format of the address data held by the Royal Mail, as well as the different data sets Royal Mail maintain.

The full PAF file is available, but you will need to check out the sample first (the full Postzon file is not available as part of this competition)

Where do I find it?
www.royalmail.com/addressproducts

What do I need?

You should email the Address Management Unit at address.management@royalmail.com 

There are few things they need to deal with your request quickly:

• 'Show Us A Better Way' in your subject heading
• Your address (so they can post the CD straight away)

Once you have had a look at what's on the sample CD, you can get in touch to request the full data set if required. This is a commercial product so you will only get one copy per person and there will be some terms and conditions to sign. We have spoken with Royal Mail and they will accept a scanned signature. Royal Mail will also ask you to link to the idea or fill in a short form so that they can get some feedback on this initiative.

---

NEW! A list of all the schools in England and Wales from the Department of Children, Schools and Families

What’s there?
A list of schools from Edubase. The data is in Excel format and contains a list of the schools, the address, the Local Authority, type of school and the age range served.

Update: Now available in SQL form thanks to the kind people at Free Our Data.

Where do I find it?
http://www.opsi.gov.uk/advice/poi/edubase300608.zip (3MB)

http://www.opsi.gov.uk/advice/poi/SUABW_schools_list_02072008.sql.zip

What do I need?
Obtain a free Click-Use PSI Licence to re-use Crown copyright information from: http://www.opsi.gov.uk/click-use/system/online/pLogin.asp

---

Mapping from Ordnance Survey

What’s there?

More detailed maps than are available via the popular Google Maps API– though these OS maps are strictly for non-commercial use only and access is limited to a few thousand queries per day. (If OS's licence or usage restrictions prove too onerous, you can also try OpenStreetMap, which does what it says on the tin and gets better by the day.)

Where do I find it?

http://openspace.ordnancesurvey.co.uk/openspace/

What do I need?

Register for an API key, using the form provided by Ordnance Survey.

---

Carbon Footprint Information from Defra (via AMEE – a 3rd party aggregator)

What’s there?
Carbon Footprint information from Defra, various UK Government agencies and others around the world.

Where do I find it?
http://www.amee.cc/
http://www.dgen.net/amee/signup.php (to get an AMEE API key)

What do I need?
An AMEE developer API key. Within reason, AMEE doesn’t aim to charge for access to the platform for projects that have no budget and are non-commercial or won’t have significant scale.

---

Public Spending information from HM Treasury

What’s there?
Tables and Statistics showing:

Public expenditure overview

Analyses of budgets

Differences from previous plans

Trends in public sector expenditure

Functional and economic category analyses of public sector expenditure

Central government own expenditure

Local authority expenditure

Public corporations

Analysis of public expenditure by country and region

Country expenditure by sub-function

Where do I find it?
http://www.hm-treasury.gov.uk/economic_data_and_tools/finance_spending_statistics/pes_publications/pespub_pesa08.cfm

What do I need?
Obtain a free Click-Use PSI Licence to re-use Crown copyright information from: http://www.opsi.gov.uk/click-use/system/online/pLogin.asp

---

Other Public Information Sources

BBC 3 Day Weather Forecast
http://backstage.bbc.co.uk/data/WeatherFeeds?v=p12
(List of all weather station forecasts as OPML feed http://www.dracos.co.uk/play/bbc-weather/countries.opml )

UK Road Travel Data from the BBC
http://www.bbc.co.uk/travelnews/xml/tpegml_en/rtm.xml

UK Public Transport Data from the BBC (Currently not working - 2/7/2008)
http://www.bbc.co.uk/travelnews/xml/tpegml_en/pti.xml

National Public Transport Access Node database (locations for all bus stops, train stations etc.)
http://www.naptan.org.uk/

National Public Transport Gazetteer (names of towns and places etc.)
http://www.nptg.org.uk/

TheyWorkForYou.com API (Parliamentary debates, MP details etc.)
http://www.theyworkforyou.com/api/

Parliament Parser project (List of all MPs, constituencies, names of Lords etc.)http://ukparse.kforge.net/parlparse/

DowningStreetSays.com Feed ("Every day the Prime Minister's Spokesperson meets a small coterie of political journalists known as 'the lobby' for a topical chat, or 'briefing'." This RSS feed is a transcript of these lobby briefings.)
http://downingstreetsays.com/feed

TellThemWhatYouThink.com (Government Consultations API, courtesy of Harry Metcalfe)
http://tellthemwhatyouthink.org/api/

FixMyStreet.com (geoRSS Feeds of problems reported by the public to their local council)
http://www.fixmystreet.com/alert

The Government Says (feeds of announcements and press releases from dozens of Government departments and other public bodies).  By Sam Smith and friends.
http://www.thegovernmentsays.com/

PlanningAlerts.com API (planning applications submitted to most local authorities, courtesy of Richard Pope)
http://www.planningalerts.com/apihowto.php

BBC Backstage Feeds (BBC News feeds, BBC Search API, TV & Radio schedules etc.)
http://backstage.bbc.co.uk/data/Data

BBC Programmes data
http://www.bbc.co.uk/programmes/developers#findoutmore

Other useful APIs

Google APIs (Google Maps is particularly useful)
http://code.google.com/more/#products-geo-maps

Yahoo! APIs
http://developer.yahoo.com/everything.html
(FireEagle is particularly useful http://fireeagle.yahoo.net/)

Wikipedia Data
http://download.wikimedia.org/

Wikipedia API
http://api.futef.com/apidocs.html

Flickr API (Photo sharing API)
http://www.flickr.com/services/

MOO.com API (Printing API)
http://www.moo.com/api/

Programmable Web Directory (all APIs known to man or beast)
http://www.programmableweb.com/apilist

Comments

[Jonathan Gray]

We've added many of these packages to http://ckan.net - a registry of knowledge packages.

For these and other resources for the UK, see:

http://ckan.net/tag/read/uk

If anyone knows of other UK government datasets (or other collections of resources) that are available for re-use please consider registering them on CKAN!

Regards,

Jonathan Gray
The Open Knowledge Foundation

[Jules]

Very interesting site.
From a Crystal Reports/SQL background new data to play around with is always a bonus.
I'll keep an eye out and see if I can suggest further ideas.

[Bill Chadwick]

OS OpenSpace API

@Mark: 1:25000 (Explorer) mapping is not available via the OS OpenSpace API, 1:50000 (Landranger) is available.

When zoomed right in, the API uses 1:10000 street map imagery without footpaths, bridleways, field boundaries etc. but with building outlines, a fantastically detailed set of waterways and lakes and a sprinkling of cave names.

The API does not include access to the OS's aerial imagery or terrain data (which would be particularly useful for planning hikes, bike rides etc). It does include the 1:50000 gazetteer which can be used to locate hills, lakes, woods, farms, towns, villages ... but not post codes.

[Bill Chadwick]

Free POI data

There's quite a bit at

http://www.poidownload.co.uk/

Accuracy? Freshness?

[Norman Teasdale]

It's very encouraging to see so many different data sources being opened up.

Would it be possible to at last persuade the Environment Agency to give access to their (or should I say OUR) extensive river level data for England and Wales? In Scotland SEPA have been doing this for a number of years -

http://www.sepa.org.uk/data/river_levels/data.htm

The Scottish Canoe Association use this daily information feed to drive a very useful web site for white water kayakers;

http://www.wheresthewater.com/

This resource has saved many a wasted drive to a river only to find it too low for canoeing.

A similar data feed for English and Welsh rivers would be of great use for canoeists, fishermen and many other river users.

[Mike Franklin]

Has anyone has a reply from Nick at Transport Direct yet? I'm still waiting for a reply..

Could anyone let me know what kind of format the data is in? I'm particually interested in Bus Stop locations and numbers.

mikee@hotmail.co.uk

Thanks

[Neil]

Any further news with the NHS, and the data not being opened up to individuals?

[Tom Loosemore]

@Normal and @Mike: We're chasing both issues, and don't anticipate any problems (other than sloth) . Apologies for the delays in sorting both issues out.

[Paul]

Not a new source of information but this site http://www.govtalk.gov.uk/ does define the specifications governing information flow across government departments and the XML schema structures they should be using.

[FatBoy6]

I think this whole exercise is a bit flawed and back to front.

There are already loads of people with interesting mash-up ideas, multiple existing data mash-up implemetations across the world and the technology is in-place and freely available.

The one component missing and the element holding development back in the UK is data access.

Until these government collected data sources are available to the public without condition this exercise is simply a talking shop. Release the data and the applications will follow!

Public data needs to be released to the public without licensing, commercial / legal agreements, logins & passwords and re-use confusion.

Why not do us all a favor and spend the 20k on getting the fundamental postcode to lat/lng database into the public domain?

The UK is in danger of falling behind on this one and missing out on a new wave of information driven web sites and technology that could generate jobs, skills and profit. Come on Gordon - give us our data!

[Peter]

We do a lot of work promoting books, authors, and publishers.

One thing that we'd love to be able to do is to not exclude people who can't afford to buy the books we promote.

There used to be (or still is?) in Scotland an API for the libary system: you could send it an ISBN (and a postcode) and it would return a list of (local) libraries where the book is in stock, or on loan.

Now, however, that information is only available to OCLC subscribers - which costs thousands a year.

[Bill Chadwick]

There are a few downloadable data sets such as SSSIs, CROW areas etc at

http://www.magic.gov.uk/

[Richard]

@Bill: The data on this link http://www.magic.gov.uk/datadoc/datadoc.asp appears to be under the same terms as the OS data as it is derived from their mastermap product.

http://www.magic.gov.uk/help/copyright.html

Is this right

Please let us know if we can add it to this list.

[Bill Chadwick]

@Richard: Surely this is an issue for ShowUsABetterWay and MAGIC (Defra, English Heritage, Natural England etc) to work out.

The copyright link you quote includes the words 'for commercial / business purposes' not sure whether this includes demos for your competition.

Also there are the words 'Non-profit and voluntary organisations wishing to include maps sourced from MAGIC on websites for information purposes only can do so, as long as the copyright statement is legible on the map. '

I am not sure that all the data sets have the same copyright.

This lack of clarity typifies the problems we have mashing data from government sources. I don't believe that government should require mashup developers to be copyright law experts.

I was actually looking for local/district council boundary polygons when I thought of posting about MAGIC here. Do you know if these are available anywhere for free use?

[Tom Loosemore]

@Fatboy: As requested, the full postcode PAF file is now available to those wanting to enter the competition. Details of how to get it here:

http://www.showusabetterway.co.uk/call/data.html#mail

Sorry to here you think the competition idea is back to front. I'd disagree. You need to *show* government that there are hundreds, thousands of ideas out there waiting to be turned into products if only more data were made available more freely. Simply *telling* them isn't enough to change the status quo.

That's why this site contains a mixture of new ideas from the public and new datasets from government.

Ying and yang, innit.

[Neil]

Glad we have PAF data, I've now requested access. . .will let you know how that goes.

Do you guys have swap with OPSI, they can cancel my request as it is now opened up!

[Gavin Jamie]

Great that Royal Mail postcodes are now available.

Any chance of the National Statistics Postcode Directory? This would link postcodes to statistical geography. I know this is possible through the web service but over 1000 web service requests to produce one web page make be impractical.

[Bill Chadwick]

Can we please have free access to a UK administrative area data set (GIS polygons), down to local authority level, for the purposes of this competition.

Thanks.

[FatBoy6]

Note that the PAF postcode file and other data sources are only released for use in this competition. Presumably you will have to shell out part (or all) of your 20k winnings to buy data licenses. Also, I presume, if you use the data you may be asked to license it after the competition has ended and remove any websites produced???? Please Clarify. As I said before, it would be much better to make this into a doing exercise rather than a talking shop. Forget about the competition - mashup ideas are all over the net - the key issue is about free access and rights to re-use data regardless of enduse.

[Bill Chadwick]

I agree with FatBoy.

The free to view community websites envisaged by this competition will not be practical without perpetually free, unconditional access to government data (Geo and other). Such sites need to be able to carry advertising to help pay for hosting and development costs. I believe that Government will have to accept this before anyone will bother developing them.

[Chris Boot]

Just noticed the following post:

"The UK Road Travel Data from the BBC page doesn't seem to work - gives "XML Parsing Error: undefined entity
Location: http://www.bbc.co.uk/travelnews/tpeg/en/local/rtm/rtm_tpeg.xml".

While this is the message being displayed on screen the underlying XML is fine (do a view source). I don't recommend accessing the XML directly if using on a web page though as it tends to be a bit sluggish since theres generally approximately 400 records in there. I've got a scheduled task to process the XML and store it to a database and then a stored procedure builds up the Google Map javascript, I've found this is quite possibly the quickest way to return this data.

[Dane Wright]

What about the National Land & Property Gazetteer (NLPG) and National Street Gazetteer (NSG)? See http://www.nlpg.org.uk and http://www.thensg.org.uk

The NLPG has more accurate and comprehensive property data than the PAF from Royal Mail and the NSG is the definitive database for street and road works information.

[Jamie Thompson]

I asked the NHS for access to the NHS Choices Web Service. Got an automated reply.. and then nothing. Go NHS!

I fail to understand why the web service isn't open. It's not like it contains anything sensitive. Indeed it contains nothing you can't get by screen-scraping their site.

Seems some parts of government are more open to change than others.

[Bill Chadwick]

No response more than a week after my request for local authority boundary data. How can we attribute environment problems (traffic, roads and streets etc) to the correct authority without this information ?

[Richard]

Bill,

Sorry for not picking up on this sooner. Have you seen the new OPSI data unlocking service?

http://www.opsi.gov.uk/unlocking-service/OPSIpage.aspx?page=UnlockIndex

That might be the best way for you to pursue this.