Nov 4, 2010
Over the opendata rainbow
Every pantomime needs a villain. Tuesday’s was clearly SOCITM – the association of local government IT leaders, famous for labyrinthine procurements, unnatural desires for mainframes, banning Facebook from the desktop and wearing Simpsons ties in the office*. Oh, and for riding a coach and horses through the spirit of freely-available data and support for small businesses in bidding for public sector contracts.
Except I don’t think they did. It’s a more complex story. And one we need to unpack to make sense of the emerging open/transparent/free data agenda. Note the deliberate scattering of terms here…
This is the background piece that sparked my interest. I’ll set out what I understand to be the facts (happy to be corrected on any assumptions if needed) before exploring the implications:
– Councils have been making details of “Supplier, Product and Usage” – including procurement – information freely available on the web (openly, useably, and without charge, just to spell it out) as part of the requirement to open up their data. But they’ve done it individually, via their own websites, or feeds, or whatever. [This is an assumption – it may have been a more sophisticated data submission and collection exercise. But the following general points would still stand.]
– The London Borough of Brent had been providing an aggregation service for this open, distributed information, compiling something known as the e-Government Register (eGR). This was provided, and funded, by Brent until June this year.
– It wasn’t sustainable for one Borough to keep funding the eGR itself. A similar register was being maintained by SOCITM. The preferred outcome was to merge the two registers into a single, SOCITM-managed one.
– Organisations providing information to the register receive access to the whole register for free. Non-contributing public sector organisations pay a smaller subscription for it. Private sector organisations, including SMEs (small/medium-sized enterprises) pay a larger subscription of £800.
So on the face of it we have something that was supposed to be transparent and free, now locked behind a paywall. We see the rapacious SOCITM portrayed as cruel exploiters of market hunger for information, setting aside transparency in pursuit of a fast buck.
But the crucial point here is that “freely available” and “free of charge to the ultimate user” aren’t the same thing. And the product that SOCITM is peddling is its own creation, albeit based on freely available raw materials. The addition of value through aggregation and presentation in one place justifies, I’m sure they would argue, a fair recovery of costs (or indeed whatever other price they feel the market could bear).
And these misapprehensions over what “free”, “transparent” and “open” mean here are rather worrying, because when we remember the commitments made in the Conservative Party Technology Manifesto earlier this year, we see that a figure of £6bn in “additional value to the UK” is supposed to be realised through the opening up of data. Some of that will be based on assumptions around making business easier to conduct, streamlining supply chains and other processes, sure – but some of it refers to the business opportunities for taking freely available raw data and packaging it into more useful things. As a commercial enterprise.
SOCITM isn’t perhaps an obviously “commercial” business, of course. Despite its .gov.uk domain it isn’t “government” as such, either. It’s run along fairly standard trade association lines. But what it is doing here – and the reaction of the market – provides a useful barometer for what I am sure will be many more “resold” information utilities following on from the release of raw data.
Now, the question of whether such a service as the e-Government Register should be centrally funded by government to support SME business growth still remains. But it is a business subsidy question – one for BIS to consider, probably, rather than an open data one. (I now refer you back up to my sleight of hand in the last sentence of the first paragraph, conflating as I did two distinct agendas in one smoooth line.)
I’d argue that a centrally-funded eGR would represent a government intervention to support business, in the same way that rent might be subsidised for startups, or bank lending encouraged. Do we even know that it would be intervention in the face of market failure? Is £800 really too high a price for SMEs in relation to the business opportunities that might ensue? (SMEs would argue yes, naturally, but they could of course see how much it would cost to do their own council-by-council harvesting of opportunities instead…)
But I don’t think there is a transparency failure here as such. I do think we’re going to see a lot more shocked reactions as free raw materials are turned into paid-for useful products. Who knew, hey?
What do you think?
*I used to be a local government IT manager. I can say these things.
————————
Update: the point about “is all the information in the eGR also available at no charge in a raw format elsewhere, albeit scattered” is an important one. I’ve left the following comment on the article which broke the story to me:
Fascinating case study. Some very powerful themes in here about what ‘free’ data actually means. And, I’d say, some misunderstandings in government as well as the wider information community about the agendas involved. I’ve written some more detailed thoughts at https://paulclarke.com/honestlyreal/2010/11/over-the-opendata-rainbow/
Can I just confirm Vicky (hi Vicky) that SOCITM are asserting that every item of data in the Register is available, free and unfettered, somewhere else (albeit on many different sites) to any public enquirer who should choose to look for it?
On this point turns much of the argument about what SOCITM are doing. If it turned out that some of the data is supplied directly by organisations to SOCITM, bypassing free, raw release – perhaps as part of SOCITM’s own data collection activities – that would put quite a different gloss on the matter.
Update 24 November: Ken Chad has dropped me an email describing the comparable situation of technology that is used in libraries, and how the LGLibTech wiki service provides a way to capture and share this information. He notes, with regret, that the wiki used to link through from each authority’s entry to the eGR, but following the changes which sparked this post, this functionality is no longer available. He writes more here about his approach to two library technology wikis he has been involved with. (LGLibTech has a sister Higher Education website: HELibTech.)
I also raised an eyebrow when I got an email inviting me to pay for access to this register. It is difficult if SOCITM are adding value to the raw data with this service, or if the process itself add value by standardising, normalising and cleansing the data, but if the raw data is not available anywhere else, then we shall never know.
I also suspect that there is considerable value to be added to the pubsec IT procurement community by people who have no vested interest in any one particular product or service. If that analysis is always hidden behind software vendors firewalls, then the community is less able to get more effective solutions and better value for money.
Saul Cozens
I’ve asked, via the comments on the blogpost referenced above, for some SOCITM clarification that ALL data in the register is also available (albeit in disparate places and formats) for raw consumption, free of charge, publicly. As Vicky Sargent has been commenting there on behalf of SOCITM I look forward to their response.
Interesting post Paul – it does raise some good points – I would add that as an SME we found the Government Register (eGR) that Dane Wright, aka pluto9 created as part of the now defunct National Projects,(I believe) was a hugely useful resource however I would say that we would also add data to help aid its accuracy, we had a vested interest after all. I for one feel disappointed that it has disappeared behind its own pay wall and would like to see it back but time will tell if it proves as successful as it once was….
Paul,
You seem to be a smart chap with the internets.
Pick a random council and tell me how long it takes you to find out who supplies their payroll software.
Then tell me how long it takes you to find all the other councils that use the same payroll software supplier.
You make a good point, as ever. I wonder if that isn’t two distinct cases though: openness at individual organisation level, and the “big picture” through aggregation.
I am somewhat reluctant to generalise (for once!) but would a possible set of principles be:
1. unfettered openness at organisational level is an expectation
2. aggregation adds value and will almost certainly have costs, which will need to be recovered
3. in certain cases, these costs may be centrally funded as a common good stimulus, or in a case of market failure
How’s that sound? We’ve got 1 (maybe – still checking), and we’re debating here whether the aggregate picture of local government supply is important enough to qualify as a 3.
Paul – thanks for attempting to unpack what you refer to as the ‘emerging open/transparent/free data agenda’ because you are right, I think a lot of people are getting confused about some of the issues.
The data Socitm is collecting and maintaining for the Applications Register is provided by local authorities direct to Socitm. They do so because they find it useful to share the information with other local authorities, partly to compare information on suppliers and partly (by examining renewal dates) to enable opportunities for jointly procuring and sharing software.
Socitm has been providing this service for many years through its own Software Index, established long before the egov register was set up as a product of the national projects programme and then merged into our activity when Brent was no longer able to support it.
The data we gather is not ours to publish as open data in the spirit of the Government’s transparency programme. Supportive as Socitm is of the transparency programme, it is for local authorities to decide whether they want to do this and how they want to do it. Right now they are focussing on publishing spending data, contracts and tenders information (from which I guess you could glean data on software contracts), salaries and job titles, council minutes and a few other things the CLG is urging them to do. Publishing their software asset register may not be an immediate priority.
If they do decide to do that at some point, Socitm might then chose to collect this open data and add value to it and publish the result as a service – in just the way the Government envisages organisations making use of such data. SMEs would have the choice of trawling the open data for ‘free’ (any business run on commercial lines that has tried anything similar knows the cost) or subscribing to an aggregation/value added service.
Until such time as that happens, Socitm will carry on providing through the Applications Register a service that local authorities find sufficiently useful to contribute to, and which many private sector organisations also seem to find sufficiently useful to pay for.
It’s the same situation that was happening with Spikes Cavell and various councils’ financial data. The councils weren’t publishing the data in a raw form but giving it straight to a commercial third party who were only delivering it through their own app under a very restrictive set of terms.
Linked open data makes it possible and in many cases easy and automatic to aggregate data from multiple sources. In this case, the councils should either publish their own data individually in a reusable format or centrally fund an aggregation process that publishes all its results openly. There is plenty of scope thereafter–indeed more scope–for others to “add value” by visualising, analysing and combining that data with other datasets than by the current approach.
Creating a monopoly around a dataset is inimical to both transparency and economic opportunity. Government shouldn’t create a privileged relationship with anyone for the dissemination of its data. If they do, you’ll have no competition and therefore stagnation.
Who owns the copyright to the work?
So, first of all, I’m not by any stretch an expert here, so I’d welcome correction from those more informed.In what sense do you think that the data is protected?Let’s put aside that copyright only applies to the fixation of a work and assume it applies to data. Although that’s a huge concession in itself.It would seem that, in this case, copyright doesn’t apply – the work is not original, not literary, dramatic, artistic nor a recording or broadcast. Authorities can hardly be said to have copyright rights over other businesses’ names and addresses, right?The database right could apply to what Socitm is doing. But information on suppliers collected by authorities is collected as a by-product of doing business with them. I believe that means that they’re unlikely to have any rights in that database.Not that rights have any particular bearing on whether you can sell access or otherwise to your database. You’re welcome to do so, as a publisher is welcome to make and sell copies of Shakespeare’s plays.I think talk of copyright and ownership of the data is a bit of a red herring here. Would be happy to be proved wrong!
I note statements that it will be free for the public sector and charged for the private sector. I cannot see where accommodation has been made for third sector i.e. non-commercial purposes?
Third sector can participate on the same basis as the public sector – we will change descriptive statements about the AR to make this clear.
Adrian, the you say ‘the work’ what, specifically, are you referring to?
I don’t think I can really add much to this by way of wrap-up here. My first assumption in the original post was clearly wrong: this is not open data which has been published by individual authorities and then scraped together by SOCITM. It is a specific submission/collection enterprise. And it concerns software assets, rather than the better-known data sets that are being trumpeted about.
I have less sympathy for the line that the data isn’t SOCITM’s to release – I can’t imagine any untoward consequence in this climate of increasing openness (or at least political push in that direction). However, it would destroy the income stream. And would result in the collection not being possible any more.
So: fair to berate an organisation for carrying out a cost-recovery data collection activity? Not really.
Right to insist they continue to do it for free? Probably not, as then it wouldn’t exist at all. If it could have continued under the preceding arrangements I’m sure it would have done.
Good idea to lobby for central funding (from BIS or CLG) to keep this free to recipients? Maybe, but it would have to go some in the current environment to qualify as a funding priority.
Sensible to keep on at councils for not opening up the information in the first place, thereby rendering superfluous much of the resulting flapping described above? Absolutely.
Excellent summary Paul.
It would be interesting to know though, from the cost recovery perspective, who funded the work to enable the Socitm/Brent databases to be merged.
Ken Chad Consulting will launch open (free of charge and re-usable) specifications for library systems in December. Openly sharing these complex specifications with *anyone* who is interested should reduce procurement costs, make a time consuming (and dull) task easier and promote dialogue to improve functionlaity.