honestlyreal

Douglas Adams got there first, of course. He usually did.

In a universe where past and future lost their rigid meanings, and time was as capable of twists and reverses as any spatial dimension, Adams gave us Dr Dan Streetmentioner.

The good doctor realised the limitations of conventional grammar in describing these temporal quirks, so produced the Time Traveller’s Handbook of 1001 Tense Formations – an exhaustive treatise on how to describe them properly.

It will tell you for instance how to describe something that was about to happen to you in the past before you avoided it by time-jumping forward two days in order to avoid it. The event will be described differently according to whether you are talking about it from the standpoint of your own natural time, from a time in the further future, or a time in the further past and is further complicated by the possibility of conducting conversations whilst you are actually travelling from one time to another with the intention of becoming your own father or mother.

Most readers get as far as the Future Semi-Conditionally Modified Subinverted Plagal Past Subjunctive Intentional before giving up: and in fact in later editions of the book all the pages beyond this point have been left blank to save on printing costs.

I’m reminded of all this by something I read tonight about Lord Winston’s response to a noisy passenger on his train. I’ve done a few notes previously in similar territory of what is and isn’t (or might and might not be) ok when private things happen in public.

The concepts of “public” and “private” spaces simply aren’t as clear-cut as they used to be: social platforms enable all sorts of breaches in traditional conventions about what constitutes fair game in terms of the public gaze. How public is public, anyway? Local to a train carriage, to the audience of a prominent media figure, or to those with special interest in a particular hashtag?

The vocabulary just doesn’t exist to describe the nuances of things that would previously never be seen, but now are.

(I toyed with the ugly term “broad-availabling” when I first saw what happened when you brought together a search function and a Twitter account with only a handful of followers. In no way could our cellar-dwelling ranter be said to be broadcasting, but given a certain level of energy and enthusiasm, his reckons could find themselves as amplified as anything traditional media might emit.)

So we need a Dr Dan for the new realms of the publics and the privates, and all the lands that lie between. Might go a bit like this:

He’s been sharing details of our purribrate life with his mates again! – [Translation] My paramour told his friends what that gesture I do in my insta selfies really means.

Problem with the railways is everything goes to shit when they’re run by companies in the pubravaging sector. – [Translation] My train is late again because it’s operated through such a convoluted structure of joint ventures, subsidies and overseas quasi-governmental entities that nobody really knows who’s in charge.

If they dare to search my privennials before my job interview, I’ll report them to the ICO, so I will. – [Translation] I am concerned that my social media trail may have unforeseen repercussions.

How do you make something freely available so that anyone can use it, but also build sustainable businesses on top of it?

It’s an aspiration that drives Wikipedia, innumerable open web projects and, in recent years, the thrust of releasing UK government – and government-funded/subsidised – data for reuse.

It’s also a complicated balancing act in terms of basic economic theory: prices find their natural level, and if something’s available for free, there’s always going to be a tension in competing with it. If there’s no added value, there’s no sustainable business.

Of all the sectors where data has been opened up, it’s in transport that I think the most visible and tangible advances have been made.

Transport data has many lovely qualities about it – it’s highly structured in time and location; it has extreme real-time relevance; and it affects people.

I don’t doubt that concentrations of heavy metals in the soil affect people, but with the best will in the world, not in the same way – and not in a way that’s likely to affect what time they get up or which route they choose to get to work.

The web, and then the apps, revolutionised the way we consume travel information, but none of this could be possible without the underlying data.

So is the freeing up of data entirely without complications?

First, a small diversion into history.

Long ago when I tinkered with these things for a living, I was much taken with the power of the bottom-up service “Fix My Street” to allow people simply and quickly to report defects in their locality. A quick phone pic, an upload via an app, and the matter was routed to the relevant authority – putting the burden on them to receive, process, and respond – all the while knowing their response (or lack of it) would lie in public view.

My exam question at the time: should such a service be given oxygen through association with government’s “official” channels for doing stuff?

There were some curious arguments thrown at me at this point: “but we spent lots of money on our own sites – people should use them” [NOPE, NOT BUYING THAT]; “it will erode understanding of who actually provides services, and therefore local accountability at the ballot box becomes less clear” [ER, MAYBE A BIT? BUT SERIOUSLY?]; and “if you put information like that into public view, people will use it to find potholes, drive into them, and claim against the council” [*WTYRF?]

And yet, and yet. We are merely fallible humans, and if there’s a buck to be made… Which brings me, in a very roundabout way, to the Delay Repay Sniper.

The train operators – possibly out of a sense of decency and fairness, but more realistically under the thumb of the regulators – now operate money-back schemes for many of their services. If the train runs significantly late, or is cancelled, you can claim from them. My operator, like some others, calls it “Delay Repay”.

The burden is on the unfortunate traveller to work out what went wrong, and to make the claim. Sometimes this is easy (I’ve known train operators bring claim forms through the carriage, but that was pre-web days) but often it’s not. You are too busy trying to rescue your day to log the details of just how long you were delayed. Or things have got so chaotic that all concept of which actual train got cancelled/delayed is lost in a mire of misinformation.

If you’re not a season ticket holder then your claim is further complicated by having to dig out the precise fare you paid, which means finding the ticket that you chewed up in disgust after sitting outside New Cross Gate for 90 minutes.

So it’s very likely that Delay Repay is massively underclaimed in practice.

And, as the theory so rightly predicts, if there’s data, and there’s untapped value to be squeezed from it, there’s a business opportunity.

Some clever folk have built this Delay Repay Sniper (DRS) service to do just this.

They get a feed of data from Network Rail every day. For a very modest monthly subscription they will then crunch it around to make sense of it (in its raw format it’s not easy to read or analyse) and email you every day to tell you which (if any) of your preferred routes had problems. They also offer more elaborate features such as the ability to make automatic claims for delays on a particular route.

This, and indeed much of the DRS service overall, has a particular appeal for the season ticket holder. Their routes, fare and train times are usually quite predictable.

But why wouldn’t the train operators just publish this performance information openly on their own sites?

Hmm. Let me think about that.

You see the problem? Although everyone involved is very clear that making a claim when you aren’t entitled to it is fraud, and this is very bad (which it is) – there are certain difficulties in practice.

You don’t even have to travel on a train to make the claim – because of course you can’t, by definition, if it’s been cancelled. You can’t even rely on a swipe at a ticket barrier to show intent to travel – who would leave the concourse (or even, in the case of my journey from home, my house) to do that if the signs (and apps) are all saying “cancelled”?

The Delay Repay claim form I use also asks me to say how long I was delayed. What does that mean? How much additional time it took me to reroute, bus, cycle and hike to my destination, end-to-end? Or how much the train I’d planned to get was delayed?

They don’t specify – because it’s not in their interests to do so – nor is it a clear concept. So they let the user choose how long they were delayed, in bands from 30 minutes to 120+. (My view on this is simple: if the train is cancelled, it’s always entered as a 120+, even if the next service comes along in 25 minutes. If they run it with a delay, then I use that time. My appointments, decisions and connections depend on trains running. They cancel; their problem. If I’m reading this wrongly, I welcome any official guidance…)

So DRS creates the potential for widespread fraud – enabled by the release of data. Perhaps “creates” is too strong – the potential already exists – but it certainly makes it a lot easier. To put it another way, DRS do show people where the potholes are so they can drive into them, exactly as my gloomy local government contact predicted all those years ago.

The train companies are fighting back, of course. Since DRS set up shop, the Delay Repay form has added a Captcha (to hamper automated applications) – an additional tick required to confirm the journeys were actually real (or really intended, I guess) – and stronger warnings against fraudulent claims. They’ve also changed the way that log-in works so that I have to manually fill out all the fields pretty much every time I use the form – passive aggression in interaction design if ever I saw it.

I’ve also had claims reduced or knocked back for being not as delayed as I’d thought – it’s not really worth fighting over each of these, because of some of the ambiguities of terminology mentioned above.

They hint that they’re using analytics to find the patterns of the “world’s unluckiest commuter” whose train is always the precise one that’s been cancelled. Or even, in extremis, would they scan social media to find those holiday snaps from Ibiza when the claim is for a dreary March morning in Ifield? Ok, maybe that’s going too far, for now…

Warnings based on statistics are one thing, mind you – prosecutions or withdrawal of tickets are an entirely different matter. I’m looking with interest for the first court case; because I am certain it will come. It’s massively in their interests to find someone to hit, and hit them hard. [See update below, 28 April 2017]

There is no doubt that an arms race is underway. DRS emailed me with the latest technical changes at their end to get automated claims working again, for example, in response to the introduction of the Captcha verification.

If one pays for a service, one wants to at least recover the cost of subscribing, so there will always be temptation. And in the mind of the commuter, perhaps the moral issues are more complex. All those missed claims because the information wasn’t at hand? Surely it’s fair to make up a few of them here and there? That time when they dumped me off the train at Purley at midnight, then fast-ran it through my bloody station…

You can see how the arguments stack up. I feel a certain level of sympathy for the operators, of course – they have to pay out for delays, and they will only ever be able to manage, not eliminate, fraud.

There’s also a strong whiff of inequality about all of this – the information-rich get a better deal than those who aren’t aware of what and how to claim. I can see ways to improve that, but they’d all require the operators to do – and spend – more. Probably unlikely to happen, in that case.

So – no great conclusion other than to marvel at what complex moral and societal issues surround even something as simple as historical train information.

I can certainly see that DRS add enough value with their unpacking of the stream of raw data, and their email alerts and other services, to give them a business model.

At least until a competitor arrives to undercut them. Market forces tend to keep running, even if the trains don’t.

You’ll be pleased to know that I wrote this over a succession of heavily delayed train journeys. And yes, I am a DRS subscriber.

*The insertions in this popular phrase are “Yellow” and “Rubbery” in this, my favourite variant of it.

Update: 31 March 2016

As ever, starting discussions in an area like this quickly leads to new and better information. What I learned, thanks to Chris Northwood and others, is that DRS don’t get a pre-packaged delivery of this data every day from the train operators. It wouldn’t make sense, really, if you think about it – why would a train operator do that?

What they’re doing (perhaps they’d like to add a comment?) is drawing on the Network Rail feeds, which are, more or less, made available as open data. I duly signed up just now and had a look. Gosh, it’s raw. Really raw. Hefty chunks of JSON, yours to do with as you wish.

It nicely demonstrates an open data business case. DRS are adding tremendous value by taking it in each day, crunching it into something usable, and sending people the precise parts that are most useful to them. Well done them for spotting the opportunity (whatever the motivations of its users may be) and creating a business on top of the data.

The argument remains open as to whether train operators should do that legwork for their customers – if they really wanted to help them – but it would simply add a cost that they’d have to cover somewhere else. Value is value – whoever adds it. There are no free rides here.

Update: 28 April 2017

And here’s that court case I predicted…

It’s quiet at Broadcasting House on a Saturday morning. Once you’ve found the right entrance, a security pass is pushed into your hand, then you’re led up to an empty corridor – reminiscent of a thousand council offices – to wait.

Tired buns and tea, a nervous tricklepee, then with the clock terrifyingly close to 9am – TRANSMISSION TIME – into the studio for a very quick hello with the Rev Richard Coles.

I liked him immediately. I knew I would, though. Someone who’s done very different things in his time, in extraordinary and challenging places, with at least the outward appearance of now having one foot rooted in a Northamptonshire churchyard and the other in the clutter of celebrity – what’s not to like, or indeed, to resonate?

Then the show’s on. No rehearsal, little briefing (“don’t mention Scottish independence” was about it – shame – I’d got some thoughtful comments ready just in case.) Head reeling a little from the dramatic trail I heard at 0830 (“Fresh from the pit of alcoholism…Paul Clarke”)

Flitting in and out of headphones, the red eye of studio vitality blinking next to the orange glow of ON AIR. On air. On air to millions (?) of people out there, including some very close to me. Some, I’m sure, for whom this would be a distinct surprise.

And yet, through all this, we managed a bit of a chat, usually only broken by Richard grabbing his headphones, or a proffered sheet of A4 with some new tweets or emails to read. A wave of the fingers, a glance at the script on screen, then honey voice gliding to the mike again.

An hour and a half of this. It flew by. Though I’m never normally short of words – especially when talking about something close to me, or that I believe in – with these two I was the rank amateur. The third person in the studio was Val McDermid – a wonderfully erudite writer with a brilliant array of anecdotes, and no shortage of opinions. I mean that in a very good way. She was strong, confident and lovely, and very reassuring to the newbie next to her.

Behind a window, a group of producers, watching, listening, typing, running… Me the goldfish, they the cats. My glasses kept falling off when I moved my headphones. I couldn’t decide if I should take them off, so I could see Richard properly, or keep them on, so I could scribble comfort notes on the pad they’d given me. I imagined the crew murmuring – ah, bless – as I wrestled for the fifth time with tangled specs and cables.

I can hardly remember any of it. Just be yourself, everyone said. Be yourself. Who’s yourself? What am I saying? What do I mean? Are they same thing?

The introduction – several times – ThePhotographerPaulClarke. Yeah. It’s true, of course. It’s what I do – intensively. But you know the little voice. The one challenging my right. The one I’d use at the radio in my kitchen listening to another privileged wannabe talk about his ohsodifficult struggle to make a living by poncing about doing a bit of clicking.

I wasn’t promoting a book. I wasn’t promoting me (I would be very surprised if anything I said in a few minutes on radio would make a material difference one way or another to my business). So what WAS I doing there? At 9.28, in the studio, it’s probably a bit too late to be overwhelmed by that thought. But as the finger waving and mouthed “Paul, you’re next” oscillated through the-gel-that-is-studio-air towards me, I was.

So I did my best. Of course, nothing that I’d prepared came up: who inspires me*, my favourite photo, most interesting anecdote from a shoot… Zip.

Instead, something about sport, Elyar Fox (not Justin Bieber), loneliness. A schoolboy joke about boobs. Oh, and drinking.

Yeah, that’s why I was there. Because through whatever alchemy of support, friendship, luck and bloody-mindedness, I was the guy who’d put down alcohol, started using a camera to cope, and to self-express, and unexpectedly found himself confounding all predictions to build a successful business taking pictures. Pictures that people wanted to see; that they paid for.

And if, just if, something I said might be useful to just one person, that would justify my early start, quaking terror, and some embarrassing family conversations in the future. Ok, maybe two people. But I was doing it because my interesting little talk had been picked up by the researchers as an example of something that might speak to a rather wider audience.

At five years in, I’d felt able to do that talk, and a year earlier, to write just this one piece which I hope some also find resonant. But that’s all. It’s not a secret, but neither do I think I wear it on my shirt.

Of course, there’s dramatic alcoholism, and there’s the other sort. In my experience, it’s the other sort I’ve come across time and time again, and maybe in some ways it’s more deadly. Because it’s the sort that rumbles on for years, for decades, eating its host away without ever quite triggering an epiphany.

Being media people, the steer was very much towards the “so just how bad was it, when you were at rock bottom and your life was destroyed, Paul?” Being me, I did what I could, not to play down those dark times, but to emphasise the grinding normality of repeatedly doing something you don’t want to do. Of using a substance or a habit to deal with troubling inner voices. That the hardest thing for me – the point that told me it was time to change – was simply the realisation that I’d lost my choice. If it was 6pm, I drank. Simple as that. Whether I wanted to, or not. And in the latter days, I rarely wanted to, nor did it lift me up anywhere other than temporarily out of a craving.

Breakfast whisky? Nope. Blackouts at work? Nope. Uncontrollable trembling? Nope. Loss of job because of drinking? Nope (though I heard that in Saturday’s script, presumably as a result of a researcher conflating some other comments in my talk about my career shifting around me. Journalism, hey?)

Relentless, day-in-day-out, repetition of patterns that were hurting me (and those around me) more and more. Yes. YES. That.

And, in the aftermath, it seems that message was perhaps the most resonant I could have made. Had I crumbled, and gone along with the offered line of park benches and breath mints, it might have made a more dramatic storyline. But it wouldn’t have reached some people – people with far less flamboyant stories and issues to deal with.

I know this, because I’ve since heard from some of them. No details, obviously, but even before the programme had finished I’d had messages from others out there. Some known to me, others not at all. I’ve done (and will continue to do) what I can to help, and am grateful that my target of two was swiftly passed. There’s also a hint of something coming out of this, which if it’s meant to be, will be one of the biggest and most exciting things I could possibly dream of. But I’m doing a serenity prayer on that one right now, because it’s definitely something I can’t directly control.

So that was live radio, on a tricky subject. Much, much more difficult than I’d realised. But I do like to take on the difficult. On balance, I’d do it again. Just better. (You can listen to the show here, and I’ve got a chunk at about 30 minutes in, plus other brief moments.)

I haven’t heard from my Dad yet.

[UPDATE – 18 Sept. I have now. It was so fantastic. He said he was proud of me.]

*if you’re interested, it’s not a photographer, nor someone particularly well-known. It’s not even an adult. It’s a 14 yr old boy called Adam Bojelian. A couple of days ago, Adam clocked up an unbroken year in hospital, with some pretty intensive physical challenges. Despite these, Adam is a regular and happy tweeter, organises quizzes and competitions including a World Cup sweepstake, campaigns relentlessly for the voice of children in hospital to be heard, and writes poetry. Great poetry. Oh, did I mention he is only able to communicate by blinking? And yet I know about his world of outings in the park with his dog, and the tireless strength of that small body. Yes, that inspires me. And if you’ve liked this post, maybe follow him (@adsthepoet) or just say a nice hello. He’d like that. If Adam can do what he does, what excuses have the rest of us?

It’s time to give you up. To let go. To say goodbye.

You were first there when I was about 10, taking me to cubs behind my dad. That was so cool. (It wasn’t, of course.) You were a clapped-out Honda 50 straining under the extra load of one small boy, but those warm evenings I was Evel Knievel and I can still feel where the battered chinstrap of that gobstopper helmet scratched me. Later I’d wobble up and down the drive on you all by myself, never quite sure whether you had a clutch or not. You were pretty ambiguous on the subject, too.

At 16, you were still pretty lame – Ju beat us up a hill on his pushbike, me and you and your puny, restricted engine. Sometimes you were a borrowed Fizzy or that DT100 I nearly threw into the hedge at Clieves Hills. Unrestricted power – the surge of passing 40mph for the first time as I rode your 100ccs up the St Helens Rd away from the shop. The smell of hot petrol and two stroke in the back lanes at Halsall. Boys heading to Southport, to girls and dunes and the chance of a snog. Jeans streaked with oil and hormones.

Riding test on a CZ125, too slow to be any danger to anyone, and that first moment without the Ls on the Abingdon Road when a passing biker dipped his head in salute for no other reason than we were on bikes, and it wasn’t raining. Oh, and we weren’t in London, where that sort of civility isn’t done. I was in. I wasn’t a learner. I was a biker. You took me through frozen fog that night to Birmingham airport, to surprise Justine on her 18th in Dublin. Gloves still stiff with ice even as I boarded the plane.

Now a GT250, you fell apart in style with that collapsed, inaccessible oil seal, but Ben and I – mostly Ben – found an old engine in Headington and patched you back together. The noise that shook the Cowley Rd when you fired into life at the first time of asking, sans exhaust! A howl of defiance, heralding new opportunity (and cash) as I rode you for real – summers as a courier up and down the M40, sometimes three times a day. Manchester in the cold, Exeter in the rain. The incredible satisfaction of finishing a day’s work with a signed slip, and the knowledge of a job inarguably completed. That didn’t return for another 20-odd years…

Then you were the most hideous pink VT500, a pimped-up despatch bike with built-in boxes and a full fairing. I was CHiPS of East Oxford, sometimes able to top the ton on the sliproad out of Headington, all rollies and camaraderie – swapping parcels for signatures in Soho design agencies.

Biker, son of biker, son of biker – and when the oldest was close to the end – “if you want to see him, you’d better come quickly” – you took me there quickly – tearing up the A43 for one last hand-hold. The one good hand that had raced his own polio-adapted bike, all those years ago. He cruised over the horizon, and I cruised back to college.

Once you were a massive Kwak 1000, collaborator in sinful liaison with her sister by the Farmoor reservoir. Oh my, those days. Or that maroon GT550, less said the better. The courier’s two-wheeled Transit, plodding and dull.

To London, and your XJ600 era. Getting a bit more grown-up now, four cylinders of balanced buzz, but how you must have hated me as I neglected your sprockets until the teeth came off. Prague with the Aces, though, slipping over rain-dashed tramtracks and cobbled streets, and the mighty solo leg home, dad of two tinies whose holiday pass had run out early, dodging the temptations of the gaptoothed sirens on the Teplice road. Hands blue with vibration, and unable to speak on arrival home.

With first proper job, I splashed out on you as a brand new CBR600. Flipped head over tail after 3 weeks but you were well enough after a rebuild to get me to Le Mans and for a few brief moments show me your absolute top speed on the 24hr course. Next, you were a VT1000 Firestorm (warning me of your lethal power with a sideways squirm of the back wheel on ice).

The litre V-Twin superbike: I’d arrived! I loved you so much (didn’t care that you were a bit tamer than a Ducati), wanted to keep you perfect, and on murky days rode you in the shape of your older cousin – the one that Mike stabled with me while he rode an XT round the world. It was one foul wet December morning that you catapulted me through a windscreen, shattering both my leg and my love for you. Two years limping with a metal thigh bone to add to my biker cred. I got rid of you after that. Crashing again with a steel femur wouldn’t have been clever.

A few years off – responsibility and rehabilitation (and poverty) setting the course – until the XT500. The classic 70s trail bike, you took me to Gibraltar after the divorce. It was needed. I suffocated you with altitude in the Pyrenees, but you forgave me as we wheelied into Spain. We sat by the port at Algeciras and I thought about inflicting Africa on you. But we turned north, and I played at being Corser through the meanders up to Albacete. I strapped petrol cans to your handlebars, and a guitar to my back, and tasted freedom and bitterness.

Then you were a VFR and I knew that you were one of the greatest bikes ever made. But…I sold you out for more power. I dabbled with the hard stuff; big Aprilia, GSXR – but also the old stuff; occasional flirtation with AJS and Norton – even, briefly, a Vincent. Dark deeds by Blackbird, as you showed me how you could Hoover the horizon towards me at full throttle. Beautiful aerodynamics stopped me feeling your 150mph blast round the M25 (thigh now rebuilt and de-steeled by this point).

You were a Firestorm again for the motorways, as I two-timed you with a 660 Yamaha thumper. The Firestorm blew out, and then you became Tweetbike. Brief global fame dodging tubestruck London, traded soon afterwards to a new model with vanity plate. Loud cans, “Let them know you’re coming!”, the most fun that could be had in a city.

Then you left me. Or I left you. Older, wiser, perhaps a little too stiff and sore. No real justification to keep you on; you sat and cried rusty tears in the garden for far too long. I’m letting you go today. It’s time. I’ll give you a last sip of petrol, wipe your arse down, and find you a new home. It’s been hard to let go. I’ve put it off for so long. So much more than a collection of metal, glass, rubber and plastic. Heat, passion, dreams, freedom. I’ll find them again, somewhere else, I’m sure.

But to you, my bloody lovely bike, maker of adventures, thriller, rescuer, it’s time to say it now. Goodbye.

Remind me again: what’s the purpose of opening up all this public data?

Ah yes, that’s it. To create value. And you can’t get a much stronger example of real value in the real world than showing people how to save money when buying train tickets.

Fare pricing is a fairly hit-and-miss business, as you’ve probably noticed. We don’t have a straight relationship between distance and price. Far from it.

The many permutations of route, operator and ticket type throw up some strange results. We hear of first class tickets being cheaper than standard, returns cheaper than singles, and you can definitely get a lower overall price by buying your journey in parts, provided that the train stops at the place where the tickets join.

The rules here are a bit weird: although station staff have an obligation to quote the cheapest overall price for a particular route, they aren’t allowed to advertise “split-fare” deals, even where they know they exist. Huh?

Why this distinctly paternalistic approach? Well, say the operators: if a connection runs late, your second ticket might not be eligible, and there might be little details of the terms and conditions of component tickets that trip you up, and, and, and…well, it’s all just too complicated for you. Better you get a coherent through-price (and we pocket the higher fare, hem hem).

There’s no denying it is complicated. Precisely how to find the “split-fare” deal you need is a tiresome, labour-intensive process of examining every route, terms and price combination, and stitching together some sense out of it all. And, indeed, in taking on a bit of risk if some of those connections don’t run to time.

You might be lucky, and have an assistant who will hack through fares tables and separate websites to do you that for you. But you’d be really be wasting their time (and your money).

Because that sort of task is exactly what technology is good at.

Taking vast arrays of semi-structured data and finding coherent answers. Quickly. And if there’s some risk involved, making that clear. We’re grown-ups. We can cope.

There’s no doubt at all that the raw materials–the fares for individual journey segments–are public information. Nobody would ever want, or try, to hide a fare for a specific route.

So when my esteemed colleague Jonathan Raper–doyen of opening up travel-related information and making it useful–in his work at Placr and elsewhere, put his mind to the question of how new services could crunch up the underlying data to drive out better deals for passengers, I don’t doubt that some operators started to get very nervous indeed.

Jonathan got wind–after the November 2011 meeting of the Transport Sector Transparency Board–that a most intriguing piece of advice had been given by the Association of Train Operating Companies (ATOC) to the Department for Transport on the “impact of fare-splitting on rail ticket revenues”.

Well, you’d sort of expect an association which represents the interests of train operators to have a view on something that might be highly disruptive to their business models, wouldn’t you?

So what was that advice? He put in a Freedom of Information request to find out.

And has just had it refused, on grounds of commercial confidentiality.

This is pretty shocking–and will certainly be challenged, with good reason.

Perhaps more than most, I have some sympathy with issues of commercial reality in relation to operational data. We set up forms of “competition” between providers for contracts, and in order to make that real, it’s inevitable that some details–perhaps relating to detailed breakdowns of internal costs, or technical logistics data–might make a difference to subsequent market interest (and pricing strategy) were they all to be laid out on the table. I really do understand that.

But a fare is a fare. It’s a very public fact. It’s not hidden in any way. So what could ATOC have said to DfT that is so sensitive?

The excuse given by DfT that this advice itself is the sort of commercial detail that would prejudice future openness is, frankly, nonsense.

I look forward to the unmasking of this advice. And in due course to the freeing-up of detailed fares data.

And then to people like Jonathan and Money Saving Expert creating smart new business models that allow us to use information like it’s supposed to be used: to empower service users, to increase choice, and to deliver real, pound-notes value into the hands of real people.

That’s why we’re doing all this open data stuff, remember?

A proposal:

At around age 14, set aside a week away from the standard education curriculum for kids to work, in groups, in a very focused way: on a project with a simple brief, but a complex reality.

What does that mean?

Well, it might be to design a way of giving everyone in the UK £100 (as an alternative quantitative easing approach). It might be to identify all public buildings to find better ways of using them. Or to model what would happen if jails were abolished (or if speeding convictions carried automatic jail sentences). Or to design a rail system that would be ultra-resilient to sudden, massive demand and freak weather conditions.

Anything really.

Or at least anything that would show that a bit more effort is required in reality to do some of the things that really matter in this world. Even though they might sound simple. So that first one: giving everyone £100? Well, you’d need to work out who “everyone” was…what would qualify as entitlement…how to get the money to them securely and trackably…how to deal with claims that it hadn’t been received (true and fraudulent)…how to deal with those who didn’t want to be on any state registers but still wanted their cash… You get the picture. Putting real world details around a nice, simple concept.

You’d cover analysis, planning, teamwork, logistics, consequences (seen and unforeseen). And probably a whole lot more. You’d learn about edge cases, the ability of a small number of difficult situations to eat up disproportionate resources, and how you have to design for the awkward, not just implement for the easy.

And out of all this, there might, just might, be a tiny chance that statements like “well, I don’t see why they can’t just…” or “how hard can it be to…” would be cast around just a little less lightly. And questioned a little harder when falling from the mouths of politicians.

Because the problem is this: when we’re small, our world is small. And simple. Decisions are clear, motives unambiguous, morality absolute. Things are, or they are not. Laws are clear, enforceable and enforced.

The King says “make it so!” and the Knights make it so. The Princess makes her choice, and the losing suitor slinks away, never to play a part in this or any other story.

And so it goes.

And then things change. Our world gets bigger, and more difficult. We realise that society is a loose patchwork of consents, of unwritten codes, of behaviours.

And do we change, too? Or are we content to carry on with an increasing pretence that the world is monochrome, that things can be made to happen by dictat, and that anything involving sixty million people need not be any more complicated than something involving a handful?

Do we continue with these childish fantasies, and follow leaders who–even if they believe in their hearts that what they propose is at most only partly achievable–must dance the dance of the simple: spouting policies that can never be delivered, just so they continue to look…like what? Like leaders. Right.

Left-wing and right-wing, we dance up and down the same spectrum of choices: of levers that can be pulled in various directions, of societal mores running from the brutal to the soft, the feudal to the flattened. We might choose different starting positions, and have certain favourite themes and moves. But if we get stuck with these lame little models of “why don’t they just…”, and come to believe that wickedly difficult problems are actually easy, then we’re all stuffed.

Because we do believe. On a mass scale. Because we were never taught any differently. We weren’t taught to think harder, to go deeper, to challenge rigorously, or to live the reality of what implementation might actually be like.

And so things like social and economic policy get very broken. Preposterous, simplistic “solutions” float around: hoodies marched to cashpoints, rioters’ families evicted, a single ID number, watertight borders, cities scoured of benefit claimants, a single central health record for everyone…the list goes on. (That last one would make a great school project, by the way. Starter: think who might need to view and/or change that record–including the patient–what their interests and motivations might be, and how all those agendas stack up against the benefits.)

It’s an awful lot easier to believe in simple magic than to work through hard science. And very much easier to whip a crowd up behind you, too. Asking those hard questions has become the antithesis of leadership. What a splendidly vicious circle!

And yet, it can be broken. With so many other large scale problems of capability or understanding, we try to fix things at source. Through education, for example. It beats me why we’ve never seriously attempted this route.

Would a more aware, canny, and yes perhaps cynical population really be that frightening? Or would we sniff out the stupid and actually become far more tolerant as a result?

I think it’s worth a try. Be a hell of an interesting week, anyway.

(It took me until I was 17, and half-way through A-level Economics, before the reality pennies started to drop. That transactions mostly have two sides to them. That someone’s good deal is often another’s bad one. That public finance doesn’t begin with some kind of magic money tree in the Treasury courtyard. That expectation can be as powerful an influence as factual evidence. That people don’t always do what you expect them to, even when we have laws to force them to. It felt like a real privilege to have my mind stretched like that, and I’ve always felt a bit more of it could go a long, long way.)

As I write this, I’m sitting on a stationary train. In a station. The rail app on my phone tells me it’s the train I want. But the signs on the platform are totally blank. And the guy in uniform on the train doing the uncoupling says he doesn’t know where it’s going.

So, do I believe what the app tells me? Rather than embark on an exercise in Bayesian conditional probability, it’s making me think about that phrase “digital by default”.

Because I’m still not entirely sure I know what it means. Or, even if I do, that I’m seeing it used consistently.

And this experience with the phone app right now is a good reflection of what I think it should mean: that a service has been built, first and foremost, so that its delivery in digital channels is the way that it works best.

–that information in the digital channel is “the truth”.

–that if the train is switched to another platform, the digital channel will be the first to reflect this.

–that train staff will be looking at their own digital devices for information before they look at platform signs, or paper print-outs of departures, or get on the internal intercom to the driver.

That, to me, is digital by default.

An underpinning design principle that the service is supposed to be like this. Not, as has so often been the case, with digital features as a sort of awkward bolt-on after the fact.

I pointed out to a member of station staff a few weeks ago, who tried to stop me, that I was going through the gates to platform 10 because this device in my hand was telling me my train would be there. And I trusted it, at least enough to wait there.

He looked in incomprehension at this device. It wasn’t part of the script. The situation was the very opposite of “digital by default”.

So, apart from this nice, rosy, optimistic definition, what else have I seen it used to mean?

Well – sadly, sometimes as the Mr Nasty of channel-shift enthusiasts: the reason why counter services will be closed, the hammer that will force people to abandon their Luddite ways, the only real means of forcing out cash savings in this techno-progressive world we were told so much about.

And if people don’t want to shift, then tough. They won’t have the option. Default, innit? Capisce? Ok, if they’re really incapable, because of disability or crap connectivity, there’ll be some sort of stop-gap. A bolt-on, if you like. After the fact.

Now, does that sound somewhat familiar?

Or, for a third flavour, how about Mr Nasty’s gentler cousin: the service redesign that still has the closure of non-digital channels at its heart, but attempts to do so by attraction to a better, digital alternative, rather than brute imposition?

The interpretation you hear is connected to the source you hear it from, I guess. These versions all have different political palatability, and provoke different passions in different audiences.

So which do you i) think it really means now? And ii) which one would you like it to mean?

A – a fundamental design principle from the ground up
B – channel shift by imposition and removal of choice
C – channel shift by being more attractive than non-digital

Your answers, below, if you please:

For a long time, I’ve shied away from writing here about personal data. Or even thinking that deeply about it. The nature of identity, yes. The usefulnesss of data, yes. Personal data, no. Why?

Not because it isn’t fascinating, or important. Mainly because it’s so…damn…nebulous. And difficult. Time to get over that, I think. Very significant things are happening in this area, and we all need to raise our game in how we understand and engage with the concepts involved.

As I’ve surmised before, the only things that are really different in the Internet age are the ease with which information can be found, and the ease with which it can be stored.

Two things, really. That’s all.

The first embraces everything around indexing, cross-referencing, labelling, structure and searching. The latter takes us into the territory of copying (and of course copyright), archiving, and the general issue of persistence.

And when we look at personal data in that context, there is an immediacy–and potential toxicity–in what emerges.

We saw early rumblings of this long before the Internet, of course, when computers were first used for the mass processing of information about people. Things could be done with databases that simply weren’t possible with big paper ledgers.

We created Data Protection legislation which attempted to put reins on the ability to make free use of some types of information. Gathering stuff about people, from the basic facts of who and where, to how to contact them, who they were connected to, and what their tastes and preferences were. Pure gold, used in the right (or wrong) ways.

Data Protection set out some pretty sound, but general, principles. The overarching one being that the purpose to which data could be put should always be made clear to whoever provides it, at the time of providing. Lots of other stuff about processing, storage, where and how long, and so forth–but that issue of consent always seemed the most important, to me.

And we scratched about a bit to actually try and define what we meant by “personal data”. Some things were easy. Names. Addresses and phone numbers. They’re just obvious.

But what about our tastes? Our buying history? The movements of our mobile phone from cell to cell? A journey we took? As one takes informational side-steps away from the individual, the obviousness diminishes, but if you can make meaningful connections back to the person…

…and remember the first thing that the Internet really changes?

Being able to make those tenuous links between blocks of information into something really substantive.

And the second thing? That information and those links are now permanent. You can’t delete them, once they’re there.

All those things that databases couldn’t previously do, because they all conformed to different standards, and weren’t connected together? They can now. Things can be done via the Internet that simply weren’t possible with just the databases.

Bit by bit, it’s been possible to build up the most humongous repositories about people. Maybe entirely within the law, maybe in other ways as well. Maybe with explicit and informed consent all the way down the line. And maybe not.

Who’s to know? We find strange things going on with data that we provide in order to use one or other service–or even to exercise our democratic rights. Didn’t it ever strike you as slightly weird that the electoral roll could be sold on for commercial purposes? (Much more on the electoral roll in another post coming soon.Update: now here)

We have big companies that have built successful businesses just like this: perhaps using aggregated personal information for credit referencing, perhaps to sell to marketeers to give them a better understanding of demographics.

The genie is very much out of the bottle. Your rights to see the information that a particular company holds on you may exist, but you have to have a fair idea of which company to ask in the first place. Can you ever see the full picture of what others know about you?

Of course not.

And it’s unreasonable to suggest that we’ll ever be able to do that. Instances of data multiply more rapidly than does our capability to track them. (There must be a Law of Internet Entropy out there that says something like that. If not, I just invented one.)

(As an aside, a dear friend once uttered the memorable line “somewhere out there, there’s a database with your dick size on it”. That was in 1989.)

So what can we do?

Realistically, all that’s available to us are firebreaks and friction.

We can’t get that genie back in the bottle, but we can slow it down a bit, and find ways to mitigate the impacts.

Do we need an updated definition of personal data? It’s MUCH harder than it seems at first glance to create one. The best I can find at the moment in terms of an “official” position is here.

And it’s clumsier than you think. Essentially, it’s a list of ever-widening filters that assess whether a particular piece of information can be connected to a specific individual. Culminating in the rather wonderful catch-all of the final category:

8. Does the data impact or have the potential to impact on an individual, whether in a personal, family, business or professional capacity?

Yes The data is ‘personal data’ for the purposes of the DPA.
No The data is unlikely to be ‘personal data’.

Even though the data is not usually processed by the data controller to provide information about an individual, if there is a reasonable chance that the data will be processed for that purpose, the data will be personal data.

That’s pretty general, no? In fact, going by that, an awful lot of things are now personal data. I really like the emphasis it puts on the outcome of the data use, not attempting to over-define things like form and structure.

I’d go as far to say we should probably throw away that big long document, and just run with this definition:

Personal data is information that affects you when it’s used. Either directly, or through being linked to other information using technologies that exist now, or may exist in the future.

Broad enough? ;)

(So my beloved photos: they’re personal data. I take them with a camera that has a unique number, held in metadata in the picture file. That provides a way to link all the pictures it takes together, and then, through the various accounts I put them in online, back to me. Think how many other trails you leave…)

But again, all we really have are firebreaks, and friction. There’s a sort of reverse entropy at work. Unlike almost every other instance of entropy–where things get more chaotic over time (china plates get broken, they never put themselves back together again)–personal information is, relentlessly, only going to get more linked. More aggregated. More pervasive. More permanent.

(So, maybe I just invented The Law of Reverse Internet Entropy as well? Not bad going for one post…)

And if someone tells you that big blocks of personal data can be “de-anonymised”, be very sceptical indeed. (You can read some wise thoughts on the issues involved here and elsewhere on that blog.)

We can undertake some pretty noble fire-breaking: like ensuring the state doesn’t become the source of a global universal identifier for you. And we will certainly see more developments around multiple personas: compartments of your life associated with particular tasks, contexts, or connections. I think we’ll have to. (The concept of federated identity helps here, but that’s too much to go into for this post. Read more thoughts from the team working up these concepts for government.)

And we’ll adjust. Society has seen some pretty dramatic upheavals. Often associated with a new technology, or philosophy. If we adjust our societal norms faster than the upheaval, we don’t notice. If we’re slower to change, it’s painful. For a bit.

But we get through. We adapt. And we change. Always.

I did a thing that might have been very wrong yesterday. But I’m not sure.

So this is part confessional, part taking advantage of it as a vehicle for discussion. (And a fair bit of hand-wringing into the bargain.)

I recorded someone’s conversation without their knowledge or consent and I put it on the Internet, amplified via Twitter and Facebook.

I’ve had some great discussions in the past about where the boundary of public and private really lies: debating the nuances of shouting “FIRE!” in a crowded theatre versus an empty street versus a whisper in a pub. Thoughts around the “Twitter joke trial” about the point at which something could meaningfully be said to be “broadcast”, or to have a particular intent, or audience. The Press Complaints Commission ruling this week on the (non)privacy of tweets.

It’s a bloody minefield, that’s for sure. And much of the mine detection equipment hasn’t been built yet, and what has, hasn’t been well tested.

I am pretty scrupulous about respecting privacy, where I understand privacy to exist. As I understand it. Which isn’t all that straightforward either.

So a person going about their job, talking to a colleague about what they think of their company’s internal training (and more) shouldn’t expect to have their conversation recorded and “broadcast” on the Internet, should they? Under any circumstances?

Questions, questions, questions.

What does broadcast mean here, really? Does identifiability of the speaker make a difference? What about the nature of his job? Where he is as he’s speaking? How long he’s been going on like this? Whether his outpourings have clearly (to my ear) left the realm of colleague-to-colleague and taken on the guise of a rather bizarre form of public performance art? What about him swearing, loudly, wearing a company uniform and in the earshot of children and those visibly (to my eye) not appreciating it? Rubbishing his employer. Crude sexism. Do any of these make a difference?

And on my side of the fence: what’s my intent? To amuse people? To hold him to account or even get him fired? And what if that’s the outcome and it wasn’t my intention? Do I become liable, legally or morally?

The Internet has changed everything. The two dominant characteristics, as I see them: ease of access to information and permanence of record are visibly in play here. I don’t know where the recording might end up. I do know it will be somewhere, for ever. With a very low threshold of effort required to find it.

Of such vapours are the clouds that we call public and private space formed. And I thought long and hard about them. And I weighed the answers I came up with as levelly as I could. And I published.

Let me know what you think. Right, or wrong? Or “complicated”?

Would make a good role play, mind you.

(I don’t do role play.)

I’m hugely sympathetic to snow and ice closures in large, complex systems. I can imagine that the cost of replacing the entire third rail system used on much of the south-eastern rail lines makes little sense in relation to the handful of extra days of operation it would provide over its life. There’s no use having salt and gritters available if you haven’t got the operators as well, or can’t get the operators to the equipment.

These are at least rational arguments, though rarely understood.

But an airport? A couple of miles of tarmac? Surely that can be kept clear? I’ve just seen this morning that “although the airfield is operational, BA have cancelled all Heathrow flights…” I really feel for my friends today who are having to rearrange travel plans in the tight window before Christmas.

We can easily compare what we think of as Heathrow airport – a couple of miles of tarmac – with, say, Moscow, or Washington DC – a couple of miles of tarmac. Planes land there all the time in snow; the snow just gets ploughed away. But perhaps within that phrase “airfield is operational” lies a clue?

Because an airport isn’t just that strip of tarmac. It’s support staff, handlers, fuellers, fire fighters, controllers, yada yada yada. It’s passenger transit systems, road networks and car parks; perhaps it’s not that sensible to fill up an airport with people if they might be stuck there for ages? Its virtual footprint is rather larger than the perimeter fence – certainly larger than the actual physical runway and taxiing areas.

So I can construct a rational narrative which says – “because we can’t access all the resources we need to operate this airport safely, and because of the extremely high probability of substantial inconvenience to passengers should they come to the airport” we’re closing it.

In truth, I can’t construct a rational one which says “because of 5cm of snow on our runway, we’ve shut the airport all day”. Yet without context, this is how the message this morning is playing out. Making everyone involved look stupid or incompetent.

Come on BA, BAA and the rest of you – don’t be afraid to talk about systems. The lack of a decent message is the unforgiveable thing here – not necessarily the suspension of flights.

Treat us like grown-ups. It’ll help us all, you know.

UPDATE 11:40, 18 December

Reports are coming in that the disruption today is confined only to BA flights at Heathrow. Does this provide further evidence for a system-level cause, or is it really just a suicidal level of incompetence from a company that doesn’t care about its reputation or passengers? I’m still inclined towards the former – YMMV.

But I spotted another interesting clue: that due to the massively complex and crammed schedule that BA operates at Heathrow, the minor delays to individual flights caused by the snow and ice would quickly magnify and lead to a general level of chaos. (Sadly, I can’t now trace the tweet giving that insight).

If true, it’s a very interesting illustration of what happens when systems run close to capacity. I remember something similar when fog caused what seemed like a disproportionate level of disruption to flights. The short increase in time due to aircraft having to move more slowly on the ground caused havoc, because – with a schedule of flights taking off every 90 seconds or so – there were simply not enough gaps in which those additional seconds could be absorbed.