In August 2011, Marc Andreessen famously remarked that “software is eating the world”. Writing in the Wall Street Journal, Andreessen set out his view that society was “in the middle of a dramatic and broad technological and economic shift in which software companies are poised to take over large swathes of the economy”.
With his background as pioneering web software architect at Netscape, and with a string of successful investments under his belt at venture capital firm Andreessen-Horowitz, Andreessen was well placed to comment on the potency of software. As he observed,
More and more major businesses and industries are being run on software and delivered as online services—from movies to agriculture to national defence. Many of the winners are Silicon Valley-style entrepreneurial technology companies that are invading and overturning established industry structures.
He then made the following prediction:
Over the next 10 years, I expect many more industries to be disrupted by software, with new world-beating Silicon Valley companies doing the disruption in more cases than not.
Industries to be impacted in this way, Andreessen suggested, would include entertainment, communications, recruitment, automotive, retail, energy, agriculture, finance, healthcare, education, and defence.
In the four years since the phrase was coined, “software is eating the world” has shown every sign of being a profound truth. In more and more sectors of industry, companies that lack deep expertise in software have found themselves increasingly by-passed by competitors. Software skills are no longer a “nice-to have” optional extra. They’re core to numerous aspects of product development.
But it’s time to propose a variant to the original phrase. A new set of deep skills are going to prove themselves as indispensable for ever larger numbers of industries. This time, the skills are in security. Before long, security will be eating the world. Companies whose software systems fall short on security will be driven out of business.
Dancing pigs
My claim about the growing importance of security may appear to fly in opposition to a general principle of user behaviour. This principle was described by renowned security writer Bruce Schneier in his 2000 book “Secrets and Lies”:
If J. Random Websurfer clicks on a button that promises dancing pigs on his computer monitor, and instead gets a hortatory message describing the potential dangers of the applet — he’s going to choose dancing pigs over computer security any day. If the computer prompts him with a warning screen like: “The applet DANCING PIGS could contain malicious code that might do permanent damage to your computer, steal your life’s savings, and impair your ability to have children,” he’ll click OK without even reading it. Thirty seconds later he won’t even remember that the warning screen even existed.
In other words, despite whatever users may say about the importance of security when directly asked about that question (“yes, of course I take security seriously”), in practice they put a higher priority on watching animated graphics (of flying pigs, cute kittens, celebrity wardrobe malfunctions, or whatever), and readily accept security risks in pursuit of that goal.
A review paper (PDF) published in 2009 by Cormac Herley of Microsoft Research shared findings that supported this view. Herley reports that, for example, users still typically choose the weakest passwords they can get away with, rather than making greater efforts to keep their passwords unguessable. Users also frequently ignore the advice against re-using the same passwords on different sites (so that, if there’s a security problem with any one of these sites, the user’s data on all other sites becomes vulnerable too).
Herley comments:
There are several ways of viewing this. A traditional view is that users are hopelessly lazy: in the face of dire descriptions of the threat landscape and repeated warnings, they do the minimum possible…
But by the end of his review, he offers a more sympathetic assessment:
“Given a choice between dancing pigs and security, users will pick dancing pigs every time.” While amusing, this is unfair: users are never offered security, either on its own or as an alternative to anything else. They are offered long, complex and growing sets of advice, mandates, policy updates and tips… We have shown that much of this advice does nothing to make users more secure, and some of it is harmful in its own right. Security is not something users are offered and turn down. What they are offered and do turn down is crushingly complex security advice that promises little and delivers less.
Herley’s paper concludes:
How can we help users avoid harm? This begins with a clear understanding of the actual harms they face, and a realistic understanding of their constraints. Without these we are proceeding blindly.
Exponential change
What are the “actual harms” that users face, as a result of insecure software systems or poor personal security habits?
We live in a time of rapid technology change. As software eats the world, it leaves more and more aspects of the world vulnerable to problems in the software – and vulnerable to problems in how that software is used, deployed, and updated.
As a result, the potential harm to users from poor security is constantly increasing. Users are vulnerable in new ways that they had never considered before.
Hacking embedded medical devices
For example, consider one possible unexpected side-effect of being fitted with one of the marvels of modern technology, an implantable heart pacemaker. Security researcher Barnaby Jack of IOActive gave a devastating demo at the Breakpoint conference in October 2012 of how easy it was for an outsider to interfere with the system whereby a pacemaker can be wirelessly recalibrated. The result is summed up in this Computerworld headline, “Pacemaker hack can deliver deadly 830-volt jolt”:
The flaw lies with the programming of the wireless transmitters used to give instructions to pacemakers and implantable cardioverter-defibrillators (ICDs), which detect irregular heart contractions and deliver an electric shock to avert a heart attack.
A successful attack using the flaw “could definitely result in fatalities,” said Jack…
In a video demonstration, Jack showed how he could remotely cause a pacemaker to suddenly deliver an 830-volt shock, which could be heard with a crisp audible pop.
Hacking vehicle control systems
Consider also the predicament that many car owners in Austin, Texas experienced, as a result of the actions of a disgruntled former employee of used car retail firm Texas Auto Center. As Wired reported,
More than 100 drivers in Austin, Texas found their cars disabled or the horns honking out of control, after an intruder ran amok in a web-based vehicle-immobilization system normally used to get the attention of consumers delinquent in their auto payments.
Police with Austin’s High Tech Crime Unit on Wednesday arrested 20-year-old Omar Ramos-Lopez, a former Texas Auto Center employee who was laid off last month, and allegedly sought revenge by bricking the cars sold from the dealership’s four Austin-area lots.
Texas Auto Center had included some innovative new technology in the cars they sold:
The dealership used a system called Webtech Plus as an alternative to repossessing vehicles that haven’t been paid for. Operated by Cleveland-based Pay Technologies, the system lets car dealers install a small black box under vehicle dashboards that responds to commands issued through a central website, and relayed over a wireless pager network. The dealer can disable a car’s ignition system, or trigger the horn to begin honking, as a reminder that a payment is due.
The beauty of the system is that it allows a greater number of customers to purchase cars, even when their credit history looks poor. Rather than extensive up-front tests of the credit-worthiness of a potential purchaser, the system takes advantage of the ability to immobilise a car if repayments should cease. However, as Wired reports,
Texas Auto Center began fielding complaints from baffled customers the last week in February, many of whom wound up missing work, calling tow trucks or disconnecting their batteries to stop the honking. The troubles stopped five days later, when Texas Auto Center reset the Webtech Plus passwords for all its employee accounts… Then police obtained access logs from Pay Technologies, and traced the saboteur’s IP address to Ramos-Lopez’s AT&T internet service, according to a police affidavit filed in the case.
Omar Ramos-Lopez had lost his position at Texas Auto Center the previous month. Following good security practice, his own account on the Webtech Plus system had been disabled. However, it seems he gained access by using an account assigned to a different employee.
At first, the intruder targeted vehicles by searching on the names of specific customers. Then he discovered he could pull up a database of all 1,100 Auto Center customers whose cars were equipped with the device. He started going down the list in alphabetical order, vandalizing the records, disabling the cars and setting off the horns.
His manager ruefully remarked, “Omar was pretty good with computers”.
Hacking thermostats and lightbulbs
Finally, consider a surprise side-effect of attaching a new thermostat to a building. Modern thermostats exchange data with increasingly sophisticated systems that control heating, ventilation, and air conditioning. In turn, these systems can connect into corporate networks, which contain email archives and other confidential documents.
The Washington Chamber of Commerce discovered in 2011 that a thermostat in a townhouse they used was surreptitiously communicating with an Internet address somewhere in China. All the careful precautions of the Chamber’s IT department, including supervision of the computers and memory sticks used by employees, to guard against the possibility of such data seepage, was undone by this unexpected security vulnerability in what seemed to be an ordinary household object. Information that leaked from the Chamber potentially included sensitive information about US policy for trade with China, as well as other key IP (Intellectual Property).
It’s not only thermostats that have much greater network connectivity these days. Toasters, washing machines, and even energy-efficient lightbulbs contain surprising amounts of software, as part of the implementation of the vision of “smart homes”. And in each case, it opens the potential for various forms of espionage and/or extortion. Former CIA Director David Petraeus openly rejoiced in that possibility, in remarks noted in a March 2012 Wired article “We’ll spy on you through your dishwasher”:
Items of interest will be located, identified, monitored, and remotely controlled through technologies such as RFID, sensor networks, tiny embedded servers, and energy harvesters — all connected to the next-generation internet using abundant, low-cost, and high-power computing…
Transformational is an overused word, but I do believe it properly applies to these technologies, particularly to their effect on clandestine tradecraft.
To summarise: smart healthcare, smart cars, and smart homes, all bring new vulnerabilities as well as new benefits. The same is true for other fields of exponentially improving technology, such as 3D printing, unmanned aerial vehicles (“drones”), smart toys, and household robots.
The rise of robots
Sadly, malfunctioning robots have already been involved in a number of tragic fatalities. In May 2009, an Oerlikon MK5 anti-aircraft system was part of the equipment used by 5,000 South African troops in a large-scale military training exercise. On that morning, the controlling software suffered what a subsequent enquiry would call a “glitch”. Writing in the Daily Mail, Gavin Knight recounted what happened:
The MK5 anti-aircraft system, with two huge 35mm cannons, is essentially a vast robotic weapon, controlled by a computer.
While it’s one thing when your laptop freezes up, it’s quite another when it is controlling an auto-loading magazine containing 500 high-explosive rounds…
“There was nowhere to hide,” one witness stated in a report. “The rogue gun began firing wildly, spraying high explosive shells at a rate of 550 a minute, swinging around through 360 degrees like a high-pressure hose.”
By the time the robot has emptied its magazine, nine soldiers lie dead. Another 14 are seriously injured.
Deaths due to accidents involving robots have also occurred throughout the United States. A New York Times article in June 2014 gives the figure of “at least 33 workplace deaths and injuries in the United States in the last 30 years.” For example, in a car factory in December 2001,
An employee was cleaning at the end of his shift and entered a robot’s unlocked cage. The robot grabbed his neck and pinned the employee under a wheel rim. He was asphyxiated.
And in an aluminium factory in February 1996,
Three workers were watching a robot pour molten aluminium when the pouring unexpectedly stopped. One of them left to flip a switch to start the pouring again. The other two were still standing near the pouring operation, and when the robot restarted, its 150-pound ladle pinned one of them against the wall. He was killed.
To be clear, in none of these cases is there any suggestion of foul play. But to the extent that robots can be remotely controlled, the possibility arises for industrial vandalism.
Indeed, one of the most infamous cases of industrial vandalism (if that is the right description in this case) is the way in which the Stuxnet computer worm targeted the operation of fast-spinning centrifuges inside the Iranian programme to enrich uranium. Stuxnet took advantage of at least four so-called “zero-day security vulnerabilities” in Microsoft Windows software – vulnerabilities that Microsoft did not know about, and for which no patches were available. When the worm found itself installed on computers with particular programmable logic controllers (PLCs), it initiated a complex set of monitoring and alteration of the performance of the equipment attached to the PLC. The end result was that the centrifuges tore themselves apart, reportedly setting back the Iranian nuclear programme by a number of years.
Chillingly, what Stuxnet could do to centrifuges, variant software configurations could have similar effects on other industrial infrastructure – including energy and communication grids.
Therefore, whereas there is much to celebrate about the growing connectivity of “the Internet of Things”, there is also much to fear about it.
The scariest book
Many of the examples I’ve briefly covered above – the hacking of embedded medical devices, vehicle control systems, and thermostats and lightbulbs – as well as the upsides and downsides of “the rise of robots” – are covered in greater detail in a book I recently finished reading. The book is “Future Crimes”, by former LAPD police officer Marc Goodman. Goodman has spent the last twenty years working on cyber security risks with organisations such as Interpol, NATO, and the United Nations.
The full title of Goodman’s book is worth savouring: “Future Crimes: Everything is connected, everything is vulnerable, and what we can do about it.” Singularity 1on1 podcast interview Nikola Danaylov recently described Future Crimes as “the scariest book I have ever read in my life”. That’s a sentiment I fully understand. The book has a panoply of “Oh my god” moments.
What the book covers is not only the exponentially growing set of vulnerabilities that our exponentially connected technology brings in its wake, but also the large set of people who may well be motivated to exploit these vulnerabilities. This includes home and overseas government departments, industrial competitors, disgruntled former employees, angry former friends and spouses, ideology-fuelled terrorists, suicidal depressives, and a large subset of big business known as “Crime Inc”. Criminals have regularly been among the very first to adopt new technology – and it will be the same with the exploitation of new generations of security vulnerabilities.
There’s much in Future Crimes that is genuinely frightening. It’s not alone in the valuable task of raising public awareness of increasing security vulnerabilities. I also recommend Kim Zetter’s fine investigative work “Countdown To Zero Day: Stuxnet and the launch of the world’s first digital weapon”. Some of the same examples appear in both books, providing added perspective. In both cases the message is clear – the threats from cybersecurity are likely to mushroom.
On the positive front, technology can devise countermeasures as well as malware. There has long been an arms race between software virus writers and software antivirus writers. This arms race is now expanding into many new areas.
If the race is lost, it means that security will eat the world in a bad way: the horror stories that are told throughout both Future Crimes and Countdown To Zero Day will magnify in both number and scope. In that future scenario, people will look back fondly on the present day as a kind of innocent paradise, in which computers and computer-based systems generally worked reliably (despite occasional glitches). Safe, clean computer technology might become as rare as bottled oxygen in an environment where smog and pollution dominates – something that is only available in small quantities, to the rich and powerful.
If the race is won, there will still be losers. I’m not just referring to Crime Inc, and other would-be exploiters of security vulnerabilities, whose ambitions will be thwarted. I’m referring to all the companies whose software will fall short of the security standards of the new market leaders. These are companies who pay lip service to the importance of robust, secure software, but whose products in practice disappoint customers. By that time, indeed, customers will long have moved on from preferring dancing pigs to good security. The prevalence of bad news stories – in their daily social media traffic – will transform their appreciation of the steps they need to take to remain as safe as possible. Their priorities will have changed. They’ll be eagerly scouring reports as to which companies have world-class software security, and which companies, on the other hand, have products that should be avoided. Companies in the former camp will eat those in the latter camp.
Complications with software updates
As I mentioned above, there can be security vulnerabilities, not only intrinsic in a given piece of software, but also in how that software is used, deployed, and updated. I’ll finish this article by digging more deeply into the question of software updates. These updates have a particularly important role in the arms race between security vulnerabilities and security improvements.
Software updates are a key part of modern technological life. These updates deliver new functionality to users – such as a new version of a favourite app, or an improved user interface for an operating system. They also deliver security fixes, along with other bug fixes. In principle, as soon as possible after a major security vulnerability has been identified and analysed, the vendor will make available a fix to that programming error.
However, updates are something that many users dislike. On the one hand, they like receiving improved functionality. But they fear on the other hand that:
- The upgrade will be time-consuming, locking them out of their computer systems at a time when they need to press on with urgent work
- The upgrade will itself introduce new bugs, and break familiar patterns of how they use the software
- Some of their applications will stop working, or will work in strange ways, after the upgrade.
The principle of “once bitten, twice shy” applies here. One bad experience with upgrade software – such as favourite add-on applications getting lost in the process – may prejudice users against accepting any new upgrades.
My own laptop recently popped up an invitation for me to reserve a free upgrade from its current operating system – Windows 7.1 – to the forthcoming Windows 10. I confess that I have yet to click the “yes, please reserve this upgrade” button. I fear, indeed, that some of the legacy software on my laptop (including apps that are more than ten years old, and whose vendors no longer exist) will become dysfunctional.
The Android operating system for smartphones faces a similar problem. New versions of the operating system, which include fixes to known security problems, often fail to make their way onto users of Android phones. In some cases, this is because the phones are running a reconfigured version of Android, which includes modifications introduced by a phone manufacturer and/or network operator. Any update has to wait until similar reconfigurations have been applied to the new version of the operating system – and that can take a long time, due to reluctance on the part of the phone manufacturer or network operator. In other cases, it’s simply because users decline to accept an Android upgrade when it is offered to them. Once bitten, twice shy.
Accordingly, there’s competitive advantage available, to any company that makes software upgrades as smooth and reliable as possible. This will become even more significant, as users grow in their awareness of the need to have security vulnerabilities in their computer systems fixed speedily.
But there’s a very awkward problem lurking around the upgrade process. Computer systems can sometimes be tricked into installing malicious software, whilst thinking it is a positive upgrade. In other words, the upgrade process can itself be hacked. For example, at the Black Hat conference in July 2009, IOActive security researcher Mike Davis demonstrated a nasty vulnerability in the software update mechanism in the smart electricity meters that were to be installed in homes throughout the Pacific North West of the United States.
For a riveting behind-the-scenes account of this particular research, see the book Countdown To Zero Day. In brief, Davis found a way to persuade a smart meter that it was being offered a software upgrade by a neighbouring, trusted smart meter, whereas it was in fact receiving software from an external source. This subterfuge was accomplished by extracting the same network encryption key that was hard-wired into every smart meter in the collection, and then presenting that encryption key as apparent (but bogus) evidence that the communication could be trusted. Once the meter had installed the upgrade, the new software could disable the meter from responding to any further upgrades. It could also switch off any electricity supply to the home. As a result, the electricity supplier would be obliged to send engineers to visit every single house that had been affected by the malware. In the simulated demo shown by Davis, this was as many as 20,000 separate houses within just a 24 hour period.
Uncharitably, we might think to ourselves that an electricity supplier is probably the kind of company to make mistakes with its software upgrade mechanism. As Mike Davis put it, “the guys that built this meter had a short-term view of how it would work”. We would expect, in contrast, that a company whose core business was software (and which had been one of the world’s leading software companies for several decades) would have no such glitches in its system for software upgrades.
Unexpectedly, one of the exploits utilised by Stuxnet team was a weakness in part of the Microsoft Update system – a part that had remained unchanged for many years. The exploit was actually used by a piece of malware, known as Flame which shared many characteristics with Stuxnet. Mikko Hyppönen, Chief Research Officer of Finnish antivirus firm F-Secure, reported the shocking news as follows in a corporate blogpost tellingly entitled “Microsoft Update and The Nightmare Scenario”:
About 900 million Windows computers get their updates from Microsoft Update. In addition to the DNS root servers, this update system has always been considered one of the weak points of the net. Antivirus people have nightmares about a variant of malware spoofing the update mechanism and replicating via it.
Turns out, it looks like this has now been done. And not by just any malware, but by Flame…
Flame has a module which appears to attempt to do a man-in-the-middle attack on the Microsoft Update or Windows Server Update Services system. If successful, the attack drops a file called WUSETUPV.EXE to the target computer.
This file is signed by Microsoft with a certificate that is chained up to Microsoft root.
Except it isn’t signed really by Microsoft.
Turns out the attackers figured out a way to misuse a mechanism that Microsoft uses to create Terminal Services activation licenses for enterprise customers. Surprisingly, these keys could be used to also sign binaries…
Having a Microsoft code signing certificate is the Holy Grail of malware writers. This has now happened.
Hyppönen’s article ends with some “good news in the bad news” which nevertheless sounds a strong alarm about similar things going wrong (with worse consequences) in the future:
I guess the good news is that this wasn’t done by cyber criminals interested in financial benefit. They could have infected millions of computers. Instead, this technique has been used in targeted attacks, most likely launched by a Western intelligence agency.
How not to be eaten
Despite the threats that I’ve covered above, I’m optimistic that software security and software updates can be significantly improved in the months and years ahead. In other words, there’s plenty of scope for improvements in the quality of software security.
One reason for this optimism is that I know that smart people have been thinking hard about these topics for many years. Good solutions are already available, ready for wider deployment, in response to stronger market readiness for such solutions.
But it will take more than technology to win this arms race. It will take political resolve. For too long, software companies have been able to ship software that has woefully substandard security. For too long, companies have prioritised dancing pigs over rock-hard security. They’ve written into their software licences that they accept no liability for problems arising from bugs in their software. They’ve followed, sometimes passionately, and sometimes half-heartedly, the motto from Facebook’s Mark Zuckerberg that software developers should “move fast and break things”.
That kind of behaviour may have been appropriate in the infancy of software. No longer.
dw2 