Smart Grid Cybersecurity Vulnerabilities Revealed

It’s a simple equation: spend pennies per unit to secure your smart grid gear now, or thousands of dollars per unit to replace it later.

Smart grid cybersecurity just got a lot harder to ignore.

This week has brought a new alert from the U.S. Department of Homeland Security, detailing some important cybersecurity vulnerabilities contained within some critical smart grid gear. Backing that up, Greentech Media has been briefed on an in-depth report on how one utility found similar vulnerabilities that forced it to replace millions of dollars of smart grid systems -- or face the threat of a potentially catastrophic hack attack.

What’s comforting is how cheap these kinds of security problems can be to prevent, if they’re planned for in advance. What’s scary, however, is how much they may end up costing to fix after the fact -- or worse yet, what kind of damage they could cause if exploited.

First, the news: On Monday, DHS’s Industrial Control System Cyber Emergency Response Team  (ICS-CERT) issued an alert (PDF) based on research that independent SCADA security researcher Rubén Santamarta had published on his blog. The report said that a programmable logic controller (PLC) made by grid giant Schneider Electric contains hard-coded credentials that could be exploited by an attacker to bypass the devices’ security mechanism and access its controls to view or alter the module's firmware, execute arbitrary code, or cause a denial-of-service attack.

These PLC devices use operating software called VxWorks, made by Wind River Systems, one of the biggest unsung providers of simple software meant to run the barely-smart remote devices that now outnumber PCs as endpoints to the internet. Smart grid devices make up a significant number of those devices -- the switches, capacitors, circuit breakers and other high-voltage gear that keeps the grid humming.

In other words, the kind of stuff you don’t want to hand control of to a teenage hacker or industrial cyber-saboteur, said Kurt Stammberger, vice president at Mocana, a San Francisco-based cybersecurity firm.

But when a major Southern California utility (he’s not saying which one) asked Mocana to test the cyber-vulnerability of its remote terminal units (RTUs) being installed in the thousands to run the utility’s substation, it took just one day to find a way hackers could penetrate that network, take over all those RTUs, and perhaps even blow up transformers, black out neighborhoods, and create other types of havoc on an utility-wide scale, based on a few lines of simple code, Stammberger told me in an interview earlier this year.

Putting the Pieces Together

The vulnerabilities described in Monday’s report from ICS-CERT and Santamarta’s blog sound “very similar to the ones we found last year on the Telvent Sage 3030 RTU” deployed at the unnamed Southern California utility, Stammberger told me in a Friday email. Schneider bought Telvent for $2 billion in June, so the two reports could be identifying similar devices.

The federal report said that Schneider Electric has developed fixes for only the most recent versions of firmware for two of the four products that might have the vulnerabilities. Of the four Schneider Electric products, each may be running one of a number of different versions of firmware with the vulnerabilities, the report stated.

So what are those vulnerabilities? While we can’t draw a direct line between this week’s ICS-CERT report and what Mocana discovered with its utility customer, here’s how Stammberger described the security flaws in the devices Mocana tested.

First, “Our analyst was able to show that he could find and alter any memory record on the Telvent RTU, remotely, over the network,” he wrote. Second, “Our analyst was able to extract administrative credentials (without permission) from the Telvent RTU.” Third, “Our analyst was able to extract live administrative session tokens (such as web cookies) from the Telvent RTU.”

All that was made possible “because SSL (secure sockets layer) was not implemented on the admin interfaces”; because “the VxWorks WDB service was exposed by default, and could not be deactivated"; and because "administrative credentials were not stored using a cryptographic hash,” he wrote.

We’re not giving away any secrets here -- this particular hack has been known since last year, when HD Moore, chief security officer at Rapid7, announced it to the world. Lots of manufacturers had to spend a lot of money to upgrade software to protect their remote-control devices against the malware, which exploited systems that were shipped with the VxWorks remote debugging feature permanently turned "on."  

Easy to Prevent, Hard to Fix -- Too Dangerous to Leave Alone

The people who build VXWorks have fixed the problem in the new version of their operating system, Stammberger said. But the RTUs that Mocana tested were built with so little memory and processing power that they probably couldn’t run the newest version of VxWorks that fixed the problem, he said.

Nor could they be adequately protected by a firewall, due to the nature of the exploit, as well as the potential for physically introducing the malware, say, via USB stick. The utility didn't even have the option of disabling the debugging feature.

In other words, keeping the units secure would require either disconnecting them from the network -- which would render them useless -- or replacing them, which is what the utility and vendor did, he said.

“It’s outrageous,” was Stammberger’s final verdict. “For what’s essentially a software patch, the utility has to throw away all this hardware.”

Still, it’s much preferable to leaving the vulnerabilities out there. Here are a few examples Stammberger shared from Mocana’s Q-and-A with its utility customer about just how dangerous these unsecured devices could be out in the real-world grid:

Q: Would these vulnerabilities allow me to blackout any part of the electrical grid that this RTU comes into contact with? Could I digitally shut down power to substations that this RTU controls?

A: Yes. Compromise of the RTU unit of the types we have described will result in the ability to enact, at will, any action the RTU unit is capable of performing. Since direct access to memory can be achieved, there are no permissions the application(s) can enforce that will negate an attacker’s ability to alter the system or its peripherals.

Q: Would these vulnerabilities allow me to override or reprogram the line protection equipment, and induce line overloads that might cause fires, explosions or physically damage substations?

A: Possibly. Any line protection equipment that is connected to this RTU can indeed be controlled or overridden based on the attacker’s ability to modify the driver. However, it’s probable that a utility has other controls in place (like fuses) between the RTU and other devices or electrical “end points” that could mitigate potential damage from line overloads.

Q: What are the ways that someone could initiate this kind of attack?

A: As long as the attacker can connect to the RTU over a network, it can be compromised. The attacker is not required to be on the same network sub-net as the device, nor is the attacker required to have physical access to the RTU -- so whether or not the RTU is in a locked room is immaterial. A visitor could also introduce a thumb-drive into any PC on the same network. That could introduce a program that would enable an attacker to connect to the RTU from outside the private network.

The Price for Smart Grid Security: Upgrade -- or Replace

Therein lies a hard metric for the cost of smart grid security. Building the RTUs with the bare minimum computing power, rather than giving them some headroom for future security upgrades, probably saved the manufacturer “pennies per unit,” on RTUs that cost “many, many thousands of dollars” apiece, Stammberger said. But replacing them was going to cost their customer millions.  

“If I were that Southern California utility, I think that would be question number one on my phone calls to the technology supplier,” he said. “This is a well-documented, easy-to-fix vulnerability, and your fix is to 'spend another $5 million?' Thanks a lot -- I’ll go to your competitors.”

The scary thing is just how ordinary this all is. Hackers have grown with, and often within, the IT and internet revolution. But it took 10 years for the IT industry to see the level of attacks on the traditional PC-connected internet that have already arisen against the far more numerous “dumb” devices in the few years they’ve been out there in any numbers, Stammberger said.

Amidst all the vague promises from the industry that smart grid cybersecurity is being taken care of, and on the other end of the spectrum, the dire threats of Stuxnet variants infecting the world’s power grids, real-world examples like these help clarify the issue of smart grid security costs and benefits. Utilities that ignore these emerging vulnerabilities in their existing gear -- or allow vendors to ignore them in their new smart grid gear -- are asking for trouble. 

Of course, we’ve got to be careful not to let speculation get in the way of sober, rational analysis. Last month’s report of a potential foreign hack of an Illinois water utility’s SCADA system has turned out to be a mere misunderstanding, with DHS and the FBI finding that it was a utility contractor logging in to the system while on a trip to Russia that caused the confusion.  

But in the case of this newest revelation of potentially serious cybersecurity threats to a key set of smart grid components, there may be more to come.

“The ball is in DHS's court,” Stammberger wrote Friday. “Let's hope the increased scrutiny motivates the industrial automation vendors to take security more seriously.”

via greentechmedia.com