Monthly Archives: February 2012

AFPM Security Conference and a question on motivation to secure systems

Hello all.

Just wrapping up another lovely trip to Houston right in the throes of the annual rodeo.  I had the pleasure of being a panel member at the AFPM (you know, the trade association formerly known as NPRA?) Security Conference.  This is primarily a physical security conference but they asked a number of us cyber types to sit on a panel and discuss the convergence of cyber and physical security efforts and programs.  Overall the feedback and participation was good and there are a number of efforts on this subject underway.  If you are interested in becoming part of the AFPM Cyber Security Subcommittee, please let me know, or if you are interested in participating in the ONG SCC Cyber Working Group (also a cyber-working-with-physical effort), then just drop me a note at rick.kaun@honeywell.com.

Now on to my question for the week.  I had the pleasure of speaking with a number of owner-operator types this week and I continue to be astounded by the number of reasons why security or IT management and maintenance in a production setting still fails to be more important.  One source suggested to me the interia comes from his management holding a more traditional view of plant management and maintenance whereby plant-based systems are typically purchased, installed and then run for 10 to 20 years before they are ever replaced.  His point is that trying to get 3 to 5 year refresh or maintenance budget in that culture is what stymies him.

A second source shared with me how open and supportive (and therefore opposite to the above example) his organization was on the topic.  They have a significant emphasis not only on cyber but on physical and corporate as well.  It was actually the first time I have seen a job title with both cyber and physical security in it.  When asked what the culture or motivation was for this perspective it quickly became apparent to me that this organization had a cyber or security sympathetic ear at a high level of the organization.  This coupled with a relatively recent “aha” moment in which a critical electronic data source just about melted down is how this organization has come to support and embrace security.  (Upon further investigation it turns out the overall effort is referred to as “risk management,” which is, of course, a common term just not usually extended to process control cyber efforts per se).

So my question to you is this:  What is it, in your opinion, that does successfully motivate organizations to do the right thing?  Or, if you can’t put your finger on it, what, in your organization, is the biggest obstacle to getting security elevated in priority?  Is it cultural?  The second example claims his company is one of the cheapest he has ever worked for so it can’t be deep pockets that allow for more “toys” to be deployed.  What do you think?

Leave a comment
Posted in General, Petrochemicals – CFATS. Bookmark the permalink.

NERC CIP – An update from our man in the field, Tom Alrich

As always, Tom is nothing if not detailed and thorough.  What follows is his latest offering, verbatim.  Sorry for the length but this is not your average water cooler topic.  And as usual, the comments, thoughts, observations and projections are solely those of Tom and his personal and professional opinion.  This is not an official corporate position or interpretation in any way.  Grab a cup of caffeine and enjoy!

 CIP Version 5: About those Large Plants…..

Tom Alrich, tom.alrich@honeywell.com

The opinions expressed in this article are those of the author, not necessarily of Honeywell International, Inc.

 It is time to talk about CIP-002-5 Attachment 1 paragraph 2.1, the “bright-line” criterion in CIP Version 5 that brings large generation stations into scope as Medium Impact facilities.  This reads (in the draft sent out on Feb. 17, and appending the preamble):

Each BES Cyber System, not included in Section 1, above, associated with the following:

2.1 Commissioned generation, by each group of generating units at a single plant location, with an aggregate highest rated net Real Power capability of the preceding 12 calendar months equal to or exceeding 1500 MW in a single Interconnection. For each group of generating units, the only BES Cyber Systems that meet this criterion are those shared BES Cyber Systems that could, within 15 minutes, adversely impact the reliable operation of any combination of units that in aggregate equal or exceed 1500 MW in a single Interconnection.

There are two big issues related to this, and they’re intertwined:

  1. Will FERC let the 1500MW threshold stand when they approve Version 5? 
  2. What does the second sentence mean, and what does it imply for the number of Medium Impact plants that will have BES Cyber Systems under Version 5?  Will FERC be satisfied with this number?

Both of these questions have big implications for the amounts that owners of large generating stations will have to spend to comply with CIP Version 5 (and Version 4 as well if that is the next version, since it contains similar language on both questions).  And other questions need to be considered in answering these two.  Let’s start with the first question.

I. The 1500MW Threshold – will it stand?

The question whether 1500MW will be a low enough threshold to satisfy FERC is certainly one I can’t answer.  I can point to the NERC survey results[1] from last year that showed that, excluding blackstart units (which is a separate discussion.  See our recent blog post), approximately 7.2 percent of total US generating units[2] would be part of Critical Assets under the Version 4 bright-line criteria (which are essentially identical to the Version 5 criteria for non-blackstart generating plants to be Medium Impact facilities[3]).

If FERC does lower the threshold, what will they lower it to?  1000MW?  750MW?  Even 500MW?  Unfortunately, the survey data don’t allow us (or FERC) to draw any conclusions about the number of additional plants or units that will come into scope with this change.  So at the moment, the main question is whether FERC will be pleased with 7.2 percent of non-blackstart generating units being part of a plant that will be a Medium Impact facility under CIP Version 5 or a Critical Asset under Version 4.[4] 

II. What about the Cyber Systems in the Plants?

How many Medium Impact generating stations will have BES Cyber Systems under CIP Version 5?  This second question is very similar to the question in last year’s survey, about the number of generating units that are part of plants that both are Critical Assets and have Critical Cyber Assets.  In the survey, that number was 4.4% of total generating units.[5] 

Of course, since Version 5 deals with BES Cyber Systems instead of Critical Cyber Assets, we really want to ask, “How many non-blackstart plants will be Medium Impact facilities with BES Cyber Systems under Version 5?”  Putting aside the difference between a CCA and a BES Cyber System (which probably isn’t big enough to make a huge difference in itself in the answer to this question), the important question is the impact of the second sentence of criterion 2.1 above (“For each group….”).

Just to be clear, what this second sentence says is that a cyber system (such as a Distributed Control System) in a 1500MW+ generating station has to by itself control generation in excess of 1500MW.  And since a cyber system controls “generation” by virtue of its controlling one or more generating units, what this really means is that the system has to control a number  of units whose total generating capacity exceeds 1500MW.[6] 

The catch here, of course, is that there are few cyber systems at 1500MW+ plants that by themselves control more than 1500MW; in fact, it is possible there aren’t any at all in the US.  This is because there are few if any large plants whose units are all controlled by the same DCS.  We have found that usually a DCS will control one or at most two units at a large plant (this is often because units were installed at different times).  Since the units are typically between 250 and 500MW, this means very few DCS will control 1500MW.  And practically, that means there will be few if any 1500MW+ plants that – under CIP Version 5 – will actually have to apply anything more than the procedural controls that apply to all Low Impact facilities.

If you saw our November post on this question, you know that we (myself and Donovan Tindill) considered this situation to be very dangerous – i.e. inviting drastic action by FERC or even Congress to “correct” this issue.  I have since modified my opinion after talking with others in the industry.  They point out that, assuming proper separation of the cyber systems, there is no difference between a 1600MW plant with say four 400MW units each controlled by its own DCS, and four 400MW stand-alone plants.  I certainly agree with that point, but then the question turns to “proper separation”.  And when you start thinking about that, it gets more interesting (and much less clear).

At this point, I want to say that I am not advocating any particular course of action on the Version 5 bright-line criteria.  And I’m not saying that they are currently “too weak” from the point of view of securing the grid against cyber attack.  What I do want to discuss is what FERC’s reaction will be if criterion 2.1 is left as is vs. if changes are made. 

Of course, discussing FERC’s possible reaction is inherently a speculative process.  Let’s go to the assumption of proper separation of the DCS systems (or rather, of the networks on which they are implemented).  It goes without saying (but I will anyway) that the only way a 1600MW plant (with four DCS each controlling 400MW on its own network) can be as cyber-safe as four 400MW plants is if there is no connection at all between the networks. 

Now, FERC certainly knows this, given the number of power industry cyber security experts they have on staff, and the fact that staff members attend all of the SDT meetings and actively participate in the conversations.  And the five Commissioners never reveal how they’re thinking to their own staff, let alone to outsiders.  But what will be their reaction if criterion 2.1 makes its way to them unchanged from what is in the current draft?

If I were a FERC staff member advising the Commission, I would ask several questions (again, I’m sure they’ve already thought of these): 

  1. Let’s say the owner of a large generating station asserts that they don’t have BES Cyber Systems[7], because they don’t have a single system that controls 1500MW of generation as required by criterion 2.1.  How do you know they are telling the truth?  There is no auditing of this assertion in the current V5 draft.
  2. But let’s say FERC gets beyond this point, and has no reason to believe the entity is lying about how much generation a DCS controls.  The next question is, “Are the networks on which each DCS is implemented sufficiently separated so that a cyber attack will never propagate to the other networks?”  Because if that isn’t the case, then it doesn’t matter if the DCS controls just 400MW or not – if the networks aren’t separated and the DCS is attacked, the attackers can still bring down the rest of the plant.  Again, there is no provision in the current Version 5 draft to be able to answer this question.  Not that there couldn’t be, just that there isn’t now.
  3. And let’s now say that FERC is satisfied that the networks are separated.  They will finally ask, “How can you guarantee that the networks won’t – through deliberate act or mere misconfiguration – be linked up again in the future?”  This is certainly a reasonable question, since I know that we see this happening all the time on industrial networks – someone just plugs a cable into a switch that connects to a device on a supposedly separate network.  What this might require is new CIP requirements for maintaining separation of internal networks, when the ‘exemption’ in criterion 2.1 is claimed.  And, of course, regular auditing to see those requirements are being met.

 As I said, I’m not advocating a course of action here.  But if nothing is done and Criterion 2.1 is approved by NERC and goes to FERC as written, I believe FERC will take some action to address the three questions above.  What action could they take?

One thing they could do is reject CIP Version 5 altogether and send it back for more work.  Given the amount of time they’ve been waiting for Version 5, that doesn’t seem likely.  Another would be to approve Version 5 but at the same time require that this issue be addressed (probably by putting in new requirements and measures to address the three questions above) in a new version (CIP Version 6!).  The third would be to just make changes themselves before approving Version 5.[8]

If they do go for the third option, they could then a) develop their own new requirements to verify the scope of control of a DCS, and to ensure network separation is implemented and maintained; b) strike the second sentence out of criterion 2.1; or c) simply lower the 1500MW threshold. 

I don’t think a) is likely, because it would be a lot of work to develop the new requirements themselves.  I think b) is possible, but I think c) is the most likely scenario.  However, if they do lower the threshold in MW, it’s very unlikely they’ll stop at 1000MW (since that may not add a single plant to the category of “large generating station with Medium Impact BES Cyber Systems). 

Rather, I think FERC will bring the criterion 2.1 threshold down to 750 or even 500MW.  At those levels, they will finally start to get some Medium Impact BES Cyber Systems being reported at large generating stations.  Of course, the “collateral damage” of all this will be that any plant rated at 500 or 750MW or higher will now be a Medium Impact one.  But this could also be part of their intention all along (this gets back to the first question at the top of this post, of course).  I have now reached the limit of how far I’ll go in trying to guess FERC’s future actions.

How can such action by FERC be averted?  I think that would require the SDT to squarely address the three questions listed above – which means probably new requirements and measures, perhaps even in a new standard like CIP-012.  However given that it is late February now and the SDT has to finalize the standards in advance of the late March posting date for the second ballot; they have neither the time nor the inclination to address this.

The moral of this story?  I think it very likely FERC will lower the MW threshold in CIP-002-5 Attachment 1, criterion 2.1 to 750 or even 500MW, before approving Version 5.  The best outcome that can be hoped is that they will also promise to raise it again (probably not back to 1500, though) if a new version of CIP that addresses the three questions above (and perhaps others that FERC will raise in their approval order) is developed.  So Version 6 is coming!


 

[1] This link is to our (corrected) blog post about the survey results.  I just tried in vain to find the actual NERC document on the NERC web site, but – as with most of my searches on that benighted site – I couldn’t find it.  If you want to see the document, email me and I’ll send it to you.

[2] Of course, a single plant can have up to ten units, with the average being maybe two or three.  But since the plants that these units might be part of are mostly larger ones, it stands to reason that the percent of generating stations that would be Critical Assets under Version 4 (or Medium Impact under V5) would be much lower than 7.2 percent.  This is because the numerator would be divided by a figure like 5 or 6, while the denominator would be divided by 2 or 3.

[3] In both Versions 4 and 5, there are two criteria that apply to non-blackstart plants: the 1500MW criterion and a criterion that corresponds roughly to “Reliability Must Run” plants.  The wording for both criteria is very similar between the two versions.

[4] I wish the survey had covered plants, not generating units, since only a plant can be a Critical Asset (or a Medium Impact facility under Version 5). 

[5] Again, this will result in a much lower percentage of plants that are Critical Assets with Critical Cyber Assets.

[6] CIP-002 Version 4 has a similar ‘exemption’, although it is stated not in the criteria in CIP-002-4 Attachment 1, but in requirement R2.

[7] For those who are familiar with the draft of Version 5 that was voted on in December but have not followed the discussion since then, the SDT has decided to be consistent and have all requirements apply to BES Cyber Systems, not BES Cyber Assets.  The latter are still defined, and that definition is still fundamental to Version 5.  But the entity can group these cyber assets into systems of one or more cyber assets for compliance purposes.  This (and other changes like fewer TFEs) should make the compliance process easier than in Versions 1-3.

[8] There might not be a way, under the current NERC Rules of Procedure, that FERC can simply make a change before approving the full set of CIP Version 5 standards.  But they effectively do have this power – as well as to do anything else they want, like write a standard from scratch – now.  See paragraphs 309 and 321, which were added to the Rules of Procedure in 2010.

1 Comment
Posted in General, Power – NERC CIP. Bookmark the permalink.

Information sharing – will it work? Maybe, but depends on who hosts.

We have spoken before here on this blog about the value of information sharing in the face of ongoing events or emerging threats.  In essence, we stated the opinion that too many people were concerned about how much specific data they would be able or willing to share and then if you dilute the data too far, you get less value.  A second challenge to the concept would be who hosted (protection of data), and would the contributors have the time/inclination to post anything anyway.  I mean if you are running a facility and the world as you know it is going haywire, it is not highly likely you are going to log onto an external web site, post important information and then turn back to your immediate crisis.

Wrote that a while ago and still stand by it.  However, I have now had more time and more discussions with related but somewhat removed people.  As in, people who get cyber security and know way more about the bad guys than I will ever know (or want to for that matter), but who are also removed enough from the day-to-day challenges of process control and so focused on security that their perspective is quite simple and honest.

During those discussions I came to a realization.  I have always agreed that there is value in sharing information about emerging threats.  Who holds that info and how it is disseminated is the challenge.  The accuracy or validity of that information is also important to its value and therefore of benefit to the consumers of that data.  Many forward-thinking organizations have already come to realize this and have formed their own security watchdog groups whose sole purpose is to monitor the cyber landscape and protect their organization.  This is very costly and requires very specialized skill sets in that department.  Now why, I ask, would these organizations share that information?  There is no reason for them to do so and they have spent significant dollars on the cultivation of that information.  So what is the solution?

A third party, wholly dedicated to posting real time cyber and physical security information, as it happens.  I mean, think about it for a minute.  A user-contributed, central information web or storage site is limited in its data and likely shared after the fact.  The perennial day late and a dollar short.  Now what if a trusted, third-party organization were to develop, fund and then execute a top clearance cyber watchdog group?  And what if that organization were to “sell” early warning data/communications as they emerged?  All sizes and shapes of organizations would be able to afford and therefore benefit from the service.  More importantly the data would be “fresh” or up-to-date as it happens.  Additional services from that source might include firewall rule set adjustments as threats emerge.  “Cleaning” instructions and/or services post-incident (should it get through) and general information/bulleting sharing.  Would you pay for it?  Who do you think would be a good candidate to buy it from?  I know there are those of you who will point out that HSIN and various governments already have CERT warnings and initiatives, but are they as effective and succinct as you need?  What do you think?

P.S. We are co-speakers with the DHS in March on a Chemical Processing webinar. Be sure to sign up soon!

 

Leave a comment
Posted in General. Bookmark the permalink.

NERC CIP V5 Fallout

Hello all.  Sorry for the delay but we took a day or two off.  As usual though, we have a solid topic on tap for today.  We have the double treat of both Tom and our senior security guy combining on today’s perspective.  Read on to find out more!

 

The Blackstart Question in CIP Version 5 

Tom Alrich, tom.alrich@honeywell.com

Donovan Tindill, donovan.tindill [@] honeywell.com 

In our previous post, we mentioned that, at the January SDT meeting, paragraph 2.4 of CIP-002-5 Attachment 1 (which is the bright-line criterion for blackstart generating stations) was amended to include an ‘exemption’ when the station was not connected using a routable protocol or dialup.  We also pointed out that this change (like any other change to V5) was still subject to approval by other NERC groups as well as FERC.

 A paper by a NERC staff member was just sent out to the SDT Plus list (and email one of us if you would like to know how to get on that list) that makes the official argument why this change should be made – although it also clearly states that it is not an SDT document.  We would like to add our $.02 to the argument in this paper, which we agree with.

Here is the problem: As the paper mentions, blackstart resources are older, smaller units with very low capacity factors and limited revenues.  From our experience, many are hydro units with single or double-digit megawatt capacities, built over 60 years ago.  A unit may be considered beneficial for blackstart because it is geographically located near a larger nuclear or coal-fired facility.  Although de-regulation allows East-coast power to be sold to the West, technically the grid is unable to do this efficiently and geographic location of the current infrastructure is how it really operates.  A lot of Blackstart resources have avoided full CIP Version 3 compliance because many are still operating on very old non-routable infrastructure and therefore have no Critical Cyber Assets.

Even before CIP Version 5 is approved, we are already seeing companies re-classifying their facilities and withdrawing themselves as a Blackstart participant.  In the case of one Generation Owner, the annual revenue for being on the Blackstart list is less than $50,000 per year.  But the capital costs to achieve NERC CIP compliance and annual sustainment could never be recovered, even in 10 years with $500k.  Many asset owners are making the decision that it is simply cheaper to withdraw Blackstart status and avoid the added expense and headache of NERC CIP Compliance.

We feel the NERC investigation will reveal that entities are already withdrawing their Blackstart status, and that NERC CIP Version 5 will further accelerate this trend.  Criterion 2.4 in the first draft of Version 5 unintentionally reduces grid reliability (in the area of blackstart only).

What do you think?

Leave a comment
Posted in General, Power – NERC CIP. Bookmark the permalink.