Monthly Archives: January 2012

SDT Meeting updates – Or, an informal open letter

Hello all.  Missed a week but have been really busy!  Today I am happy to report we have a fresh update from our man in the field Tom Alrich.  What you read below is the first of two reports on specific developments last week.  The SDT team met in Taylor Texas last week and by all accounts is sounds like significant progress is being made!  What follows is Tom’s report in its entirety.  These are 100% Tom’s observations and are therefore 100% his view and not that of any person, company or organization.  Now it is a lengthy read but for those who know CIP, it is not a simple subject.  As always we welcome your thoughts and observations.  Now grab a fresh coffee and settle in: 

NERC CIP Version 5: Progress to Report

 Tom.alrich@honeywell.com

Note: The opinions expressed in this open letter are those of the author, not those of Honeywell International or any of its divisions).

I’ve just returned from the NERC CSO706 (CIP) Standards Drafting Team in Taylor, Texas.  This was the best-attended SDT meeting in two years, with something like 25-30 industry observers, plus almost all of the SDT members.  The purpose of the meeting was to start on the road to recovery from the overwhelmingly negative first ballot on CIP Version 5, by starting to develop a new draft that will pass on the next ballot (which is scheduled to start with posting of the new draft in late March).  I am pleased to report that the SDT did make a lot of progress, and I am almost certain the second draft of Version 5 will be much more acceptable to the NERC membership than was the first draft.

Because there was concrete progress, I would like to discuss what I see as the most important steps that were taken.  Because asset identification (CIP-002) is the foundation for the rest of the CIP standards, and because I (along with about two thirds of the other observers) attended the CIP-002 subgroup meetings, that is what I will focus on.[1] 

Below are, in my opinion, the most important proposed changes in CIP Version 5 that came out of the meeting.  I do want to emphasize that none of them are set in stone, because something could always change between now and the end of March (and of course, FERC will have the final say on what is in CIP Version 5!).  It did seem like the SDT and its leadership were generally agreed on all of the items below.

I. Inventory for Low-Impact Assets

Many commenters pointed out (and rightly so) that the first draft of Version 5 would require NERC entities to conduct an inventory of all of their cyber assets, including those that would fall into the Low-Impact classification – in spite of the SDT’s statement in the first draft that this would not be the case.  The reason for this is that one or two of the requirements for Lows in the first draft required actions to be taken on the individual cyber assets, meaning they had to be inventoried.  The commenters argued that the security gains from those two actions would be more than offset by the burden of conducting an entire inventory and then visiting every cyber asset to make the one or two required changes.

The SDT agreed that this didn’t make sense, especially since they had been saying all along that the requirements on the Lows would be almost entirely programmatic.  So it is highly likely that the next draft of V5 will have no requirements that would lead to an inventory being required for Lows.

II. Asset Identification

During the comment period, we had argued in the blog (and others had pointed out in their comments as well) that the asset identification process in the first draft of V5 was fatally flawed, mainly because it required a first step – going through each asset and identifying the BES Reliability Operating Services it supports – that was very burdensome, completely unnecessary, and unauditable.[2]  However, when I came to the meeting I was fully expecting there to be some pushback from the SDT on this assertion, since this process had been in Version 5 for at least two years and one would expect inertia alone to make the members reluctant to remove it.

I and a few others were very pleasantly surprised to find, within the first hours of the meeting, that the CIP-002 subgroup had already agreed to take this out entirely.  That is, the asset identification process will no longer start with an audit of the BES Reliability Operating Services performed by every cyber asset that touches the BES. 

As per last weeks discussions, the new proposal is that the Version 5 asset identification process will now start with the facilities themselves – the generation stations, control centers, substations – just as it did in Versions 1-4 (where the process started with identification of Critical Assets).  The entity will first go to Appendix I to see if it has any facilities that meet the bright-line criteria for High- and Medium-Impact.  If it has any such facilities, it will then identify the BES Cyber Systems “used for the reliable operation of”[3] those facilities.  These will then all inherit the classification of the facility, High or Medium.[4]  This is very similar to the process in CIP Versions 1-4, where the entity starts at the Critical Asset, then identifies the Critical Cyber Assets associated with it.[5]

Low-Impact facilities will still be identified as they were in the first draft: they are BES facilities (and Systems) that don’t meet the bright-line criteria for High- or Medium-Impact facilities.  If an entity has a Low facility, it will need to follow the requirements for Lows at that facility (whether or not it also has High or Medium facilities).  As mentioned above, the Low-Impact requirements will not require an inventory of cyber assets, or that any measures be taken on individual cyber assets.

III. DPs and LSEs

Version 5 is the first version that applies to Distribution Providers.  Versions 1-4 applied to Load Sharing Entities (although it’s doubtful many entities with just that designation had to comply with all of CIP-002 through -009); LSEs are still in scope for Version 5.

Some entities that only have DP or LSE designation had commented on the fact that they could at most have only Low-Impact BES Cyber Assets (and many would have no BES Cyber Assets at all), yet they would still have to put in place a full CIP compliance program like the big guys do.  This anomaly would be heightened because of the new improved asset identification process just discussed.[6]  There weren’t any pure DP/LSEs at the meeting this week, but their case was eloquently argued by other attendees (especially those from G&T coops, whose member coops often only have those designations).

The SDT leadership agreed that this was a great example of a big paperwork burden that would produce little reliability benefit.  Accordingly, the Applicability section of each standard (Section A, Paragraphs 4.1.2 and 4.1.6) has been modified to show that only DPs and LSEs that have facilities or systems that meet one or more of the bright-line criteria (i.e. they have High- or Medium-Impact facilities) will have to take any compliance steps under Version 5. 

IV. TO Control Centers

Another comment that appeared a lot was that Transmission Owners should not be on the hook for CIP compliance at their control centers, since it is the TOPs (Transmission Operators) that actually run them (and since TOPs are already on the hook, that means that two entities would be responsible for compliance even though only one actually runs the facility[7]).  The SDT agreed with this reasoning.  The proposed revision to the bright-line criteria for control centers will no longer apply to TOs.

V. Blackstart plants

It was argued both in the comments and at the meeting that making all blackstart plants Medium-Impact would be extremely counterproductive and would produce a negative impact on reliability.  This is because generators have the choice whether to have their plants be in a regional blackstart plan or not.  While they are paid to be in the plan, the cost is certainly well below the cost of complying with CIP Version 5 as a Medium-Impact facility (which they are per Appendix I, paragraph 2.4).  The result will be a large withdrawal of blackstart units nationwide.

While this might seem like blackmail, it is simply economics.  Generators are mostly unregulated, and therefore can’t pass compliance costs to their ratepayers as regulated utilities can do.[8]  A meeting participant from a large generation entity told me that they have scoped out the cost of implementing CIP compliance at up to $5MM for their larger facilities (of course, blackstart plants are usually not large, but even if their compliance cost is one tenth of that, it would far exceed any revenue from being in the blackstart plan).

This participant said he had talked with generators across the country and that this process is already happening.  Since generators typically have to sign up for two years as a blackstart resource, they are even now telling the RTOs that they will withdraw from the plan as of the end of their contract.  I can verify that one entity has told me they will withdraw all but two of the twelve or so plants they currently have in their regional blackstart plan.  This is not good.

The proposed solution to this problem was not to simply remove Appendix I, Paragraph 2.4.  Instead, it was proposed that blackstart plants without external routable or dialup connectivity not be Medium-Impact (they will still be Lows), on the theory that it would be much harder for them to be victim of a coordinated cyber attack on the grid.

If you’re also a NERC CIP junkie, you probably realize this is déjà vu all over again.  CIP Versions 1-4 excluded facilities lacking external connectivity from having Critical Cyber Assets (and thus from complying with almost all of CIP-003 through -009).  One of the selling points of Version 5 – to FERC and Congress anyway – is that that particular exemption has been removed.  Now it would be back for all blackstart plants that want to take advantage of it (and let’s face it, there will be some plants that rip out their current routable connections, just as happened with some substations under Versions 1-3.  To be under blackstart, a plant has to be manned when needed for blackstart anyway, so ripping out IP wouldn’t impair its ability to be in the program).  In fact, there was discussion of whether this provision should be extended to substations that were part of the blackstart cranking path as well.

When this proposal was discussed in the full session on Thursday morning (it had been worked out in the CIP-002 subgroup on Wednesday), the question also came up: Why don’t we just do this for every other bright-line criterion?  That is, why don’t we just go back to how Versions 1-4 read?  The answer, of course, is that because of the special nature of blackstart (that an owner can withdraw from the plan), it is virtually certain that there will be a negative impact on reliability if the change is not made.[9]  The same argument cannot be made for other bright-line criteria (except perhaps for substations in the blackstart cranking path, in Attachment 1, paragraph 2.5).

But even though Attachment 1 paragraph 2.4 has now been changed to reflect this provision, I don’t want to say it is certain to happen.  I believe the SDT as a whole wants this change to happen, but there are others at NERC – and ultimately FERC – who will have the final say.

VI. 1500 Megawatts

As almost everyone knows, both CIP Versions 4 and 5 have a 1500MW threshold for a power plant (not blackstart) to be mandated to be included in scope for compliance (in Version 4 as a Critical Asset, in Version 5 as a Medium-Impact facility).  It might not be quite as well known that both versions state that cyber systems that don’t have an impact on 1500MW of capacity at the plant will not be in scope (as Critical Cyber Assets in V4, Medium-Impact BES Cyber Systems in V5).

This last fact is important because it significantly reduces the “footprint” of compliance for 1500MW plants.  Very few have a single DCS that controls the entire plant.  Usually, there will be two or more DCSs, each controlling one or two units in the plant (and these will often be different models and even from different vendors, since new units are added to a plant sometimes 10-20 years after it originally opened).  Because there is no system whose compromise can bring the whole plant down, the argument for this provision is that the impact on the BES of an attack on one DCS (or other system) that controls say 5-800MW within a 1500MW+ plant is no more than that of an attack on a standalone plant of 5-800MW (which of course would not be covered by the bright-line criteria). 

The bottom line is that it is likely that there will be few medium-impact BES Cyber Systems at 1500MW plants (and of course, these are the only plants other than blackstart that would be Mediums under V5.  All others would be Lows).  Why is this a potential problem?  Because FERC (and Congress behind them) got very upset a few years ago when they found out that very few power plants were declared Critical Assets under CIP Version 1.  The bright-line criteria resulted from this.  It isn’t likely they will be too happy when they realize that, even though all 1500MW plants are technically in scope, few have any cyber assets that are (they already do realize this anyway, of course, since there are FERC and Congressional staffers who come from the industry and know its ins and outs well.  There are always two or three FERC staffers at every SDT meeting, and they heard discussions about this issue this week as well as previously).

 I believe it is likely that FERC will do something about this.[10]  I don’t think they’ll change the provision that a system has to control the entire amount of the threshold (now 1500MW) in order to be in scope.  I think it is more likely they will try to address the issue by simply lowering the 1500MW threshold.  This would also address another problem that FERC sees, which is the general lack of plants that are in scope other than blackstart (in fact, this is more likely to be the primary reason that the threshold is lowered, rather than the single-DCS issue). 

What will they lower it to?  At least to 1000MW, maybe even below that.  Don’t shoot me, I’m just saying what I think will happen.

 [1] It did seem that the other standards didn’t have anywhere near the level of “problems” that CIP-002 did, and the other subgroups also reported a lot of progress at the end of the meeting on Thursday.

[2] It would also have the same effect on Lows as the first problem would: requiring an inventory of all cyber assets.

[3] Note the phrase “used for the reliable operation of” replaces the phrase “essential to the operation of”, which was how Critical Cyber Assets were identified in CIP Versions 1-4 – those that were essential to the operation of a Critical Asset.  I believe this is a big improvement.  There is no equivalent phrase in the first draft of Version 5, since BES Cyber Assets were supposed to be identified before the facilities themselves.

[4] You may wonder whether there could be a mixture of classifications at a facility – i.e. some cyber assets would be High or Medium, while others would be Low.  The answer is no.  Any cyber asset that meets the new definition of a BES Cyber Asset will inherit the classification of its facility.  Any cyber asset that doesn’t meet the definition isn’t a Low-Impact BES Cyber Asset, it is no BES Cyber Asset at all (there were a lot of comments suggesting there be a new category of “no-impact” cyber assets.  That’s not necessary.  A cyber system that has no impact on the BES is by definition not a BES Cyber Asset at all.  15 minutes is the cutoff for determining what “impact” means).

[5] One FERC staff member at the meeting expressed the opinion – not the Commission’s! – that this was a step backwards due to its resemblance to asset identification in CIP V1-4.  The fact is, the asset identification process in the first draft of V5 would never have worked, period.  The process outlined here is workable, but it still fixes the biggest problem with CIP-002 in V1-3: that an entity pretty well could decided for itself what its critical assets were.

[6] The exact reason for this is kind of complicated, but if you want to email me I’ll let you know how I think the argument runs.

[7] Of course, there are many TO/TOPs for whom this is a moot point.  There is only one entity involved anyway.

[8] Even for regulated utilities, there is always pushback from the PUC, and it may take years for it to go through.

[9] This is a good example of what I call the Elephant in the Room for Version 5: a lot of the arguments about what should be required really come down to dollars, not what is good security.  I will expand on this idea in a subsequent blog post.

[10] And if you’re of the belief that FERC can’t change the standards once they are approved by NERC, send me an email and I’ll be glad to explain how they can.

Leave a comment
Posted in General, Power – NERC CIP. Bookmark the permalink.

Apology to Dave Norton from Tom Alrich

I am posting the below on behalf of Tom Alrich.

I regret that, in a recent post on LinkedIn, I asserted that Dave Norton of FERC had stated, at the December NERC CICP meeting in Atlanta, that FERC would approve CIP Version 4 in Q1 of 2012.  I based this on a statement to that effect that was made at a meeting of one of the NERC Regional Entities last week.

I have subsequently been told that Dave did not make that statement, and wish to sincerely apologize to him for not verifying its validity before posting it.  I have known Dave personally for a year and a half, and have found him to be a person of the highest integrity.  Since I knew that such a statement would be a violation of FERC policy, I should have realized that he would never have made it.

–Tom Alrich

Leave a comment
Posted in General. Bookmark the permalink.

NERC CIP – V5 Results and a discussion about CAN-0024

Hello all.

As many of you may already be aware (we tweeted this yesterday), the ballots for the V5 draft are in and the results are overwhelmingly negative.  In fact, every standard in the draft was voted down, with the highest approval rating (for implementation plan) at barely 40% approval.  All other standards only received around 30% approval!  Much more to be done yet!  If you are interested, there is a LinkedIn group and discussion going on about this now.  As the details emerge and we see what happens next, we will keep you posted.

In the meantime I wanted to offer a discussion from our man in the field Tom Alrich.  He and I were discussing the CAN-0024 document on the NERC website.  I mentioned this CAN in a past blog but here is a more detailed discussion by Tom and he also includes a discussion with Mark Tucker of BPA.  Below is Tom’s letter, word for word.  As always, this is Tom’s opinion and Mark Tucker is aware and in support of us posting this.  Please do provide your input as dialogue is always encouraged.

Last week, Mark Tucker of BPA asked me a straightforward question: It seemed to him that CAN-0024 (posted in its final version on December 15) requires that any cyber asset within an ESP, which communicates outside the ESP with a routable protocol, must be deemed a Critical Cyber Asset (CCA).  And this is completely independent of whether or not the cyber asset meets the primary criterion for being a CCA – that it be “essential to the operation of” a Critical Asset (per CIP-002-3 R3).

I at first pooh-poohed his concern, pointing out that the stated purpose of CAN-0024 was to close the “loophole” under which entities could claim they didn’t have CCAs if they were protected by a data diode, not to make any other changes in CIP-002-3 R3.  But when I looked at CAN-0024 more closely, I decided I had to draw the same conclusion that Mark did: As currently worded, CAN-0024 requires that any cyber asset within an ESP, which communicates outside the ESP using a routable protocol, be listed as a CCA, regardless of whether or not it is “essential to the operation of” a Critical Asset.  In other words, all cyber assets within an ESP with external routable connectivity must be declared to be CCAs.

There are three passages from CAN-0024 that make this clear.  Here are the first two:

“Therefore, under sub-requirement R3.1, the relevant criteria for determining whether non-critical Cyber Assets are to be classified as CCAs is the use of a routable protocol to communicate outside the Electronic Security Perimeter (when not at a control center).”

“Therefore, a CEA is to find a Possible Violation when a registered entity does not identify a Cyber Asset that uses a routable protocol to communicate outside the ESP, regardless of whether that routable communication is direct through a firewall, or indirect using a stand-alone data diode device.”

The third passage is from the Evidence of Compliance section:

If the Cyber Asset was deemed not to be a CCA, the CEA should verify that:

  • no routable protocols were used to communicate to Cyber Assets outside the ESP; and
  • the cyber asset is not dial-up accessible.

CIP-002-3 R3 clearly requires a two-step process for identification of a cyber asset (that isn’t in a control center) as a CCA: a) determine whether it is “essential to the operation of” a Critical Asset and b) determine whether it communicates outside the ESP using a routable protocol or is dial-up accessible.  In each of the three passages above, CAN-0024 ignores a) and makes b) the sole criterion for a CCA.

However, there is one place in which CAN-0024 does get it right.  Here is the first part of the Evidence of Compliance section (my underline):

A CEA (Compliance Enforcement Authority) is to consider the following to obtain reasonable assurance of the entity’s compliance:

• Evidence of the registered entity’s assessment as to whether a Cyber Asset not located at a control center was determined to be a CCA, based upon whether the Cyber Asset:

  • is associated with a Critical Asset,
  • is essential to the reliable operation of the Critical Asset, and
  • meets the connectivity qualification described in CIP-002 R3.1 and R3.3 (e.g., uses routable protocols to communicate to Cyber Assets outside the Electronic Security Perimeter (R3.1) or is dial-up accessible (R3.3)).

In this passage, at least, it is clear that both criteria need to be applied. 

There was of course a comment period on this CAN.  A large NERC entity has furnished me the comments they submitted, which dealt entirely with this issue.  At the time of their comments, there were three sections in CAN-0024 that had this wrong, one in Background, one in Compliance Application and one in Evidence of Compliance.  NERC does seem to have addressed the Background problem, but not the other two.

Speaking for myself, not Honeywell, I assert that the fact that CAN-0024 makes the same mistake in three places, and only gets it right in one, means it needs to be corrected (it would really need to be corrected even if it just made the mistake once).  Were I an auditor with this CAN in front of me, I would have to conclude that the wording that occurs three times is much more likely to be the intent of the drafters than the wording that occurs once.  I don’t believe that these mis-statements were deliberate, but they do seem to be the result of sloppiness.

Should this stand, what would be the implication?  Since non-CCAs within an ESP have to be protected in almost exactly the same way as CCAs, entities will at least not be applying a lot of non-required controls if they do follow CAN-0024 as now written.  However, any entity that has already identified its CCAs (which is of course almost all entities with CCAs) will be in violation of CIP-002-3 R3, unless they have explicitly designated all cyber assets in the ESP as CCAs.  In other words, they will have an incomplete list of CCAs.  This is no small matter.

Even more importantly, if such a big error can occur three times in one CAN, after it has been explicitly pointed out by at least one entity during the comment period (and corrected in a separate passage of the CAN), what does this say about the CAN development process itself?   I’d be interested in your thoughts on this.

2 Comments
Posted in General, Power – NERC CIP. Bookmark the permalink.

Welcome to 2012! A straw poll and what to look for…

Hello everyone!

Hope you all had a refreshing break (whether you took time off or most of those you know did and you were left to catch up on things!)  We certainly did but now it is back to work!  First and foremost thank you to all who have followed us so far.  We have seen a record number of people mosey on over here to see what we have posted about the recent NERC CIP topics which is of little surprise.  To that end, rest assured, we will continue to closely follow and share with you all that is NERC CIP related.  However we also know that there are many other topics of interest and as this general subject of Industrial Control System security continues to evolve we will see more of the same in additional arenas.

That is where I want to conduct a ‘straw poll’ for you all.  Either by way of commenting on this blog post or by emailing me directly (if you dont want the public attention) at Rick.Kaun@Honeywell.com  to let us know what else you might like us to cover.  We will always offer our opinions and will try to share breaking news and developments.  In the absence of time-sensitive content we also want to share ideas in emerging trends we see.  For example, I see more and more emphasis on the convergence of Industrial cyber, Industrial physical and corporate security efforts.  We have a huge proponent of this topic queued up to offer a webinar in the new year.  As always we will keep an eye on emerging regulations and guidelines like the ISA, NIST, WIB, the Pipeline sector, CFATS, Secure Roadmaps, International efforts like the CPNI in the UK, the Critical Infrastructure Resilience effort in Australia and others.

But here is your chance to either vote for a subject above or to add to this list.  We want to provide a venue for information sharing and exchange and therefore need to make sure we are delivering that which you value.  I am looking forward to the year ahead, and to hear what you all think.

Leave a comment
Posted in General, ISA, NIST, Petrochemicals – CFATS, Power – NERC CIP. Bookmark the permalink.