Assessing the effectiveness of defensive cyber operations

UNCG Author/Contributor (non-UNCG co-authors, if there are any, appear on document)
Brian C. Prior (Creator)
The University of North Carolina at Greensboro (UNCG )
Web Site:
Hamid Nemati

Abstract: Enormous amounts of resources are being allocated for defensive cyber programs. The White House’s Cyber Security National Action Plan proposes a 35% increase in federal spending on cyber security during Fiscal Year 2017. Without an appropriate understanding of how well the people, processes, defenses, and risk are measured, there will naturally be unproductive tasking, inefficient spending and ineffective reporting. In 2016, the White House established the Commission on enhancing National Cybersecurity to assess the state of our nation’s cybersecurity posture. The report recognized both the difficulty and the need to develop meaningful metrics for cybersecurity in order to better secure the cyber landscape as it pertained to the broader digital ecosystem and its connection to our economy, government, and defense. The commission focused on both the private sector as well as the government and suggested the need to perfect policies, practices and technologies. Additionally, the Marine Corps University recently released research topics addressing some of the most important concerns affecting warfighters. One of the concerns was the lack of a methodology for determining the performance of Defensive Cyber Operations (DCO). Specifically addressed was a need to better understand how actions taken by network defenders facilitate network protection. Previous analysis of this topic led to a reactive and un-actionable approach which was tied to negative events such as the quantity and category of incident reports. As there is currently no framework or scorecard built to evaluate DCO as a whole effort, a methodical approach was taken to scope the problem, compare existing frameworks, develop a framework, and present a scorecard. The first phase of research required scoping exactly what is involved in DCO at the most basic level and understanding how the DoD evaluates performance. This resulted in an understanding of the actionability of metrics, the levels of warfare, and the counterbalance of cyber asymmetry. Also identified was the military doctrine for assessments, which frames evaluations in terms of Measures of Effectiveness and Measures of Performance and supports continuous assessments that provide actionable information to decision makers. The second phase required a detailed analysis of existing frameworks that measured related functions of cybersecurity. Specifically utilized were industry accepted compliance, incident handling, governance, and risk management frameworks. The outcome identified four functional areas common to most frameworks; people, processes, defenses, and risk. The third phase involved developing a framework that evaluated the four functional areas of DCO identified in the problem-framing phase, utilizing the most appropriate features of the already established frameworks. A key facet of this evaluation was that assessments should be weighed over time to demonstrate progress but also be measured against standards, peers, and the adversary. The final phase identified the continuous reporting criteria and the tangible mechanism for evaluating an organization in terms of a scorecard. The framework is not a static list of measurements but rather supports tailoring metrics to the organization’s specific requirements. The fundamentals of the framework are organized into elements, levels, categories, ends/ways, and measures. These metrics should be documented utilizing a standardized rubric that assesses the capability and performance of the metrics. The results should be reviewed and analyzed to determine trends, areas for improvement or investment and actionable information to support decision making. Additionally, a modified Delphi analysis with expert consensus validated the major concepts put forward in this paper. Overall, this research provides a comprehensive framework to evaluate the performance of Defensive Cyber Operations in terms of people, processes, defenses, and risk, filling a knowledge gap that is increasingly vital.

Additional Information

Language: English
Date: 2017
Cyber, Defensive, Effectiveness, Operations
Cyber intelligence (Computer security) $z United States

Email this document to