A Blue-Collar Approach to Assessments

If you’ve been here a while, you know how I feel about assessments and measurements. If you haven’t, you can go through the social sciences as sorcery posts.

But, simply stated, I think that our hyper-focus on assessments and our obsession with metrics hinders us.

Early this summer, I wrote about the ‘dysfunctional consequences’ of measurements (originally in the newsletter), and a friend of the blog recommended this article – A Blue-Collar Approach to Operational Analysis: A Special Operations Case Study (NDU Press).

The article discusses how the authors ditched traditional assessment methods in favor of a different approach.

That, by itself, is brave.

We say we want to be innovative, but we often roll our eyes or dismiss new ideas. Especially when it comes to assessments and measurements.

The article is worth reading in its entirety by anyone interested in the topic, but I’ve excerpted some particularly interesting points below.

On how they did it before – no surprise here, always with math.

We compounded these mistakes by quantifying and aggregating everything through a complicated system of questionable mathematical models.

The new method – ditch the MOPs and MOEs and go for RoI (Return on Investment).

In late 2015, we scrapped our existing methods and charted a new path. We stopped adhering to common practices, including the strict mechanical process rooted in MOEs and MOPs. Instead, we developed what we view as a “blue-collar business case” analysis focused on measuring and articulating SOCCENT return on investment (RoI) to resources in areas of operation (AOR).

When there isn’t a clear requirement (ie: sell more widgets), measuring returns is difficult. So how do you do it? Well, this does require a little sorcery, unfortunately.

In the financial world, the requirement is clear: apply human and physical capital to generate a profit, and measuring returns is a simple accounting drill. In organizations not driven by profit, such as the military or other public-sector entities, measuring and articulating RoI is more challenging. For example, no commonly agreed-upon method exists for measuring and comparing investments and returns between training a partner SOF unit, conducting a key leader engagement with partner special force commander, or exploiting the information environment to degrade support for violent extremist organizations.

I liked their deliverables: desired returns and actual returns.

• Desired returns: Objectives in regional plans, or the state of the operational environment that SOCCENT expected to materialize by applying SOF resources to them.
• Actual returns: The observable impact SOF resources—through the execution of operations, actions, and activities (OAAs)—had on objectives

On the importance of going to the source to collect data, as opposed to relying on passive collection streams. Related here, is the concept of doing things in person, as opposed to via email or over the phone.

To overcome this plight, we physically went to the source of the data. 

On resisting the urge to turn everything into data. This is important. Saying things with stories and pictures is way more effective than bludgeoning with data.

After we collected, validated, and adjudicated the data, we did not aggregate the results. We kept our sleeves rolled up and wrote nuanced, qualitative descriptions of progress and gaps at the effect and IMO levels.

And finally:

Instead of stating we were yellow on a scale of red to green, we found that a narrative focused on successes and gaps in the context of each objective was the most effective form of articulating RoI. 

A good paper. And while this is geared towards operational assessment, it has application in other domains.

But, there is still work to do.

Enjoy the posts? Subscribe to the monthly newsletter.