A Blue-Collar Approach to Assessments

If you’ve been here a while, you know how I feel about assessments and measurements. If you haven’t, you can go through the social sciences as sorcery posts.

But, simply stated, I think that our hyper-focus on assessments and our obsession with metrics hinders us.

Early this summer, I wrote about the ‘dysfunctional consequences’ of measurements (originally in the newsletter), and a friend of the blog recommended this article – A Blue-Collar Approach to Operational Analysis: A Special Operations Case Study (NDU Press).

The article discusses how the authors ditched traditional assessment methods in favor of a different approach.

That, by itself, is brave.

We say we want to be innovative, but we often roll our eyes or dismiss new ideas. Especially when it comes to assessments and measurements.

The article is worth reading in its entirety by anyone interested in the topic, but I’ve excerpted some particularly interesting points below.

On how they did it before – no surprise here, always with math.

We compounded these mistakes by quantifying and aggregating everything through a complicated system of questionable mathematical models.

The new method – ditch the MOPs and MOEs and go for RoI (Return on Investment).

In late 2015, we scrapped our existing methods and charted a new path. We stopped adhering to common practices, including the strict mechanical process rooted in MOEs and MOPs. Instead, we developed what we view as a “blue-collar business case” analysis focused on measuring and articulating SOCCENT return on investment (RoI) to resources in areas of operation (AOR).

When there isn’t a clear requirement (ie: sell more widgets), measuring returns is difficult. So how do you do it? Well, this does require a little sorcery, unfortunately.

In the financial world, the requirement is clear: apply human and physical capital to generate a profit, and measuring returns is a simple accounting drill. In organizations not driven by profit, such as the military or other public-sector entities, measuring and articulating RoI is more challenging. For example, no commonly agreed-upon method exists for measuring and comparing investments and returns between training a partner SOF unit, conducting a key leader engagement with partner special force commander, or exploiting the information environment to degrade support for violent extremist organizations.

I liked their deliverables: desired returns and actual returns.

  • Desired returns: Objectives in regional plans, or the state of the operational environment that SOCCENT expected to materialize by applying SOF resources to them.
  • Actual returns: The observable impact SOF resources—through the execution of operations, actions, and activities (OAAs)—had on objectives

On the importance of going to the source to collect data, as opposed to relying on passive collection streams. Related here, is the concept of doing things in person, as opposed to via email or over the phone.

To overcome this plight, we physically went to the source of the data. 

On resisting the urge to turn everything into data. This is important. Saying things with stories and pictures is way more effective than bludgeoning with data.

After we collected, validated, and adjudicated the data, we did not aggregate the results. We kept our sleeves rolled up and wrote nuanced, qualitative descriptions of progress and gaps at the effect and IMO levels.

And finally:

Instead of stating we were yellow on a scale of red to green, we found that a narrative focused on successes and gaps in the context of each objective was the most effective form of articulating RoI. 

A good paper. And while this is geared towards operational assessment, it has application in other domains.

But, there is still work to do.


Enjoy these posts? Enter your email below to join the monthly newsletter.

Processing…
Success! You're on the list.

One thought on “A Blue-Collar Approach to Assessments

  1. I’m not sure where measurement and assessments work and when they’ve gone too far. I feel like the PT test (at least when I was in, the AFPT) is fair enough but some junior enlisted counting the pushups didn’t call them fair. I felt like if you had an NCO count it wasn’t fair in the other direction. Are there certain measurements and assessments that bug you more than others?

    Like

Leave a Reply to lostresearchers Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s