You can’t hit if you don’t swing

The Mad Scientist Laboratory recently had a good episode with author Zach Schonbrun to discuss his work researching cognition and performance.

It’s great when military podcasts speak with military folk – but it’s refreshing when they step away and speak with the rest of the world.

Since that’s the world we live in.

Two things struck me in this episode:

“What does it mean to say that he’s skilled? What does that actually mean? The sports industry has not really grappled with this question because it involves very difficult assessments beyond just how fast an athlete runs or how high he jumps. Those are the metrics that they tend to focus on because they’re easily measureable [but] I don’t think that tells you very much about what athlete they’re going to become.”

Zach Schonbrun, 414. It’s All In Your Head

“Those are the metrics that they tend to focus on because they’re easily measurable…”

Over and over again, we’re hearing this. We have a problem with metrics. There are dysfunctional consequences of relying on metrics.

And the answer isn’t simply “we need better ways to measure” or “big data and AI will save us.”

There are tangible things that are worth measuring, but there are also intangible things that we’re not paying attention to. And just because we can’t measure them, doesn’t mean they’re not there.

And #2. How do hitters know when to swing?

They’re using prediction. They’re picking up on very subtle cues, that take years and years of practice and expertise, and that has told them this is what they should be expecting in this situation.

Zach Schonbrun, 414. It’s All In Your Head

Prediction versus analysis. Does the hitter have to “prove” that they know when to swing? That they’ll get a hit?

Sometimes (most times) they miss. But each swing is a rep.

We expect batters to miss. It’s part of the game. What would be the effect on a batter if they received a steep penalty for missing?

Think about where that might be happening in other organizations.


Enjoy these posts? Enter your email below to join the monthly newsletter.

Processing…
Success! You're on the list.

A Blue-Collar Approach to Assessments

green goo dripping

If you’ve been here a while, you know how I feel about assessments and measurements. If you haven’t, you can go through the social sciences as sorcery posts.

But, simply stated, I think that our hyper-focus on assessments and our obsession with metrics hinders us.

Early this summer, I wrote about the ‘dysfunctional consequences’ of measurements (originally in the newsletter), and a friend of the blog recommended this article – A Blue-Collar Approach to Operational Analysis: A Special Operations Case Study (NDU Press).

The article discusses how the authors ditched traditional assessment methods in favor of a different approach.

That, by itself, is brave.

We say we want to be innovative, but we often roll our eyes or dismiss new ideas. Especially when it comes to assessments and measurements.

The article is worth reading in its entirety by anyone interested in the topic, but I’ve excerpted some particularly interesting points below.

On how they did it before – no surprise here, always with math.

We compounded these mistakes by quantifying and aggregating everything through a complicated system of questionable mathematical models.

The new method – ditch the MOPs and MOEs and go for RoI (Return on Investment).

In late 2015, we scrapped our existing methods and charted a new path. We stopped adhering to common practices, including the strict mechanical process rooted in MOEs and MOPs. Instead, we developed what we view as a “blue-collar business case” analysis focused on measuring and articulating SOCCENT return on investment (RoI) to resources in areas of operation (AOR).

When there isn’t a clear requirement (ie: sell more widgets), measuring returns is difficult. So how do you do it? Well, this does require a little sorcery, unfortunately.

In the financial world, the requirement is clear: apply human and physical capital to generate a profit, and measuring returns is a simple accounting drill. In organizations not driven by profit, such as the military or other public-sector entities, measuring and articulating RoI is more challenging. For example, no commonly agreed-upon method exists for measuring and comparing investments and returns between training a partner SOF unit, conducting a key leader engagement with partner special force commander, or exploiting the information environment to degrade support for violent extremist organizations.

I liked their deliverables: desired returns and actual returns.

  • Desired returns: Objectives in regional plans, or the state of the operational environment that SOCCENT expected to materialize by applying SOF resources to them.
  • Actual returns: The observable impact SOF resources—through the execution of operations, actions, and activities (OAAs)—had on objectives

On the importance of going to the source to collect data, as opposed to relying on passive collection streams. Related here, is the concept of doing things in person, as opposed to via email or over the phone.

To overcome this plight, we physically went to the source of the data. 

On resisting the urge to turn everything into data. This is important. Saying things with stories and pictures is way more effective than bludgeoning with data.

After we collected, validated, and adjudicated the data, we did not aggregate the results. We kept our sleeves rolled up and wrote nuanced, qualitative descriptions of progress and gaps at the effect and IMO levels.

And finally:

Instead of stating we were yellow on a scale of red to green, we found that a narrative focused on successes and gaps in the context of each objective was the most effective form of articulating RoI. 

A good paper. And while this is geared towards operational assessment, it has application in other domains.

But, there is still work to do.


Enjoy these posts? Enter your email below to join the monthly newsletter.

Processing…
Success! You're on the list.