Highlights
A bank's own security operations manager admitted to softening phishing tests to hit target numbers.
Security teams grade themselves: the same people who build the test are the ones scored on its results.
One credit union has years of failure data it cannot access, locked inside a report built for a different question.
The real fix is scoring exposure severity, not raw clicks, and separating test design from test grading.
The scoreboard you write yourself
Every quarter, security awareness leaders walk into a board meeting with a phishing click rate on a slide, presented as a clean number: a proxy for how well the organization resists social engineering, trending the right direction, ready for applause. It is treated as a scoreboard. This week, several conversations made clear it is closer to a mirror, and the person holding it up is also the one being graded on what it shows.
A bank's security operations manager put it more bluntly than any consultant deck ever would: you're crafting emails based on the level of maturity in your organization, which means you're fudging the numbers. That is not a confession of laziness. It is the predictable output of asking the same team to design the test and pass the test, quarter after quarter, in front of a committee that only sees the final score.
Goodhart's law wearing a headset
There is an old, useful idea in measurement: when a number becomes the target, it stops being a good measure of the thing it was meant to track. Phishing click rate has quietly drifted into exactly that orbit. The test settles into a shape that produces a defensible score, not a shape that resembles the threat it is supposed to simulate. Templates get gentler. Difficulty gets tuned down. Nobody decides this in a meeting; it happens one reasonable-sounding adjustment at a time, the same way a satellite drifts off course through a thousand tiny, unlogged corrections.
The opposite failure showed up a few calls later, from a credit union's IT security lead, describing a system that isn't gamed so much as sealed shut: the data is there, the data is available, it's just not exposed. Their platform tracks failures but won't let them customize something as basic as a rolling window, so years of signal sit unused behind a report built for a different question. Gamed or simply locked away, the number leadership sees is rarely the number that describes real risk. Both failures end at the same place: a board slide that reassures instead of informs.
What would count as a real signal
The same bank's team pointed toward the actual fix without naming it as one. They wanted a platform that distinguishes a click from a credential submission, because, in their words, there should be a distinction, since at least the person who only clicked recognized something was wrong before going further. That is a severity-weighted signal, not a binary pass or fail, and it cannot be hand-tuned by the team being graded on it, because the difficulty and the scoring live outside their reach.
The fix here is not a stricter test or a longer report. It is separating who designs the simulation from who is measured by its results, and scoring exposure rather than clicks, so that a near miss and a compromised credential are never the same line on the same slide. Until that separation exists, every phishing metric on every board deck is, to some degree, a self-portrait.
Key Takeways
Separate who designs your phishing simulations from who is evaluated on the resulting score.
Score severity, not just clicks: a click that stops short of credential entry is a different signal than a full compromise.
Push your vendor for raw, exportable data on custom time windows, not just pre-built quarterly reports.
Ask your own team, honestly, whether this quarter's templates got easier than last quarter's, and why.
