Vanity metrics vs impact metrics: how to tell them apart.
A vanity metric can only go up and changes no decision. An impact metric can disappoint you. Five levels of evidence, one test for each number, and what to report instead.
A false balance is an abomination to the LORD, but a just weight is his delight.Proverbs 11:1
A vanity metric is a number that makes the work look bigger without telling you what to do next. Followers, page views, impressions, total sign-ups ever, “people reached”: each one can only grow, each one feels like progress, and none of them can tell you whether the thing you built is useful to anyone. The problem is not that these numbers are fake. They are usually accurate. The problem is that they answer a question nobody needed answered.
An impact metric is the opposite in one decisive way: it can disappoint you. It is tied to a specific promise, measured over a specific window, and capable of coming back lower than you hoped. That possibility is what makes it worth reporting. A number that cannot go down cannot correct you.
This guide gives you a practical way to sort any metric you already track, the five levels of evidence that sit between attention and real change, and the sentence to publish when the honest answer is still small. It uses the same proof ladder SIX33 runs on — and SIX33's own published count is zero verified lives impacted, so none of this depends on having a success story first.
1. Run the one-question test on every number you report.
For each metric on your dashboard, ask: if this number moved sharply in either direction next month, what would we do differently? If the honest answer is “nothing — we'd keep doing the same thing and feel better or worse about it,” it is a vanity metric for you, whatever it is called. If the answer names a decision — change the offer, fix onboarding, stop a channel, double down on one audience — it is doing real work.
The same number can pass for one team and fail for another. Page views are vanity for a ministry measuring discipleship and genuinely actionable for a publisher testing headlines. The test is not about the metric's name. It is about whether it is wired to a decision.
List every number you report publicly or to a board.
Beside each, write the decision it would change. Leave it blank if there is none.
Anything left blank stays internal — or gets reframed honestly as reach.
2. Separate the five levels instead of adding them together.
Most inflated impact claims come from adding numbers that measure different things. A useful way to stop that is to keep five levels on separate lines: reach (who could have encountered the work), delivery (who received the complete promise), use (who meaningfully used it), return (who came back, replied, or referred), and change (who has credible evidence that the intended difference happened).
Each level is real and each one matters. Only the last one supports the word impact. A campaign that reached 50,000 people, delivered to 900, was used by 300, brought back 40, and produced 6 evidenced changes is a far more convincing report than “we impacted 50,000 lives” — because a reader can see exactly where the funnel narrows and judge the result for themselves.
Reach — impressions, views, visitors. Distribution, never impact.
Delivery — orders shipped, sessions held, tools received complete.
Use — completions, active users, people who did the thing.
Return — repeat use, replies, referrals, second purchases.
Change — a defined outcome with a stated method and limitations.
3. Recognise the common vanity metrics — and what to track instead.
A handful of numbers show up as vanity metrics again and again. In each case there is usually a nearby number that measures the same activity at a level that can actually inform a decision.
Total followers → people who acted on a specific post this month.
Page views → readers who reached the end, or took the next step the page offered.
Total sign-ups (all time) → sign-ups in a window who completed the first meaningful action.
Downloads → people who used the download more than once.
“People reached” → people who received the complete promise, then people who came back.
Hours volunteered or money raised → what those inputs produced, reported as outputs, not outcomes.
4. Do not let outputs pose as outcomes.
Outputs are what you did: workshops run, meals served, products shipped, articles published. Outcomes are what changed for someone because of it. Outputs are worth reporting — they prove execution — but they answer a different question. Ten workshops is an output. Seven participants who report, at a 30-day follow-up, that they now do the thing the workshop taught is an outcome, with a method attached.
The quiet failure mode is gradual: a report that starts by listing outputs slowly begins describing them in outcome language. “Served 2,000 meals” becomes “fed 2,000 families” becomes “fought hunger for 2,000 families.” Each step sounds harmless. Together they claim something nobody measured.
5. Measure change with a method you can name.
Evidence of change does not require a research budget. It requires deciding in advance what change you expect, choosing a way to observe it, and being honest about what that method can and cannot prove. A short follow-up question asked to everyone served is a method. Observed behaviour — they came back and did the next step unprompted — is a method. Independent verification is a stronger method. Self-report collected only from your happiest customers is not a method; it is a testimonial.
Whatever you choose, publish it beside the number, along with the reporting window, who was asked, how many responded, and what the result cannot establish. A modest number with a clear method is more persuasive than a large number nobody can check.
Write the intended change in one sentence before you count anything.
Ask everyone in the cohort, not only the people likely to say yes.
Record non-responses and negative results alongside positive ones.
State the window: 7 days, 30 days, one season.
6. Report the small true number on purpose.
The strongest defence against vanity metrics is a habit: publish the most specific true sentence the evidence supports, even when it is small, and date it. “Between 1 and 31 August, 412 people visited, 38 completed the course, 11 returned for the second module, and 4 reported the change we were aiming for in a 30-day follow-up” will survive any scrutiny. It also tells you exactly where to work next.
If the honest answer is zero, say zero. SIX33 publishes zero verified lives impacted with a date because a definition written while the count is zero cannot quietly bend later to absorb a flattering metric. That is the whole point of choosing impact metrics: they keep the story tied to what is actually happening.
The short versionIf a number cannot disappoint you, it cannot guide you. Keep reach, delivery, use, return and change on separate lines, attach a method to the last one, and publish the small true number with its date.
Direct answers
Questions about vanity metrics and impact.
What is a vanity metric?
A vanity metric is a number that makes work look successful without informing any decision — typically cumulative totals like followers, page views, impressions, or all-time sign-ups. It usually only grows, so it cannot tell you when something is not working.
What is the difference between vanity metrics and actionable metrics?
An actionable metric is tied to a specific decision and can move in either direction in a way that would change what you do. A vanity metric would not change your next step whether it rose or fell. The same number can be vanity for one team and actionable for another.
Are followers and page views always vanity metrics?
Not always, but they measure reach, not impact. They are useful for distribution decisions such as which channel or headline works. They become vanity metrics when reported as evidence that people were helped or changed.
What should I track instead of vanity metrics?
Track five separate levels: reach, complete delivery, meaningful use, return behaviour, and evidenced change. Report each with its window and method, and reserve the word impact for the last one.
How do small organisations measure impact without a research budget?
Define the intended change in one sentence in advance, ask everyone served a short follow-up question at a fixed interval, record non-responses, and publish the result with its method and limitations. Observed return behaviour is also strong, cheap evidence.
