Who Is Missing From Your Engagement Score?

Survey platforms have become genuinely good. Benchmarks, trend lines, a confidence interval on every question, a dashboard that refreshes itself overnight. All true. But every one of those numbers is calculated over the people who replied, and there is no line in the report for the ones who stopped.

I have sat in enough of these meetings to know how the next twenty minutes go. The score goes up on the screen, and the room moves straight to what to do about the people behind it. The response rate is never on that slide, and I have yet to watch anyone stop the room to ask for it. The way I see it, that missing line is the whole argument, and it is worth slowing down long enough to see why.

Why the silence leans

Statistics Netherlands has documented this for decades, in public, as a matter of method rather than opinion. Non-response in its surveys is selective, and the illustrations it publishes both run toward the people the survey is about. People who felt unsafe at home in the evening were less willing to take part in the crime victims survey. People who moved around more were harder to reach for travel behaviour research. In both cases, the group least likely to answer is the group the question was actually asked about.

From what I have seen in the field, that pattern travels well past national statistics. When what you are measuring is also the reason someone stays quiet, the silence leans, and the number gives you no way to tell which way. An engagement score is the everyday corporate version of exactly this problem: the people most frustrated, most disengaged or most on their way out are also the people least likely to open a survey link and spend ten minutes telling you about it.

The error CBS writes down

CBS weights its results to correct for under- and overrepresented groups, and states in the same public methodology that this selectivity can often not be corrected sufficiently, so the results stay distorted. Read that sentence again slowly. The most careful surveyor in the country, with statisticians whose entire job is getting this right, tells you in writing that its own correction has limits and that the residual error is real.

In the engagement surveys I have come across, nothing is weighted at all, and the score reaches the leadership slide without a response rate anywhere near it. That is not the analyst’s doing. Nobody asked them for the denominator, and nobody built a place on the slide for it. The gap is not a shortcut someone took; it is a step nobody ever added to the process.

Why the check stays green

The same gap opens wherever a measurement lives inside one digital channel. If the survey goes out by email and the reminder arrives through an app, the people who use neither are absent by construction, not by choice. They never open the message, never see the link, and never get counted as a non-response either, because as far as the system is concerned they were never asked.

The completeness check most BI shops run compares row counts against what the source system sent. It catches the rows a pipeline dropped somewhere between source and dashboard. It never catches the people the source never held in the first place, because they generate no rows at all. Green means the rows that existed arrived intact. It says nothing about who was never in a position to generate one.

This is the wider version of a problem that sits underneath most operational data quality work. Validity, freshness, duplicates, completeness against source: every one of those definitions measures rows that already exist. The person who never entered the source falls outside all of them at once, and no dashboard flags an absence it was never built to see.

Whose number this is

The analyst produced the number, and the platform vendor sold the tooling that renders it on a dashboard. I will plant a flag on the rest: the risk belongs to whoever reads the number and moves. The decision is yours, and the consequence lands on the group thin in the denominator, not on the person who built the report.

The correction costs nothing. Print the response rate next to the score, and name out loud which group is thin in it before anyone proposes what to do about the result. That single line turns a number that looks complete into a number that admits what it actually covers, and it puts the decision back where it belongs, in front of the person about to make it.

One question does most of the work here, and it costs nothing to ask in the meeting you have this week: who is missing from this number? If nobody in the room can answer that, you already know what the score is worth.

This is the second half of a question I have been circling for a while. “The empty chair” asked who the work is for. This one asks who is actually in the number you are using to decide. Happy to think this through with you.