Autonomy: When a Recommendation Becomes an Instruction

A recommender spends consent that was never given.

People said yes once, to data collection, in a contract or a cookie wall. The steering built on that yes arrived later, and nobody asked again. This is part 6 of the data professional’s oath, and the value is autonomy, the fifth of eight, after dignity, interpersonal trust, justice and self-control. The question I want to work through here: when does a recommendation become steering that nobody consciously agreed to?

Marketing to me, I approved it

Personally, I do not really mind being advised about products, marketed to if you want. I think it can even be a service, when a company offers me something I need and it solves a problem I have. In that case I have no wish to leave the steering. It is exactly what I approved tracking for.

I have no quarrel with recommenders as such. Ranked lists, default views and “you may also need” boxes are what decision support looks like, and the people who build them are doing their job.

When the screen is the only goal

But there are cases, not known to everyone, where the steering came from an algorithm designed to pull your attention towards the screen. That was the only goal it steered towards, so whatever held attention got promoted, including media that should never have seen the light of day.

Facebook in Myanmar is the hardest case I know. The UN’s Independent International Fact-Finding Mission on Myanmar wrote in 2018 that the platform had been a useful instrument for those seeking to spread hate. Amnesty International concluded in 2022 that Meta’s engagement-based algorithms amplified anti-Rohingya content, which fed violence against the Rohingya. Nobody tapped “I agree” to that.

Regulators have started to price this line, at least for consumers. Article 5 of the EU AI Act, applicable since 2 February 2025, prohibits AI systems that use manipulative or deceptive techniques to distort behaviour materially and cause significant harm, while transparent, user-controlled personalisation remains lawful (this is a reading of Regulation (EU) 2024/1689, not legal advice). In September 2025, Amazon settled the FTC’s dark-pattern case over Prime enrollment and cancellation for 2.5 billion dollars. Both concern consumer-facing products. I am not claiming that the ranked queue inside your own tool falls under either, which leaves the line to the builder.

A brain can refuse

Above all, I do believe that human beings with a brain can decide not to be steered, even when it is tried in a sophisticated way. That defence works a lot better when the steering is out in the open. A person who can see that the list is ordered by someone’s goal can ask whose goal it is. A person who cannot see it has nothing to refuse.

The autonomy test

Here is where I land, and it is my position, so judge it as one. Steering should not happen when it is harmful, or when it serves no just purpose. It should never take away someone’s autonomy to choose for themselves. The oath’s autonomy value says support must leave the choice intact, and for me that makes the test behavioural: it looks at what the person on the receiving end can do, whatever we meant. Two questions:

1. Can the person see the steering?
2. Can they leave it at no cost?

When either answer is no, the recommendation has become an instruction, and nobody signed for instructions.

My own tracking consent passes. I know a recommender is at work, and I can switch it off. The Myanmar feed could not pass, because the people on the receiving end could neither see the amplification nor opt out of what it did to their surroundings.

Say you build a ranked call list in a CRM (a made-up example, no client involved). The order looks like plain sorting. If the agent can see why a customer sits at the top, can reorder the list without a penalty on their targets, and a change in what the ranking optimises for gets put in front of them again, the list is support. If the order is a given and deviating shows up in someone’s performance review, it has become an instruction, whatever the screen calls it. At the design table that comes down to three moves: show the ordering, keep the exit free, and ask again when the stakes change.

Where this leaves part 5

Self-control, part 5 of this oath, decided whether a thing ships at all. Autonomy decides what the shipped thing does to the person choosing with it. I am still drawing that line in my own work, so I am curious: where does a recommendation in your product become an instruction?

Happy to think this through with you.