What People Still Own When the Analysis Is Automated
The easy answer is the final decision. That is not enough. Four things a person has to own all the way through the work.
What do people own as more data science and analytics gets automated? We’ve been thinking about this at the lab. The easy answer is that people own the final decision. I don’t think that’s enough.
By the time an analysis reaches a final decision, dozens of smaller decisions have already shaped it. What question gets asked. How the metrics are defined. Which statistical methods get used. What counts as enough evidence. What a result means in the context of the company.
Even when those upstream decisions are bad, the query still runs, the chart still looks compelling, and the story around it still sounds reasonable. The work can be technically correct while the company does something worse than it would have done without it.
So I think people need to own four things throughout the work.
The direction
An automated system can turn a request into every reasonable question and approach you can think of. We still need to decide what someone is actually trying to change and which question is worth answering. Doing the rest of the work faster does not help if we start in the wrong place. We’ll just automate our way down the wrong path.
The standard
Automate the queries, the calculations, the code, the tests, and the charts. We still need to decide what the metric means, which method fits the question, and how much evidence is enough. You will probably find multiple definitions of the same metric and nobody who wants to pick one. That debate, and the alignment that comes out of it, is part of the analysis too.
The judgment
An automated system can look across more segments, time periods, edge cases, and explanations than you could ever explore yourself. A lot of what comes back will look interesting while meaning absolutely nothing. We decide what is plausible, what the evidence actually supports, and what deserves a closer look.
What happens next
A system can draft the story. It can adapt it for different people. It can monitor what happened and flag what changed. But a person still has to own the decision, take the action, and decide what the team should learn from the result. Otherwise the analysis is just more output.
Owning the work is not doing every step
A good manager does not do every task on their team. They are still responsible for whether the team is working on the right things and whether the result is good enough. I think data work is moving in the same direction.
You still need technical depth. You can’t set the standard for work you don’t understand, and you can’t evaluate a query you can’t read. But I’ll spend less time proving I can personally execute every step and more time learning how to direct the work, evaluate it, and make something happen from it.
Automate as much of the work as we can reliably check. Then get very good at owning what the work is for.
10+ years in product data science, causal inference and AI evaluation at Stripe, Nextdoor and Ontra.