The Weekly Query: What is the new job of a data leader?
What's inside: the new job of a data leader, Hex PROMPT takeaways, a 30-second lesson, leadership jobs, and upcoming workshops.
This week's query
What is the New Job of a Data Leader
By Hai Guan
Hint: it's self-serve AI analytics
What is the job of a data leader now? The directors of data science, the heads of analytics, the heads of data. What are companies actually looking for when they hire one?
We read 25 data leader job postings from September and October. These are roles running analytics, data science, and analytics engineering teams at places like GitLab, Snowflake, Robinhood, DoorDash and Pinterest.
What stood out is how many of these want you to build self-serve AI Analytics at the enterprise level (we call this Enterprise AI Analytics), where non-data people inside the company can ask data questions to AI and get answers back. 14 of the 25 put AI in the job itself, and 12 want you to build self-serve. 8 of those 12 tie self-serve directly to AI.
Here's an example from Crusoe for their Senior Director, Analytics & Data Science role.
Read the highlighted part. They want the business answering recurring questions on its own, I imagine to shorten the time from question to decision.
A few more:
This is fast emerging, and as more companies catch on to the theoretical promise and read more blog posts like Anthropic's and DoorDash's, this is going to show up in more and more data leader job descriptions. Anthropic says about 95% of its business analytics questions now go through Claude. DoorDash took its data agent from passing 43% of its tests to 90%.
12 months ago, this requirement barely existed. 8 months ago, when AI got good enough to do analytics, knowing how to implement enterprise wide AI Analytics put you at the frontier. Now it's the baseline.
And self-service isn't enough. The answers have to be right, which is still the hardest part no matter how smart LLMs get. Shane got into why in last week's Weekly Query, and we wrote up the ten ways an AI analyst breaks that we keep running into. 8 of the 25 postings ask you to test the AI's answers, and I expect that number to climb quickly over the next few months.
In this Weekly Query, I'll break down what this all means for data leaders, and what the job becomes once everyone wants AI analytics.
What is "self-serve" AI Analytics?
Here’s Google’s answer.
In other words, someone in marketing or finance types a question like "why did signups drop last week?" and gets a correct answer back, without filing a ticket with the data team.
If you think about it, this has always been the job. The form factor changed. The substance didn't.
Data teams have always been (or perceived to be) a bottleneck for business questions. The business wants to make better decisions with data, and the best data teams want that too. Wouldn't it be great if business users could answer their own questions, especially ones that have been asked before?
Before AI, the best solution was to build dashboards that answer recurring and related questions. Or perhaps teaching business stakeholders SQL and Excel so they could pull their own data. But by and large, it was a promise that never materialized.
Then came agents and capable AI models. How confidently they reason through data and ambiguity captured people's imagination, especially executives’.
But honestly, even with frontier models, most business users still bring most of their questions to the data team. They don't trust the numbers, or can't verify them. More likely than not, it all comes back to the data team as extra work.
Implementing a successful AI Analytics program takes a different skillset than the job did a year ago. If you're further from how AI in analytics works under the hood, it's worth getting up to speed.
What does it actually take?
Anyone can spin up a coding agent to build the pipeline, or write the context docs, or put things in an markdown file. How does a data leader bring together all the different capabilities across data and non-data teams into something centralized and governed, and what exactly is their role?
In my opinion, a data leader's job is to steer the company on the benefits and limitations of enterprise AI analytics. That means working with executives, clearly communicating (perhaps over and over and over again) with cross-functional partners around expectations, and defining the end state: what the data team is responsible for, what other stakeholder teams are responsible for, and what the sequences and milestones are, so the company trusts there's a well thought-out plan.
This isn't the only way. If you've found something that works better, please drop a comment. I'd love to learn from it.
How can a data leader succeed at this?
With everything being so new, a data leader needs to understand the details. Or at least conceptually. Context and evals get thrown around a lot, but it’s extremely important you know how they actually work.
The components helpful to know:
- Data foundation, including metric definitions. The tables, the models, and the agreed definition of every number that matters. If finance and marketing each have their own definition of "active customer," the AI will happily pick one.
- Context. Everything the AI needs to know about your business that may or may not be in the data itself. What each table and field means, which metric to use for which question, and the gotchas your senior analysts carry in their heads. We went through how to build this in Build the context that makes AI data agents reliable.
- Evals. A set of questions where you already know the right answer, run against the AI every time something changes, so you know whether it's getting better or worse. More on this in AI evals for Claude Code analytics output.
- And knowing how moving one drives the others. A new metric definition changes the context, and it should come with new eval questions. A failed eval usually points back to missing context or a broken definition.
As an aside, we go deep on context and evals in our 5-week course, Agentic Analytics: Build an AI Analyst.
Knowing the key ingredients buys you the “what”. Working with the right people and laying the bricks gets you the how. I think about the how in five parts. I call it SERPP: strategy, expectations, roadmap, people, and process.
- Strategy. Does everyone understand what the end result is? We always push for an explicit strategy (could be a one-pager or a short doc): why you're doing this, which part of the business goes first, who's involved, what you'll fund and what you need, how you'll know it's working, and what you won't do yet. To get more granular: What eval framework does it need? What data assets does the team have, and where should the AI Analytics agent live?
- Expectations. Early on, the answers will likely be wrong more often than they’re right. How do you bring this up with executives to buy yourself time, and set the right expectations on timeline, scope, and what it can and cannot do? This step is delicate because one failure can set the whole program back (or kill it altogether).
- Roadmap. Which business domain do you "open" this up to first? Is the data foundation sound for AI? If not, what's missing and how can the organization get there? (This is what our free lightning lesson on Nov 4 is about: Check if your data is ready for AI.)
- People. Who can help contribute to the overall initiative? How can you enlist owners of each business area to own the definitions and context for their part of the business?
- Process. How do you spin up committees, for example, to align on metric definitions, settle disagreements, and approve changes before they reach the AI? And when someone spots a wrong answer, where do they flag it, who turns the fix into a new eval, how does it get incorporated so the system compounds? Where should the context live and how should they be maintained and updated. What questions go into the eval sets?
The biggest failure mode
In my experience, the worst outcome is when the data team’s job becomes "can you check this analysis I got from [insert LLM of choice] for me?"
At Hex's PROMPT conference last week, a common complaint, besides the growing barrage of data questions, was "hey, I got this HTML from Claude, can you validate it for me?" Some of this is inevitable as more people get AI tools before the data foundation is ready. How well you steer the organization decides how many of these requests are expected of you (and thus the data team's quality of life).
I really like the 2x2 decision framework Mariam Melikadze, VP of Data Analytics at The New York Times, presented at PROMPT. She sorts questions by two things: how much is the likely impact of the decision, and how hard is it to reverse.
- Lower impact and reversible: support self-service with guardrails.
- Higher impact but reversible: speed up self-service, with an expert signing off.
- Higher impact and hard to reverse: expertise and human judgment come first.
- Lower impact and hard to reverse: maybe don't do it at all.
What I like about it is that it gives the data team a reason to say yes to most of the volume, and a clear reason to slow down on the rest.
One of my strongly held beliefs is that a data leader, and in turn the data organization, really gets one shot at implementing a successful enterprise AI Analytics program (unless there's a change of regime, of course), before either the business loses trust completely or the culture makes the data team permanently a QA department. And we all want to be on the right side of that.
In a future issue, I'll go deeper on what makes up each part of SERPP work.
If you're working through this right now, Shane and I are running a 3-hour live workshop in December on exactly this: Enterprise AI Analytics for Data Leaders. Before that, we're hosting two upcoming free lightning lessons: Check if your data is ready for AI (Nov 4) and Decide which data questions AI answers (Nov 11). We hope you’ll join us.
New in data this week
By Hai Guan
Shane and I attended Hex's 1-day conference last Tuesday.
A common theme throughout is that the key to success for a data person has not changed: truth-seeking, outcome-oriented, trusted relationships with the business. These mattered before AI, and they matter even more now. AI is now the accelerator of work, and the general thesis is that context and control will dictate whether enterprise AI Analytics is successful at any company.
Some notable takeaways from talks across the day:
- Meghana Reddy and Nova Wang shared how the data team at Stubhub went through their AI transformation's existential crisis, but soon realized the bread and butter never changed. The role starts as context curator and grows into builder and manager of agentic systems, with acting as a systems thinker being critical throughout.
- Mariam Melikadze showed that more analysis does not necessarily lead to better decisions, and she shared a decision framework that guides how her team at NYTimes think about where AI fits: start from the decision, and don't over-engineer the system.
- Izzy Miller walked through Databench, an AI benchmark for data work on top of a synthetic company built iteratively until an LLM couldn't tell it apart from a real one. Super interesting to hear that models do way worse on decision questions ("should I fire my sales team?") than on lookup questions ("what's this number?"), and more thinking models tend to do worse.
- Andy Cotgreave discussed something many have felt, that faster is not always better when it comes to data work and the evolution of tasks that's underway.
- Olivia Koshy and Izzy showed off some cool features in Hex 4.0, including a self-learning context graph that seems to be able to curate context every night, debunk bad entries, and surface contested things like unclear metric definitions to their human owners.
Other News
- Oracle announced a Business Semantic Layer where agents can pull approved definitions straight from the database without guessing. Feels like every vendor is converging towards the importance of metric definitions.
- Jedify launched Data Agents. Teams can describe a task in natural language and get an AI analyst that runs through their metric definitions and can be called from Claude, ChatGPT, or Slack.
- Databricks added a guided setup for Genie Agents. Building the agent keeps getting easier. It’s still the data team's job to know what context to give it.
The 30-second lesson
Explaining complexity in your work
By Hai Guan
If you ever struggle with highlighting complexity in the work you do - whether it’s for a promotion case, scoping out a project, or describing why something takes so long - try to describe using this categorization framework.
- Technical complexity - Why the work is just hard (e.g. data could be messy, formula could be complicated, model could be sophisticated, checking the AI system is nontrivial, etc).
- Organizational complexity - Why navigating through teams and personnels is challenging (e.g. perhaps it’s a bureaucratic company you’re a part of that’s resistant to change).
- Ambiguity complexity - Why the work is ill-defined and what you had to do to narrow down the problem.
If you can specify why the task you dealt with belongs to one of these three categories, chances are the person you’re talking to will understand too.
Jobs this week
Hai's question is showing up in the work companies want their data leaders to own. GitLab wants a director responsible for semantic models and evals. Snowflake wants a senior director to own its internal analytics agents. Robinhood wants its team operating through agentic workflows.
The definitions, context, access, and checks all need owners. These eight roles give you a pretty concrete look at what that responsibility can involve.
Director, Analytics Engineering at GitLab
Remote, US. Listed US base salary: US$139,200–$285,600; location affects the offer.
GitLab wants its director to own governed metrics, the semantic model, AI access through Claude and Slack agents, and the evals that measure answer quality.
Senior Director, Analytics Engineering, Data Analytics & AI at Snowflake
Menlo Park, California. Listed salary: US$292,000–$383,250 a year; bonus and equity are additional.
Snowflake wants its senior director to own internal analytics agents, including their context and semantic layers, evals, orchestration instructions, and ground-truth datasets.
Head of AI/Data Science, Finance & Strategy at Robinhood
Menlo Park, California; in office five days a week. Listed Zone 1 base pay: US$247,000–$290,000.
Robinhood wants its head of AI/data science to make agentic workflows the team’s default, with reusable skills, context files, MCP integrations, evaluation harnesses, and guardrails.
Senior Director, Analytics & Data Science at Crusoe
San Francisco. Listed compensation: US$310,000–$355,000, plus bonus; RSUs are also offered.
Crusoe wants its senior director to build the analytics team, own certified data marts and metric definitions, and prepare the context its data MCP uses for self-service.
Manager II, Engineering- Analytics Engineering, Core Data Science at Pinterest
US; office collaboration one or two times a month. Listed US base salary: US$177,185–$364,795; location affects the offer.
Pinterest wants its analytics engineering manager to build AI analysis tools and pipelines, expand self-service investigations, and verify AI-assisted work through testing, source checks, and data validation.
Director, Analytics Engineering at Oscar Health
New York City; hybrid, three office days a week. Listed base salary: US$198,720–$260,820 a year.
Oscar Health wants its director to lead analytics engineering across teams, establish ownership and definitions for AI-ready data, and improve self-service adoption and reliability.
Senior Manager, CS Data & Analytics at Toast
Boston; hybrid, two office days a week. Listed pay range: US$145,000–$232,000.
Toast wants its senior manager to lead two analysts, expand self-service through AI and documented data, and turn customer experience questions into useful analysis.
Strategy & Analytics Director at Faire
San Francisco or New York City; hybrid, three office days a week. Listed San Francisco base salary: US$237,000–$325,500 a year.
Faire wants its fulfillment analytics director to own metrics and experimentation and use AI to automate analysis and build reusable workflows. This is a senior individual contributor role with cross-functional leadership.
What's happening at AI Analyst Lab
Last week
Ace Behavioral Interviews with AI
If you're interviewing, this one's for you. We worked through how to build a database of your own experience, lead answers with the result, and use AI to practice and catch weak spots in your delivery. You can watch the recording at the original Maven lesson link above.
Building Context for Agents with Skills
On October 30, Rachel Herrera from Hex joined us to talk about using skills to find missing context, build and maintain guidance for your agents, and test changes before your team uses them. You can watch the recording at the original Maven lesson link above.
This week
Getting AI analytics to work for yourself is one thing. Opening it up to the rest of the company brings a whole set of questions. Is the data ready? What should AI answer? What does the data team still need to own?
That's what we're working through in a new series on scaling AI analytics across your company. The first three sessions are free. Then we've got two half-day workshops for people who want to spend more time building the context and the plan for their company.
Check if your data is ready for AI
Wednesday, November 4 · 10 to 11 a.m. Pacific · Hai and Shane · Free
Take one question your company asks often, check the tables it depends on, and figure out what needs fixing before AI can answer it reliably.
Next week
The New Data Job: What AI Does, What You Own
Tuesday, November 10 · 10 to 11 a.m. Pacific · Shane · Free
Work through what AI can answer, what needs review, and what data people should lead, so your team's job doesn't become checking everyone else's AI output.
Decide which data questions AI answers
Wednesday, November 11 · 10 to 11 a.m. Pacific · Hai and Shane · Free
Sort questions by what a wrong answer could cost and whether the person asking can catch it. Decide what to open up first and what needs more work.
Coming up
Enterprise Context Management for Analytics
Saturday, November 14 · 7 to 11 a.m. Pacific · Shane, Hai, and Rachel Herrera from Hex · US$25 with CONTEXT25
We'll work in Hex on the definitions, trusted sources, and business guidance an analytics agent needs, then use evals to check whether the context improved its answers. You'll leave with the context you built and a 90-day plan for applying it at work. Use CONTEXT25 at checkout.
Enterprise AI Analytics for Data Leaders
Saturday, December 12 · 8 to 11 a.m. Pacific · Hai and Shane · US$380
This is where we take the organizational questions from Hai's article and work through them for your company: who owns the definitions, how changes get approved, what to promise executives, and where to start. You'll begin a 90/180-day plan in the workshop, with priorities, owners, and milestones.
Join us for the free lessons, the workshops, or whichever parts are useful for the work you're doing. Each session has its own registration link above.
That’s it for this week. If you’re working through this at your company, hit reply and tell us where you’re getting stuck.
See you next Tuesday,
Shane