The Weekly Query: Why is AI analytics harder than AI coding?
What's inside: building an AI analyst you can trust, AI analytics headlines, capability and reliability, eight data jobs, and upcoming workshops.
This week's query
Agentic Analytics is NOT Agentic Coding.
AI analytics has been around for years. But now it’s getting baked into the products people are already using day to day, like ChatGPT and Claude Chat. People outside the data team are connecting AI to their company’s data and asking questions.
And while I’m very excited about more people getting access to data to make decisions and get shit done, there’s some risk when they start treating agentic analytics in the way they’ve become accustomed to using agentic coding for software development.
These two things are not the same.
I think it’s pretty amazing what you can build with agentic coding now. Suddenly, even if you've never coded in your life, you can make something for your own use that mostly works for some specific thing you want to do.
Say you’re trying to lose weight and you want an app where you log your meals, see how you're doing against your goal weight. You can use Claude Code or Codex to explain the product experience you want, have AI build it, and start using it. If something doesn't work the way you expected, you can give it feedback and it’ll act on improving the experience.
Now I don’t believe this means coding is solved. You can have bugs you haven't observed, use cases you haven't thought of, or huge security gaps like credentials stored where they can easily leak. If you’re not a developer, you probably have no clue what's happening behind the scenes. If you wanted to scale your vibe-coded app to other people, well, that brings a whole bunch of other things your own use might never have exposed.
Agentic analytics is a different beast. Yes, it often uses code (SQL or Python) to accomplish part of the analytical task, but the workflow and goal are quite different.
With coding, a user is trying to have a specific in-product experience, get some job done, or be entertained, etc.
With analytics, someone is trying to make a decision. They need information to reduce their uncertainty so they can decide whether to act and what that action should be.
The hard part is knowing whether the analysis actually helped them get there. Regardless of AI, answers to analytics questions can be wrong in ways the person asking the question has no idea how to check.
In last week's Weekly Query, I talked about the judgment and company knowledge data people still need as AI handles more execution. Today we dive into WHY that's so important and how it actually shows up in your work.
What does a metric even mean?
Think about a question like, "What does engagement look like in our product?" Maybe you ask AI to take a look at sessions.
There are a bunch of caveats and business decisions wrapped up in even something as simple as what counts as a “session.”
There could be sessions from internal engineers testing the product we need to filter out. Or sessions from bots and crawlers. Folks who visited the login page but never actually logged in and used the product. People who logged in for a second or maybe 5 seconds, didn't do anything and left. People who logged in but their browser just sat there open while they went off to have lunch.
Maybe you care about people spending some time in the product actually doing stuff. Maybe, for a social product, then sessions must consist of someone having posted, commented, reacted to some piece of content, or talked to someone else.
What does engagement even mean for this product, and what are we trying to decide about it?
Say we know the answers to all those questions, and then we look at the data warehouse and find a bunch of different tables with session information. Which one should we use? What does this weird column sess_rdv_flag mean? How was this field created? What does one row represent? Do these even represent the behavior we care about?
A good data person knows it's part of their job to investigate all this stuff. They might talk to the relevant prod dev team about where an event originates from, or ask in the analytics engineering channel what each session table means or who would know, or maybe they open up GitHub and examine the data models and pipelines themselves. Then check with a real human to find out if they've interpreted them correctly.
These people know the company. They learn what people mean when they say "sessions" or "active users." They know the weird-ass decisions behind these definitions because they were in the meetings where those decisions were made. Maybe some had really important reasoning behind them. Maybe some were choices people made because they needed to align on some definition and move along.
Figuring out and aligning on metric definitions was already hard before AI. We’ve all been in meetings or threads where dashboards and decks supposedly measuring the same thing showed different numbers.
With software development and agentic coding, we’re constructing a behavior in a system we control.
Data and analytics are a representation of part of what's happening in the business or product. The choices about what we record, what we exclude, and how we define things affect what we can learn from it. Getting the code to run is just one part of that work.
The whole workflow has places to fail
With analytics, a lot of the work is trying to remove bias so our data and methods represent the real world as accurately as possible.
It’s very easy to screw this up because there are so many opportunities for failure across the analytics workflow.
When I review an analysis, I want to go back to the very beginning, the goal. What are we trying to do? What decision do we have to make, and what questions would help us act? Lots of data folks get this wrong. Lots of stakeholders get this wrong too. They are vague and don't really know what they want. We need to frame those goals into decision-driven questions.
From there, we need hypotheses we can test and metrics that actually measure what we mean, including the population, filters, and time periods. We need to find the right data and translate those definitions into queries. We need a method that can answer the question, and we need to check that the data satisfies its assumptions. Then we need code that does the analysis correctly. And once it’s done, is our interpretation actually backed up by the results?
When we share it, we need to ask ourselves: does the meaning come through as intended? Could someone misinterpret how we visualized or explained it? And does that answer actually inform the decision we started with?
Anything that fails upstream can cause failures downstream. If we count the wrong sessions, it doesn't matter if we chose the best method or interpreted the results correctly. It’s incorrect.
Human analysts screw this up all the time. But doing the work takes time, which adds some friction to check it along the way. They've also been burned in the past and learned from their mistakes, so know where to be detail-oriented in those checks. They go through reviews with their team or work with peers to confirm things.
AI can run through this whole workflow in minutes. And it will make a lot of assumptions if it doesn't have explicit direction and context as well as a eval loop available to keep it on track.
When agentic analytics scales to enterprise, a business user with no data background may not have any way to tell if the answer is correct. They get an answer, don’t know if it's right, so they bring it to the data team: "Hey, is this correct?" Now the data team has to redo the entire analysis while also tracing what happened in the conversation. That’s a lot of extra work.
Or even worse, the user just assumes the answer is correct, and acts on it. Maybe they even share the results with someone else who then shares it with another person, and it gets lost that it once originated from AI. It goes through a game of telephone and ends up informing what a team works on or how the company allocates resources, while the underlying workflow could have failed at many points and no one checked. What a mess.
How do we already solve for this with humans?
I like to think of an AI analyst as a solid analyst or data scientist who just joined your company.
They could be a top performer at every company they’ve ever worked at, have years and years of experience, but when they first join, what they don't know is you. They don't know your business, your product, or the teams that make them work. And they certainly don’t know what's in all of your heads and all the history behind the decisions that put that knowledge there.
Imagine two analysts are practically twins…equally good at SQL, Python, statistics, communication, and getting shit done, but one has been at the company for five years while the other is on day one.
There's no contest in who will be more effective. The new person might not even have data access. While the tenured analyst knows where to look, what to avoid, and who can explain the thing that doesn’t make any sense. They hold all the company context.
So when I’m approaching agentic analytics, I approach it with the same expectations and requirements I’d have with a new joiner to the company. I want to help them build up that company knowledge. Basically I want to onboard ai to my team.
For the session tables, I need to proactively explain which ones we endorse and what they're useful for. Which ones to avoid and why. And which ones they can kinda use if you understand their caveats. I need to explain the differences, the edge cases, and who to ask if something is unclear.
Maybe this already exists in a data catalog or an onboarding guide, but a lot of it is probably still in people's heads across the teams. The new team member would set up time with them to learn about it, but for AI… I need a way to make it available to the system and keep it current.
Then I need transparency into whether it actually used that information and used it correctly. For sessions, that means: what table did it query? What filters did it apply? An does that actually line up with the definition we agreed on? (Just because I gave it the definition doesn’t mean it followed it.)
Someone has to keep this stuff up to date
Building this context and history adds work to the job, maybe even a new role. I mean, all this should have been done before but maybe it never got staffed. Companies that weren't doing this before are going to have to start now. And that's going to require a real layer of governance and maintenance.
Say you already have metric definitions somewhere AI can access. Maybe that's an existing data platform like Hex or Snowflake or Tableau or dbt, or maybe a version-controlled GitHub repo with files organized by product area or business line. In there you have what the metric means, when to use it, how it's calculated, any exclusions or caveats, who owns it, and who has verified it. For a rate, that includes the numerator and denominator.
Even once you get that set up the first time, the job never ends. What happens when a new metric is introduced or, more importantly, when someone changes that metric on their team?
We need to save those changes wherever the team keeps its metric knowledge. We need single points of ownership, even if multiple people help maintain it.
And this has to be a proactive process to work well.
I've worked with analytics engineers who would put a long list of SQL links in Slack, tag a bunch of people, and ask, "Hey all, please confirm if these are all correct? Can you tell me by end of week?"
I've got a bunch of other shit to work on. I'm probably not going to stop and review all that code because someone tagged me. That process doesn't give me much confidence that anyone really checked it and is acting like a true owner.
If I'm responsible for a metric and we change it in a meeting or through some analysis, updating its definition in the context layer needs to be part of my job. I can update the repo myself or work with the person who maintains it. But whoever owns that knowledge needs to be connected to the team making those decisions.
We talked about this in our context-planning workshop this week. Hai made the point that the owner doesn’t need to be someone on the data team, it can be someone in the business. Ideally, those people use the AI analyst themselves, so maintaining the definition helps them get better answers too.
Start with a question you already know well
If you’re looking to get started with agentic analytics, I'd recommend starting with a recurring analysis you've done enough times to understand the work and recognize its mistakes. Like something that's come up at least 3 times before. You already know some of the questions, definitions, and caveats and you have work you can review and compare against.
You can use that experience to work out what the AI needs to know, what evidence you need to log, and where a human needs to make the judgment vs. review the judgment of AI. When you find mistakes, create feedback loops and ownership around bringing back what you learned to the system so that work can help the next analysis.
Think about that analyst who's been at the company for five years. A lot of their value comes from conversations, decisions, and mistakes they've learned from over time. I personally have learned far more from my mistakes than my successes. We have the opportunity to make more of that knowledge useful to other people now.
It’s the same job the data team always had: help people make better decisions by asking the right questions and then reducing their uncertainty around the answer so they can do something useful with it.
New in data this week
Hex tests Meta’s Muse Wonder on DataBench
October 21. Hex shares results from its agentic analytics benchmark, including examples of the model working through difficult data questions.
Zoom announces a Unified Context Layer for its AI tools
October 20. Zoom’s Unified Context Layer connects business conversations with documents and data so people and agents can use the history behind the work.
Databricks publishes its MemCurrent agent-memory research
October 20. Databricks’ MemCurrent research describes how agents can keep records of past work and turn them into reusable knowledge, including definitions and workflows.
Snowflake previews controls for the MCP tools AI can use
October 19. Snowflake’s Cortex AI Gateway preview lets admins choose connected servers, limit exposed tools, and review tool-call activity.
Meta is acquiring AI analytics platform Sundial
October 23. Axios reports Meta is acquiring Sundial, which will stop serving external customers when the deal closes.
The 30-second lesson
Evaluating your AI analyst
When I’m evaluating an AI analyst, there are two things I want to know:
Capability: Can it actually get the correct answer?
Reliability: Does it do the work consistently across repeated runs?
Try the same question five or ten times in separate conversations, using the same data and context. Check the answers against one you know is correct.
If it can get the right answer once but can’t do it reliably, it’s not really usable. If it consistently gives you the wrong number, that’s not useful either. If it’s both capable and reliable, you’re in a good position to start using it for that question.
This is key when you start scaling your AI analyst to the rest of the company. Most people aren’t going to ask five times to see what happens. They’re going to ask once and take whatever answer they get. You want a high probability that it’s the correct one.
Jobs this week
The work we’ve been talking about is showing up in job descriptions. Block wants someone to build and evaluate the pricing intelligence behind Square’s DealBot. Replit wants an enterprise data layer people can query through AI agents. Zapier wants to turn repeated analytics requests into governed self-service tools and AI-assisted workflows.
The SQL, data modeling and business judgment are still there. Companies also want people who can make that knowledge usable by AI, build workflows around it, and evaluate the results. Here are eight worth a look.
Staff Data Scientist, Pricing at Block
Bay Area, California; open to remote work. Zone A listed salary is US$198,000 to US$297,000; other US locations have separate ranges.
Block is hiring someone to build pricing models for Square’s DealBot, then test and monitor the AI for accuracy, edge cases, bias and mispricing.
Senior Business Analyst, Revenue Analytics at Reddit
Remote in the US, within commuting distance of NYC, Chicago, SF or LA. Base salary is US$167,500 to US$234,500.
Reddit wants someone to build semantic models and business logic for AI analytics while owning revenue data pipelines and reporting.
Data Scientist, Enterprise at Replit
Hybrid in Foster City, California; office days Monday, Wednesday and Friday. Base salary is US$210,000 to US$327,000 a year, plus equity.
Replit’s data scientist will build the enterprise data layer that lets teams self-serve through AI agents, alongside experimentation, forecasting and shared business metrics.
Manager, Internal Product Analytics at Datadog
Hybrid in New York City. Listed salary is US$160,000 to US$200,000 a year.
Datadog wants a manager to lead a team building agents that query, synthesize and report analysis, while owning the metrics and definitions Product teams use.
Data Science, Finance & Strategy at Anthropic
San Francisco; office attendance required. Listed salary is US$270,000 to US$345,000 a year.
Anthropic wants someone to own executive business metrics and build governed definitions and Claude-powered workflows for recurring analysis.
Manager, GTM Analytics at Zapier
Remote, North America, per the current board listing. US base salary is US$174,200 to US$261,300 a year; Canadian compensation is listed separately.
Zapier wants a manager to lead analysts turning recurring GTM questions into governed self-service tools and AI-assisted workflows, with QA and shared metric definitions.
Analytics Engineer II, Full Stack (Revenue Analytics) at Affirm
Remote in Poland. Listed base pay is PLN 226,000 to PLN 346,000; the posting does not specify the pay period.
Affirm’s analytics engineer will build semantic, metadata and context layers that help AI understand revenue metrics, alongside dbt models, reporting and data governance.
Data Scientist, Agent at Lovable
Office-based in Stockholm or London. The posting does not list a salary.
Lovable is hiring a data scientist to build continuous evals and experiments for its software-building agent, using traces and telemetry to catch regressions and guide fixes.
What's happening at AI Analyst Lab
Last week
Last week we got into the practical side of what we’ve been talking about here: what does AI actually need to know to answer a question at your company? In Build Your AI Context Plan, we worked through a drop in iOS sessions, figured out which sources and definitions mattered, and put together instructions and checks for the analysis.
You can watch the recording on YouTube and use the free Context Plan Builder to work through a question from your own job.
This week
Ace Behavioral Interviews with AI
Wednesday, October 28 · 10 to 11 a.m. Pacific · Free
Interviewing right now? Join Sravya and the team to turn your actual experience into stories you can use in an interview, then use AI to practice and get feedback on how you deliver them.
Building Context for Agents with Skills
Friday, October 30 · 10 to 11 a.m. Pacific · Free
Rachel Herrera from Hex joins us to show how their Context Consultant skill helps an agent build and improve context, and how to test those changes before your team relies on them.
And a reminder: I’ll be at Hex Prompt today, Tuesday, October 27, at the San Francisco Mint, and speaking at ODSC on Thursday, October 29. My session is Why AI Analytics is Harder to Crack than Coding, and How Teams are Solving it, from 3:45 to 4:15 p.m. Pacific. If you’re at either, hit me up. I’d love to say hi.
Next week
Check if your data is ready for AI
Wednesday, November 4 · 10 to 11 a.m. Pacific · Free
Join Hai to work out whether your data is ready for AI. You’ll take one question your company asks often, look at the tables it needs, and figure out what to fix first.
Agentic Analytics: Build an AI Analyst
Starts Monday, November 2 · Paid course
Our flagship five-week course takes you through building an agentic analytics system end to end: designing and building the system, in-depth context engineering, AI evals for analytics, working across Claude, Codex and open-source models, and starting to roll it out at your company. You’ll build the system and learn how to check whether you can actually trust its work.
Coming up
Enterprise Context Management for Analytics
Saturday, November 14 · 7 to 11 a.m. Pacific
We’re partnering with Hex for a hands-on workshop on building, managing and evaluating the context your AI analyst uses. You’ll work with supplied data and leave with a plan for applying it at your company. It’s $25 with promo code CONTEXT25.
Enterprise AI Analytics for Data Leaders
Saturday, December 12 · 8 to 11 a.m. Pacific · Half-day workshop
Hai and I are running a half-day workshop for the people responsible for rolling this out across a company. We’ll get into who owns the definitions, how you evaluate the tools you already have, and how to build a rollout plan with clear milestones.
Anyway, I’d love to hear what this looks like on your team. Are people already asking AI questions and bringing you the answers to check? Hit reply and let me know what you’re dealing with.
Shane
10+ years in product data science, causal inference and AI evaluation at Stripe, Nextdoor and Ontra.