The Weekly Query: What should data teams actually learn about AI right now?

Welcome to our first edition

Hello reader, and welcome to our first edition of The Weekly Query! We’re stoked to have you join us as we set aside some time each week to figure out WTF is going on in data.

Each week we’ll go deep on one question. (And by the way, if you’ve got a question, hit reply and let me know what’s on your mind, so we can bring it to the newsletter.) We’ll also talk about what’s new in the industry that week, a 30-second lesson, some of the more interesting jobs we’re seeing on the market, and the free resources and workshops we’ve got coming up.

This week we’re going deep on a question I know is on a lot of your minds, given how the industry is changing in response to AI: What should data teams actually learn about AI right now? How much is BS being pushed by leaders and people trying to sell you pipe dreams, and what’s actually needed in the day to day (for now at least)?

We’ll also look at what Claude Dashboards, released a couple of weeks ago, and OpenAI’s Data agent, released last month, could mean for data teams: where they might save us time, and where they could create more work. In the 30-second lesson, we’ll explain context engineering vs. context management. And of course, we’ve got jobs and skills-related goodies for you too.

So grab your coffee, tea, or whatever it is you like to imbibe while reading, and settle in for The Weekly Query.

Today’s query

What should data teams actually learn about AI right now?

Yeah, I mean, it’s a tough question, right?

For a while there, we were hearing a lot:

“Learn AI now or get left behind!”

“Tokenmaxxxxxx!!!!”

“I don’t care what it is, you’d better be doing it with AI.”

Fortunately, I think we’re starting to ease off the accelerator a bit. We’ve seen what happens when we let AI steer and keep telling it to go faster. It’s expensive and I’m sorry, but some shit is just a waste of time and money to have AI do. Some things are better automated by code because it’s cheaper and more reliable. Some things are so simple (or so complex) they shouldn’t be automated at all. And some steps really need human judgment, even when AI can handle the rest.

I mean, the larger problem I see is automating our thinking to the point we’re not doing much thinking ourselves. I’ve watched smart people offload so much of their thinking that they can’t even answer simple questions without reaching for AI. It encourages laziness.

Writing is a great example. Writing is a thinking exercise, working through our thoughts in real time so we can present them to others. If we stop writing for ourselves, we lose some thinking along the way too. I actually feel the same way about note-taking to some degree. Writing notes helps me process what I’m hearing. But man, I am addicted to the AI transcription tools.

Reminds me of Dune:

OK, this is happening, though. AI analytics is here.

It’s showing up in our companies, on our teams, and job expectations are changing. So how do we know what we actually need to learn?

Here’s the framework when thinking through how much of a given task should be human-led and AI-supported, or AI-led and human-reviewed…and, consequently, which skills to invest in.

I first ask myself two questions:

How well do I understand the work?

How consistently does AI meet the standard the task requires?

If your understanding is developing and AI needs frequent correction, invest in learning the foundations of that skill.

If AI consistently meets the mark but your own understanding is still developing, learn how to evaluate its work.

If your understanding is strong but AI keeps getting it wrong, turn your expertise into guidance and evals for AI.

And if AI consistently meets the standard and you get it well enough yourself, you can probably delegate the execution to AI.

For data teams, I’d use those questions to decide what to keep learning, where to spend less time, which new skills are worth the added effort, and which new tools or skills I can pass over for now.

The answers are going to change based on the person’s expertise, the task at hand, and the environment they’re working in.

The foundations are still critical

If you’re still learning the work and AI needs frequent correction, invest in foundational learning.

The foundational skills of being a data scientist, data analyst, analytics engineer, or a data person in general are all still critical. You still need to know SQL and Python to a point. You’re probably going to write a lot less of it yourself, but you need to know how to read it so you can evaluate AI-generated analytics code. Agentic analytics is very different from agentic coding. It’s not as easy to evaluate on the basis of output. Coding agents can inspect files, dependencies, documentation, and tests, and we can run the software to catch many mechanical failures. But analysis has to make sense of a world the data only partly captures.

A data warehouse is shaped by tracking choices, missing events, changing features, disputed definitions, and human workarounds. A query can execute while the underlying data misrepresents reality. You need enough Python and SQL to judge the logic, and enough knowledge of the business to question what the data represents.

The same goes for data visualization. You might spend less time learning the technical steps in Tableau, Power BI, Google Sheets, or whatever your platform may be, but you still need the principles behind data visualization. Does it tell a story that is accurate, useful, and easy to understand? You need that understanding to guide the system and review what it produces. You need more to go on than “well, that looks pretty.”

You also still need statistics and the fundamentals of an analytics workflow…how to think in terms of the scientific method. AI will do weird shit or start down a path that doesn’t make sense. The questions you’re asking, the assumptions you’re making, and how you interpret the result still matter a lot.

You don’t get to skip those foundations. But I do think you can spend less time learning the specific mechanics of individual tools. For instance, it took me years to get really good at Tableau. I started my data career building life-size interactive data visualizations on an “analytics wall”: an 8-foot-by-16-foot touchscreen wall of screens. It got me pretty good at D3, Tableau, Power BI, and a few SAP products. Data visualization and knowing those tools launched my career.

Today, I would still invest a lot of time in learning the principles of visualization. I probably wouldn’t invest much time in mastering the mechanics of all those specific tools. Where AI can reliably handle those steps, I don’t need to be super fast at them myself.

If you understand the work and AI reliably handles those steps, you can delegate more of that execution. Your responsibility shifts to quality control and taking action on the results.

The two other situations are where I think this gets more interesting. What if AI can do the work, but you don’t understand it well enough to judge it? Or you understand it really well, but AI keeps screwing it up?

Develop your judgment

So what do we do if AI is actually better at a task than we are? Uh oh! I do think the way we approach learning has to change in these cases. When AI consistently meets your company’s standard and you’re still learning that skill, I’d consider spending less time mastering the manual execution and more time learning how to evaluate the work.

There is a catch here, of course. If you’re still learning, how do you know AI consistently meets your company’s standard? You still need the principles. You need support from someone who has those skills and can teach you what good looks like: why certain answers hold up, where they don’t, common pitfalls, and what to check next time, even if you can’t yet carry out the whole analysis yourself.

Think of a Manager or Staff/Principal IC reviewing a more junior IC’s work. They can judge the choices, the assumptions, and what the result means because they understand the work. That’s a skill you can still develop even as you spend less time on the technical execution yourself.

Evaluating analysis goes a lot further than checking the code. Does it tell us something that matters? Does the recommendation make sense for the business or product? Wherever you’re working, whether it’s go-to-market, product, or another domain, that judgment becomes more important as we automate execution and take the work further into the business.

Lead the work. Improve the AI.

Now consider the other situation. You understand the work really well, but AI keeps getting it wrong. The useful thing to learn here is how to turn your expertise into guidance for the AI system and evaluations of its work.

This is where context management and engineering, AI evals, and building agentic systems come in. You need to work out what the system needs to know about your business and data, how that information is delivered, and how you’ll check whether it uses it correctly. How the workflow should work. Which steps can run as deterministic code? Where is reasoning needed? What should be human-led vs AI-led? What needs review? If AI keeps getting something wrong, improving the workflow might mean giving that step to code or it might mean looping in a human.

These are some of the new skills I’d spend time on, though I wouldn’t take that to mean learning every new AI tool or forcing AI into every step. The same question about investing years in tool mechanics applies to AI tools too.

What I’d learn if I were starting today

So if I were starting again today, yes, I’d learn differently. I’d keep the foundations, spend less time mastering tool mechanics where AI reliably handles the execution, and put more time into judging the work and building systems other people can use.

When I think about those years getting good at Tableau, what I’d want to carry forward is the understanding of the data, the visualization principles, and the judgment about what the result means. Those are things I’d still want to get better at regardless of AI.

That’s where I see room to be ambitious. Something you learn while evaluating one analysis can help the next person get a better answer. The context you know like the back of your hand can become something other people can use. A mistake you catch can become a test that helps you recognize the problem before someone acts on it.

I want to use the time AI can give us to understand the business better, ask questions worth answering, and help people act on what we find. We still have to think for ourselves to do that. And I think there’s a lot to look forward to if we keep developing that ability as deliberately as we learn the tools.

New in data this week

Claude Dashboards

One of the headlines that kept coming up this past week was Anthropic’s release of Claude Dashboards. I can’t begin to tell you how many messages I got with “Have you seen this?!” With OpenAI’s Data agent released last month too, both companies are making it easier for people to do their own analysis directly in their chat products. The question on my mind is what that means for the data team.

OpenAI introduced its Data agent in ChatGPT Work on September 10, directing analysis “without writing queries or learning a new analytics tool.” You can connect company data and business context, investigate a question, ask follow-ups, build a dashboard, and carry out actions you approve.

Anthropic launched Claude Dashboards in beta on October 8. Claude’s pitch is quick, exploratory questions against connected data, with a dashboard you can inspect and refine. Anthropic also describes sending the work to existing analytics tools for deeper analysis, though I haven’t tried that myself.

What this means for the data team

So I actually really like where they’re going with this. And yeah, I get why people want it. Being able to investigate a question yourself could save a lot of waiting. If the answers are reliable, correct, and trusted, the data team could spend less time on ad hoc stakeholder requests and more time on the in-depth strategic questions they didn’t have time for before. They could also use that saved time to stretch their area of influence, taking the analysis further into action in the product and business, beyond the recommendation.

The usefulness doesn’t stop at stakeholders. This could speed up exploratory analysis for the data team itself. The promise of learning less of a tool is what I was getting at with Tableau. If these systems handle the technical steps well, data teams can spend less time mastering tool mechanics.

I wouldn’t say either of these solutions is plug and play, though, and I’d be careful about how you roll them out at your company. Users still need to understand what to ask for and whether the answer makes sense. If the user can’t evaluate that, the data team can become a human review step for everyone’s uncertain analysis. Sometimes that means redoing the entire analysis while also diagnosing what the stakeholder tried on their own. In the end, this can take longer than doing the work yourself from the start. And once you open the gates to leveraging the data team purely for review, it’s very hard to shut down.

This sort of ungoverned rollout becomes dangerous to the company, as it lets a lot of analytical noise through. Previously, the data team operated as a filter on, well, I’ll just be blunt, dumb questions. Some questions are just curiosities that won’t change what anyone does. It’s easier to talk that through when the question is just an idea in someone’s head, but once there is an analysis artifact to review that request turns from “do you have bandwidth to do this analysis” to “I need you to fix this analysis AI screwed up.” The former is much easier to say “No” to when the analysis is not worth the time. If we’re not careful, we could spend a lot more time producing and checking analysis that doesn’t do a damn thing.

On that note, I really enjoyed this LinkedIn post I saw the other day in response to Claude Dashboards: give data teams a way to try it with a few people before turning it on for the whole company.

They’re both pretty cool leaps in analytics, though. I’m impressed. My wishlist is more structure around context management and evals. OpenAI says its context comes from semantic layers and sources including Databricks Genie Ontology, dbt, GitHub, Snowflake Horizon, and BI dashboards.

I’d like to see them build their own context layer into the product too, with a clear way to define, review, and maintain the knowledge the agent uses. GitHub has been one of my go-tos for this, and I really like how Hex has structured context inside its product. I want more than connections to other systems and the hope that the right context makes it through.

Before opening either product up more widely, I’d want to know which questions it handles well, what context it’s missing, and how much noise it creates for the data team. That’s the work I’d want to do before going into analytics YOLO mode.

We did a live workshop stress-testing the ChatGPT Data agent’s reliability. You can watch the recording to see the questions we tried and what happened.

The 30-second lesson: context engineering vs. context management

As we’ve been investing a lot of our time as of late around context, I thought we’d do our 30-second lesson this week on the difference between context engineering and context management…

Context engineering is figuring out what AI needs to know for a task, what form that information should take, and how to get it to the agent.

Context management is keeping that knowledge useful over time: who owns it, how updates get reviewed, and how you check whether changes improve the answers.

For example, getting the right revenue definition to an agent is engineering. Keeping it current when Finance changes the rules is management.

If you want to go deeper, we’re running Enterprise Context Management for Analytics with Hex next month. It’s $25 with code CONTEXT25. You’re welcome to join us.

Jobs this week

These skills are showing up in this week’s job postings. Harvey wants someone who can make financial definitions and accounting context usable by AI. Smartsheet wants someone to build the evals that determine whether an agent is good enough to use. Plaid is hiring for the metric layer and AI experiences that let Marketing self-serve analytics.

If you’re choosing what to learn next, I find job boards to be a good pulse check on what companies expect from their data teams. Here are eight jobs I’d recommend looking at to get a feel for what you may soon be responsible for. Think about which parts you already understand and where you’d need to build the skills to do the work.

Senior Analytics Engineer at Vanta

Remote in the US. Base salary is $190,000-$224,000 a year.

Vanta wants someone to build the semantic layers and monitoring that help AI tools work with company data.

Senior Analytics Engineer, Finance at Harvey

Hybrid in New York or San Francisco. The listed salary is $155,000-$235,000 a year.

Harvey wants someone to make financial metrics, business rules, and accounting context usable for AI-assisted reporting and analysis.

Senior Analytics Engineer at Plaid

Hybrid in San Francisco. Target base salary is $157,200-$214,800 a year.

At Plaid, you’d help build the metric layer and AI experiences that let Marketing partners answer their own analytical questions.

Sr. Data Scientist II (Remote Eligible) at Smartsheet

Remote in the US. Base salary is $155,000-$185,000 a year.

Smartsheet wants a data scientist to build evals that check whether an AI sub-agent is ready to use and catch regressions after release.

Staff Analyst, Investor Analytics at Upstart

Remote in the US. Base salary is $157,000-$217,500 a year, depending on location.

Upstart is looking for an analyst who can use tools like Claude Code and Codex to build and maintain analytics pipelines.

Principal Insights Analyst - League of Legends at Riot Games

Based in Los Angeles. Base salary is $187,700-$261,700 a year.

Riot wants an analyst to build reusable agentic analytics workflows for League of Legends that improve the speed, consistency, and reliability of the work.

Analytical Engineer at Asana

Hybrid in Warsaw. Base pay is 20,833-23,700 Polish złoty a month, before tax.

At Asana, you’d help people query governed company data in plain language through Claude and Databricks Genie.

Senior Data Scientist - AI Tooling at Fin

Hybrid in Dublin or London. The posting doesn’t list a salary.

Fin wants a data scientist to build and evaluate AI-powered internal tools for its go-to-market teams.

All eight were checked on October 19. The links above go to the full postings, including the rest of the requirements and compensation details.

Free resources and workshops

A lot of the context and eval work in those roles is what we’ve been working through in our free sessions lately. If you want to apply it to your own work, we’re continuing that this Thursday. Here’s what’s available now and coming up over the next two weeks. These workshops are free. If you can’t make it live, sign up anyway and we’ll send you the recording and materials.

Last week: How to Teach AI How Your Company Works

On Wednesday, October 14, we ran How to Teach AI How Your Company Works, Part 1, in partnership with Hex. We looked at how your existing reports, docs, and code already hold a lot of the definitions and business rules an AI analyst needs. You probably have more of that knowledge written down than you think. We talked about setting up processes to find it, review it, and make it available to the rest of your company.

Watch Part 1 and read the transcript.

This Thursday: Part 2 with Hex

Thursday, October 22, 10 a.m. Pacific: Teach AI How Your Company Works, Part 2.

We’ll review the context management principles from last week and work with you to start planning a real agentic analysis at your company. What does AI need to know to answer the question? What knowledge already exists? What’s missing, and where would you start? This will be a hands-on session where you can start putting structure behind the context these analytics tools need to work reliably.

Next week: interviews and AI mistakes

If you’re interviewing for roles like the ones above, Sravya Madipalli is teaching Ace Behavioral Interviews with AI on Wednesday, October 28, at 10 a.m. Pacific. You’ll work on turning your real experience into clear stories about what you did and what happened, then use AI to practice how you tell them.

And on Friday, October 30, at 10 a.m. Pacific, Rachel Herrera from Hex joins us for How to Fix the Mistakes Your AI Analyst Keeps Making. If you’re correcting the same mistakes in every chat, we’ll look at the missing rules and context, use Hex’s context-consultant skill to draft better guidance, and test whether the changes improve the answers.

You can also browse the past free workshops and free resource library.

AI Analytics for Everyone

We’re super excited to have kicked off our fourth cohort of AI Analytics for Everyone this week! Just yesterday, actually. If you’re interested in learning the foundational judgment skills across the analytics workflow we talked about above, that’s exactly what we’re working on over in this course. You’ve only missed one day (which was recorded), and you can still join through Sunday, October 25th.

Over the next five weeks, you’ll work through the kinds of decisions that come up every day in product and business: choosing useful metrics, figuring out why they changed, evaluating experiments, deciding what to do next, and getting shit done.

You’ll use AI to help carry out the analysis. You, the human, will practice asking a clear question, prioritizing the work that matters, evaluating the answers, and knowing when the evidence is strong enough to act. No SQL or Python experience needed to join.

You can catch up with the session recordings in your own time. We record the office hours too, and you get unlimited retakes in case things come up along the way.

Take $500 off with code MONDAY500 if you join this week. Join the cohort.

Meet us next week

We’ve also got a couple of chances to talk about all this in person next week.

The whole AI Analyst Lab team will be at Hex’s Prompt conference on Tuesday, October 27, at the San Francisco Mint, 88 5th St. I think it may be fully booked, but you can join the waitlist. Hit us up if you’re joining. We’d love to catch up.

I’ll also be speaking at ODSC on Thursday, October 29, from 3:45 to 4:15 p.m. My session is Why AI Analytics is Harder to Crack than Coding, and How Teams are Solving it. We’ll get into how the technical steps can run correctly while the analysis answers the wrong business question, relies on assumptions that don’t fit, or leads to a bad decision. And what teams are doing to solve those problems.

See the ODSC schedule, and come join my session or catch up with me while we’re there.

Send us your query

Alright, that’s our first Weekly Query. Thanks for spending some of your day with us.

Hit reply and tell me what you’re trying to learn, where you’re getting stuck, or what you’d like us to put to the test. You might just get your answer in our next issue ;)

See you soon,

Shane