Hai Guan: Cool, alright. Well, welcome, everyone. We’re gonna do a little round of intro here, for Shane and myself, and then we’ll get right into today’s topic, which is about Claude Code 101, Grounding Claude in your business data. So, yeah, my name is Hai. I currently work at a legal tech company. I have roughly 20 years of experience in the data science and analytics space, and have previously worked at companies like LinkedIn, Pinterest, Nextdoor, mostly consumer tech companies. A co-founder in AI Analyst Lab, where we almost do these weekly lightning lessons.
We… Offer, sort of like, free lessons to, general public around Agentic analytics and AI, and we also have other paid courses, that we, that we go much deeper into some of these topics as well. Shane, over to you.
Shane Butler: Yeah, hey everyone, I’m Shane, also the co-founder here of the AI Analyst Lab, been in data science for a little over 10 years, across B2B SaaS and consumer tech. Yeah, I worked at Nextdoor with Hai. Worked at Stripe, worked at a legal tech company with Hai most recently, but I… Just run, working on the lab full-time these days. What else? Yeah, last couple years, been focused, less on, kind of, classical product analytics and product data science, and more around AI evals and Agentic analytics, so… obviously… Been interesting to see how the, The role of data professionals has kind of been changing, evolving in the past couple years.
Hai Guan: Yeah, and we have a third colleague who is not able to join us today. Her name is Sravya, but you guys, if you have attended our previous sessions, you might have seen her run some of these, and she’s gonna run one of these in the next couple weeks as well. All right, cool. So we’ll get right into it. Today’s topic, again, is on how do we ground Claude in your business data, making sure that it doesn’t guess, or I guess, kind of, like, try to minimize that to the extent possible.
LLMs are all non-deterministic, so there’s a… there’s a There’s a component of randomness, in all the outputs that it generate, but we’re gonna try to see how we can, get it to be as, as, You know, like, as attuned to our own data as possible. We are gonna spend maybe roughly, like, 15-20 minutes on slides, just the concept of some of these things that would be helpful, and then we’ll get into a little bit of a pretty simple live demo in Claude Code Desktop. It’s gonna be… It’s gonna be a pretty simple setup, and I’ll send out kind of, like, the practice there to folks afterwards as well, and then leave plenty of time, or the rest of the time, for any questions. Cool.
All right, so just a very quick poll for the folks here. Where, if you want to drop the numbers here, one that, that resonates with you, where does your metric definition live today in your company, in your work? Drop a 1 if it’s in someone’s head. A 2 for somewhere in a doc, or a wiki. A 3, maybe in code, dbt, or you have a dedicated semantic layer, or number 4, which is, somewhere where the AI already has context.
Shane Butler: Or maybe they live in a slot machine, and every time you do an analysis, you just pull that lever, and it’s a little different each time.
Hai Guan: I like Attendee’s response, nowhere, okay.
Shane Butler: Yep.
Hai Guan: Should’ve… should’ve included that option.
Shane Butler: Other than that one dude’s head, who was here for a few years, then he just quit last week.
Hai Guan: Tom’s head, great, awesome.
Shane Butler: Tom said.
Hai Guan: Just as specific as, as, as, nothing.
Shane Butler: Keep Tom happy.
Hai Guan: 1, 2, 3, 4. Wow. Nice. It’s everywhere. machine. Okay, cool. Awesome.
Shane Butler: A lot of twos, a lot of twos. Yeah, Docker, Wiki, nobody has updated in months.
Hai Guan: Nice. Cool. Yeah, this is… I mean, you know, like, I think, probably none of these are surprising, right? I think previously, even before the world of AI, we struggled a lot in trying to get things in our head, down on paper, or stored somewhere and keep it Up-to-date, and all that kind of stuff. And I think AI, since it made analysis, since it’s made data exploration so much faster and so much more accessible, now it’s even more important than ever to make sure it understands your business instead of it trying to Trying to… trying to, guess its way through what your business means.
So, we’re gonna talk about some of the major components of those, and And, and some of, kind of, like, what are most helpful in getting it to behave the way that we expect. And, you know, on this… on that point, I think, most people really underestimate how much a model is guessing when it answers a question about their data. you know, like, generally, it knows, because it’s trained on such massive amount of data in how these LLM models are trained. It knows what a customer is, in general, right? But it has no idea what a customer is at your own company, and so… Typically what happens is it would just pick Right?
For example, if it has access to your data warehouse, to a Google Sheet, or some CSV files, it would just guess, or it would just pick, not guess. It would pick. It would pick a column, perhaps it would pick what a week means, if you’re pretty vague in your questions to it. Or if, let’s say, there’s a dip in June in your own business, in your company. whether the dip in June was real. And then it will answer, like, it’s really sure that, you know, all of these things are correct, or that these things are accurate. Back in June, I think June or July, I forgot which one, Shane ran a whole session around asking AI the same question 5 times, and we get 5 different numbers.
And I think even when we ask humans multiple times across the room, we get very different answers. And so, I think that’s kind of, like, the… the highlight of some of these, just… almost like symptoms of what’s happening when the underlying, underlying kind of, like, definitions and data identifications, those are not set. So I’ll drop it in the chat here for those who are interested in, in… re-watching that, lightning lesson. And generally, what… how people sort of, like, solve for it is, perhaps in their prompts, they might add in more context, or fix it in the prompts, basically.
you know, like, that… that’s… that’s a fair state… that’s a fair practice of, you know, like, trying to correct it on the spot. But it only works once, and then next session, it’s gone. And so… Really, like, the prompt is never a place for it, and what we’re… what we’re gonna do today is to look at from a very simple example perspective, where these things can live, and how does that change, the answers and the consistency in the answers themselves from AI. Cool, alright. So, for any analytics projects, right, the way that I think about Claude Code is a smart new hire, for example. Like, I think that’s a really helpful analogy to, to draw.
And a couple months ago, we did a Claude Code 101 Lightning lesson on that idea. So, what skills are, what agents are, what hooks are, that is kind of like an introductory, lesson around, how Claude Code works, and how that fits into, sort of, like, the, the, the workflow, and what it’s capable of. So I’ll drop that in the chat as well, if folks wanted to… Check that out. today is gonna be the part that is almost like cross-cutting, or maybe even before all of that, right? Like, you know, Claudeco has skills, has agents, we have different ways to half plot.
work for us, but that if it doesn’t have the, kind of, like, the context and stuff, that’s almost like a new hire without an onboarding doc. So, you know, like, just… just imagine if, today, the new hire starts, and it’s day one, it’s 10 o’clock in the morning, you ask the new hire how many new customers we got last week. they’re probably, like, what are they gonna do, right? Like, you know, assuming there’s no, like, onboarding sessions or anything. They’re probably gonna make their best guess, or perhaps, probably most likely, is they’re gonna ask you what a customer is. Like, how do you define it? What does that even mean in this company?
And, you know, it would be… it wouldn’t be… Very wise to ask a new hire that question, like, how many customers do we have? before we even onboard that person around, you know, what all these things are. And so, this is kind of like that equivalent onboarding analogy for, for Claude, or for AI in general, and typically involves a few things. Or, I guess, four things here. Number one is what the data is, right? That’s pretty intuitive. Like, you have to explain to it, what each table is and all that kind of stuff.
what the metrics mean, like, you know, if we… if we talk about this, this, this term here, exactly what does that, you know, like, like, semantically, what does that, what does that mean? The third is how we report things. So maybe at your company, you have different ways of, of, of, you know, like, looking at time windows or, different filterings or parameters and stuff like that. So there’s, like, quirks and specific things that’s, that’s really, unique to a company. For example. And then finally, what I call, what you would only know from the hallway.
So, you know, things that’s happened in the past that, that gen… that these are even more so living in people’s head of, oh, 2 years ago, this might have happened, or two months ago, this happened. And, you know, those would be really good to, to also, to also have, such that, we’re not sort of, like, guessing based on some anomalies from the data itself. We’ll walk through each one of these pretty… In a little bit more detail, and then, And then right after that, we’ll, get into, sort of, like, the, the demo. All right, cool. So, the first one, pretty intuitive, hopefully, what the data is.
So, you know, like, what… what does each table or file mean in… in the… in the case of, in the case of the project that you’re working on? So the thing that I’m gonna show you guys later is gonna be a pretty simple setup of a coffee subscription service called Brulee, and in it, it only has two CSV files. One is around metrics, the other one is ad spend, for example. And so, it’s, you know, like, super intuitive, really easy, but then we still need to under… to have, kind of, like, AI understand exactly what these are, right? Like, hey, what is metrics.csv? What is adspend.csv? What does it entail? What does it contain? And then the second one is… could be, like, what one… what is one row?
Maybe it’s one row per day per channel, so different grains that would have to… that we… we would have to sort of, like, let… teach AI that, this is the shape of the datasets. And then the third is what are the columns, right? Like, what each value, what the column values mean? Like, what are these fields? What do they mean? We have, in that example, we’re gonna have a column that is called Activations. So we would have to describe what that actually means, right? Activations feels or sounds pretty… non-specific. It could mean a billion different ways, depending on how you think about it.
Similarly, let’s say, like, you know, in your work, you might have come across a term like retention, which itself can have many different meanings to it, so on and so forth. And then the second big bucket component here is what the metrics mean. So… Sort of, like, on the left side here, we have a, you know, bad definition. If, let’s say you wanted to define what an active user is, a really bad definition is probably not going to be it would not be something like a user who is active, right? Like, none of that is specific. Like, active could mean, again, a billion different things. So, that’s not useful. A good one. would… could be something on the right, for example.
Hey, in the coffee subscription business, really, a new customer is actually an activation, which means The day the first paid box, so assume that, it’s a subscription service that, that mails out, coffees on a regular basis, the first… the first paid box is delivered, that is when a a new customer is defined, and that’s what an activation is. For example, and maybe, in addition, we have a sign-ups column, which is never… which is not what we mean when we say a new customer. So, all of these are pretty nuanced, right? But they’re really important to the business, and very specific to your business.
And so… One of the ways to think about it is if, let’s say, like, you have to do the exercise, and a lot of you who answered, it is in someone’s head, for metrics definition, or it lives in a wiki that’s outdated somewhere. If you have to sit down and, like, update all of these metrics definitions, have a file for it, you know, like, if you have to write it down, and if you cannot write it down in plain, simple words. Then, most likely, your team has not really agreed on the definition, the specific definition of that metric.
And I’ve seen that all the time nowadays, because everyone wants access to, access to data, but then, you know, last week, found out, for example, or two weeks ago, found out that, there’s four different ways to define one single metric across departments, and so that’s not great.
Shane Butler: One thing I really like about this, Hai is, like. it’s not just about us, like, knowing how to, like, take our metrics and have it so AI can access it reliably. It’s just such a good practice and forcing function for us to go revisit. Are our metrics any good, like you said? There’s the reliability part of, like, do that mean the same thing? Are we all thinking the same thing in our head? Probably not. But it’s also, like. hey, does this actually reflect the business value, or the user value that is set out, or the goal that we’re actually here at this company to achieve? Or is it some kind of vanity thing that doesn’t really move?
Or have we been gaming it to, like, make ourselves look better, and this metric’s just trying there to, like, make this team look good? So it’s just such a good, like. forcing function to, like, go through that process, which is often a process that’s just, like, kicked down the road again and again and again. So, like, we talked about this, we ran a 5-week course where we build out an Agentic analytic system, and to end, and on Monday we were talking to our students in that cohort that’s going on right now.
And we’re saying, like, if… if the robots take over, if Judgment Day comes and Skynet goes, omniscient or whatever, self-aware, and we gotta… and we gotta unplug the robots, like, we fight the battle, we fight the human vs. robot war, right? We win. And then… it’s all good, we gotta unplug them, but we still have all this great, data foundations and metric definitions we at least lined up. So, you know, AI is not doing the analysis for us, but at least we have aligned on the metrics, so it’s a really good thing that you should… we should be doing for decades, that… It’s so easy to skip over.
Hai Guan: Yeah, totally. And my guess is that that exercise is now going to be very well funded, for different companies, whereas previously, it’s always been, hey, what is the ROI for this? Like, no, like, just go patch it, or whatever else. But what we’re… what we’re seeing and what we’re hearing is that this becomes more and more important as people understand that if you build AI and Agentic systems on top of a shaky foundation. Even previously, it’s not great. Now, it’s even worse, because if it hallucinates, if it makes up stuff, if it guesses the wrong things, you’re gonna make bad decisions off of it. Cool, alright.
And then the third component here is, actually, why don’t I make this full screen? The third component is, this is, like, for example, how we report numbers. You know, for example, what is a week? Perhaps it’s Monday to Sunday, that’s always the case for your business. Perhaps it’s always trailing 7 days, and and you look at it on a given weekday or something. So there’s… there’s there’s… there’s nuances there. Perhaps what’s last week? What does that mean? Perhaps you have a way to compare it, always, and it’s… it’s different across different companies, across different departments, and maybe even preferences, like how you round a number, right? Like, you know, one decimal or whatever it is.
Each of these decisions is if you have specific requirements, for example, And, and you don’t specify them, then the model would just make them for you. It would just be, you know, hey, like, it would be, I don’t know, like, 5 decimal places. It’s probably not gonna be… not gonna do that. But, like, you can… you can… you can, you can imagine the… the extremes there, and certainly from run to run. There is no guarantee if there’s… if it’s not written down how you… how you do things. Cool. And then, touched upon this a little bit earlier, which is… The fourth component, it is what you would only know from the hallway, right? I mean, not quite hallway, but, like, you get the… you get the idea.
So, for example, like, in the example that we’re gonna see, in the In the demo, there’s gonna be what looks like a data issue from June 30th to July 1st, where the paid campaign for the for the, coffee subscription business was turned off, was on pause, actually, because the card, the credit card on file expired, for example, like, just all fictional stuff. So this is a spend outage, right? Ad spend problem, not a demand problem.
And so, you know, like, those would maybe, you know, like, they shouldn’t happen all the time, but when they do, it would be a really good idea to have a way to sort of, like, log those and store it somewhere, such that, again, the… the AI has the context for when it hits that, it’s not going to go down the rabbit hole of trying to either guess what happened, give you a wrong hypothesis, or flagging something that everyone should be aware of already. And so, some of these examples are, you know, for example, like, one-time things, like outages. It could be patterns that you are deeply aware of. It could be, for example, like a migration, like a code migration, or, or something.
Where, you might have changed, you know, how you log, certain… certain things, or how the data is labeled, behind the scene in your data warehouse. So, all of these are, are really useful context, and so if you don’t have it stored somewhere, it’s not gonna… it’s not gonna travel, like, the… the AI is not gonna… is not gonna know any of this. Now, specifically to the question around, where all these live, right? Like, you, you know, like, these are great, like, but where, where, where do I put it?
If you’re familiar with Claude, if you’re familiar with Claude Code, you know, like, and in the, I would call it, like, a baby example that we’re gonna walk through, if it’s, like, very localized, one person, one project, very specific, very narrow, very unique, sort of, like, analytics analysis thing, it could be as, as, as simple as put it in the Cloud.md, if, if it’s not, you know, like, super, you know, like, very generalizable, or it’s very, very tailored to just one situation. That’s fine. And, the idea is not like, you know, hey, Claude.md is where everything should live.
For those of you who are familiar, Claude.md is almost like the system prompt for Claude, where every session, when you hit that specific project, or that specific folder, it’ll always read through it, so you don’t want to bloat it with all… all of these things. But again, if your project is really, really tiny, then that could be… that is a better place to store than not storing it at all. And, you know, like, certainly that’s not the destination, and you know. Shane alluded to… to this a little bit. What it can grow to is certainly a much more elaborate and, more sophisticated system. Like, no company is gonna be, you know, is sufficient to run on just a just a Claude.md file, for example.
So we, in our… in our course, in the five-week course that’s… that’s currently ongoing, we actually get into, how to, how to manage some of these, some of these contexts, and then how to evaluate them. in terms of outputs, in terms of improvements to the AI’s answers, and things like that. So you can imagine, and for those who are familiar, and actually we’ll drop the link to the AI Analyst repo here as well, so you can take a look, where a fully built-out Agentic system can have things like this, right? Like, you know, the first component of what the data is could live somewhere like a schema file with entities, with relationships defined.
The second thing around where the metrics live could be in in the metrics, folder, and you have one metric YAML file for each, for each metric, and then on the right, when you expand that, it would have things like, what does this mean, what is the grain of it, how is it calculated, what are the filters, who’s the owner, so on and so forth. And you can also have, for example, the third component here, how do we report on certain things and have a policy around it? And then, the… all the different patterns and the one-off events and, and corrections and logs, those could live As well, in these, different files, like works for corrections.yaml.
So these… this is kind of like the, you know, like a one flavor of what a, what a, a more complex system can look like that would cover the span of a very generalized, you know, like, analytics sort of, like, agent, for example.
Shane Butler: Yeah, I think one or two people said they had some sort of semantic layers in dbt or something, so, like, dbt, Cube, Snowflake, like, all these other solutions have, like, different, ways of structuring these layers for you. This isn’t, like, THE way to do it. I would say there isn’t THE way to do it yet. So you don’t have to take this, but it is to say, like, hey, like, there is… there’s some pretty simple things that you can do creating a structure like this that will make your system, like, much more trustworthy, reliable, and performative.
Hai Guan: Yeah, totally. And there’s a billion other companies doing the, doing, what do you call that? Doing context management and, and all that, so this is certainly an unsolved, place, and we’re trying to just give some inspirations for how that could look.
Shane Butler: I honestly think, like, there’ll be whole new, like, roles and teams that emerge that are responsible for… context management and context engineering. Maybe some of that… maybe some of the, like, data engineering, analytics, engineering kind of roles turn into roles that do some of this, but also I think there’s roles where it’s, like. Business owners, product managers, especially in terms of, like, the, like, metrics definitions parts, where there’ll be just, like. different things that emerge, where it’s like, what’s your job? Oh, I, like, manage the context for all of the metric definitions for this company, and I keep them up to date.
Hai Guan: All right, cool. So those are the big building blocks, I would say, for, you know, like, really important information and very basic information that would be very helpful to up the reliability of the, of the, of the agents. And so we’re gonna get into a quick demo here, where, again, we have this company, Fictional, really, coffee subscription app. We’ve got two CSV files, which contain things On daily sign-ups, activations, where we defined a little earlier, revenue by channel and ad spend. And then we’re just gonna ask it one question over and over again, and see what happens, which is, how many new customers did we get last week, and how does that compare to the week before?
We’re gonna run it through a series of different scenarios. One is absolutely no context, other than, you know, some of the base files that I’m gonna show you. The second one is when we have the definitions and all those components in the skill, for example, what does that look like? Third one is, if we… maybe we’ll skip this one, as well. But we will, you know, try to put that into the Claude.md file, and then see what happens, what changes to the, to the output. And then we’ll add one more line to it, and then we’ll ask again. And, and then we should, we should get, We should get a very different, different, answers with, with additional context. All right.
So, I’m gonna switch… I am going to pull up… my Claude code here, I am going to, put it in… I’m gonna use the desktop app. So, for folks who are familiar with the, with Claude Code Terminal, or, or, Yeah, for those who may not have, have seen it, Claude code can be run on their Claude desktop app. It is this… this guy is the usual chat, the general chat, and this guy here is the code, interface, so Claude Code interface. Before I get into the prompt, I’m gonna show you… the, the… the projects. So this is the Bruli demo. Folder that we’re gonna run Clot… that Claude Code is gonna sit on top of. I saw a question around what is revenue by channel, I’ll show you.
So we have a plot.md file in this, it’s very simple, a very simple project here. It’s just got a plot.md, it’s got a couple data, a couple CSV files, and that’s about it. everything is, is pretty empty. There’s one skill here that is called… Not sure if you guys can see it, but it’s called Weekly Review, and then we’ll take a look at what that does. So, I’ll quickly open, for example, the the different, CSVs here. I’ll show you metrics. Let’s see… So, metrics contain… how do I expand this? View… Zoom… So, for the metric CSV, for example, it’s got a date column, it’s got channel, so this is where the acquisition, took place. For example, is it organic? Is it through paid social?
Is it through referral? So, assume these are the three different channels. That, that we… that this company is able to track their acquisition source. It’s got sign-ups for on this, you know, like, for this channel on that day. It’s got how many activations. Activations is, if you recall, is defined as, first order basically received. Sign-ups is just like, you know, you sign up for an account on Brulee’s website, for example, or Brulee’s app, and then the revenue that’s associated with this channel on this date. So, pretty simple stuff. And it goes all the way to… Like, July. So May to… May to July, something like that.
And then the other… File here is, even more simple, like ad spend, which is just every single day, paid social, for the channel, how much, how much How much ad dollar did the company spend on acquiring users? So, really simple, two CSVs, nothing, nothing, nothing crazy here. And then, the pod.md, I will show you a little bit what that looks like. It is also extremely simple. It’s just got two sentences. describing what this project is, right? This folder is. So, this is Brulee Analytics Workspace. Brulee is a coffee subscription app. Metrics live in data. Which has these things that, that we just looked at, and, finished reports go into reports, the folder itself.
And then, you know, nothing, nothing crazy here, keep answers plain English and concrete, and audiences, the whole team, not just analysts. So it’s really got nothing, not much around here, so there’s no context or anything. And so, if we ask it… so in Claude Code here, we have it pointing at the Bruli demo folder, so it’s got the… it’s got access to all the files in this project, in this folder that we just walked through. And if I, for example, ask this question, That we saw. Which is… How many new customers? Did we get last week, and how does that compare to the week before? It’s gonna look at the… it’s gonna look through the data, right?
The two datasets, the two CSVs, because in Claude.md, it points it to, you know, hey, for the datasets, they live. They live in these, in these two files over here. And it will do… it’ll run some analysis, it will look at, okay, it’s got May 11th to July 5th, all the… the data in the, in those files, and so, I guess, yeah, today is September 16th, so it has to make a decision on what last week means, and then map to it, and so on and so forth. And then it’s gonna run analysis on those CSVs, and it’ll come back with some results. This is strictly off of just, you know, like, again, no context, very little, information about our preferences, the data quirks, or anything like that.
And it is almost solely based on trying to infer the intent and what these things… what are appropriate and sensible for these, based on the datasets that it has access to. Okay. Cool. So, in this one, There are two things you need to know before trusting. So… This is the… kind of, like, the results that it gave, which is, okay, it picked the most recent week. from the datasets, and then compared to the week before. And it did flag, for example, new customers is pretty ambiguous, so I don’t know what to pick, so I’m gonna give you both. Which is activations and signups. And then in this run, it actually thinks, okay, maybe new customers means activations.
And so that’s what it’s kind of, like, anchoring on. But then it did caveat that, hey, it’s not really sure, What that is. And then it flagged that, hey, there’s a negative 11%, negative 14% week-over-week drop in these two metrics, and This is a tracking glitch. So, because… so basically in the… in the data itself, They, there are a couple days where the spend, like, ad spend, basically dropped to zero, and so, but then to the LLM here, because it’s only looking at these two metrics, what it’s looking at is, like, hey, looks like there’s some tracking issues, because it just fell off the… fell off the, the face of the earth. For those two days.
And so, this is kind of like, hey, this is the conclusion, tracking glitch, and and this is what it is, and new customer, based on its best guess, is activations, which is actually correct. But in other runs, for example, if I run this a billion times. it’s gonna be a coin flip between what it’s gonna think activation means and what signups mean. So that’s sort of, like, the first one here, no context. I’m gonna clear this out, and I am going to… I’m going to… run it in a skill, and I’ll show you a little bit what that skill is. So, in this same project.
I’ve got a Claude skill that’s called Weekly Review, where it prepares a report for for… For an executive audience, that looks at the… the, the Bruley… the Brulee data. So… This has gotten the… the context, and the definition, and what the metrics mean, all the components are baked in to this skill, specifically. So, you know, like, the skill itself is called Weekly Review, Generate Monday Metrics Review for For the metrics.csv, and it’s got instructions for For what that is, like, where the data sits, how do you compare How do you compare the data, where to write the report to, what the format of the report should look like.
what the, what the preference for what the report should say, and so on and so forth. And here’s the context for the different components that we walked through, which is what the data is, so it’s got more information already here about the describing what the data… profiling what the data is. It is… it’s got what the metrics mean, so here… very, clearly, it says, hey, new customer is actually in activation. So someone counts as a customer on the day their first paid box is delivered. So that’s a very different thing than, let’s say, someone that’s a sign-up. And, and so on and so forth.
And then how we report things, and things you should know, that these are the hallway, you know, one-off events, or different patterns that people already know about in their business. So, I’m just gonna run it here, back to Claude Code. Just gonna run the weekly review, skill, and… It should. pop up with that exact report with at least all the definitions, what the data is, all of that, already known to the AI itself. So it shouldn’t be like, hey, this request is, like, you know, it’s ambiguous to generate a new customer’s report. So, while it runs… Might take a little bit to run.
But what it’s doing is, it’s reading the… reading the skill, redoing the… redoing the analysis, and stuff like that. Because we cleared context, so it doesn’t know, we already ran this before, because it didn’t actually generate a report out there. And that is all. That is all cool, so it’s doing all of these from scratch again. I don’t know about you guys, but Claude for me has been pretty slow this morning, so… Let’s see, so while it runs, I will show you what our next… Thing… what the next thing we’re gonna do is… Attendee, what model… which model do you use? I have Opus 4.8 here. I could switch it to Opus 5, but I really dislike Opus 5’s, way of responding.
Like, the way that it writes is, Just drives me nuts. Alright, cool. So, this came back. So, today, for… for the response, I have the data. Today is September 16th, so again, because it knows the range that we… that we, that we have preference for is always Monday to Sunday, so it knows how to pick the right Monday to Sunday, for example, from the file, even though there is a mismatch in terms of, you know, like, last week, where, you know, the data doesn’t span that far. And it doesn’t give you… it doesn’t give me, Or, I mean, it still, still gives me, kind of like, both metrics here, it looks like. Oh, I actually didn’t specifically ask it, the new customer’s question, but that’s okay.
So this is kind of like the summary here, right? Okay, last week looks like a bad week on the surface, sign-ups is this, and new customers. So it understands that activations actually means new customers, but because this skill is generating a report in… in totality, so it… it didn’t actually kind of, like, just mention, just focus on, activations specifically. But it knows that, new customers is activations, and that’s what we care about.
And then… And then it knows that, hey, this is a paid social outage, because in the… in the, what you should know, section, we specifically specify These… this event that we are aware of, where the campaign was paused, whereas previously, when it has no context, it was like, hey, that’s certainly a tracking, a tracking issue, for example. So, you know, like, one common question, for example, would be, hey, why don’t we just embed everything in skills, right? Like, hey, this works just as well. With skills. they only get invoked if Claude decides that the request or the prompt that you’re giving it, needs to use that skill.
And so, you know, like, maybe, if you want something that’s persisting throughout, you don’t want it to be in skill. But because here, we know that, hey, this is repeatable all the time, we would like this format, this is one option to put it in. But we certainly don’t recommend, kind of like, you know, putting things that’s common across. To be, to be in the, in the specific skill. Like, a definition skill would be, Wouldn’t be… Too useful, in general. Alright, cool. So, I’m going to clear context here again. And this is what we’re gonna do next. I am going to… Let’s go to the quant.md file. I’m going to embed all four components in the Claude.MD file directly.
I can put it in a separate file, but for the sake of simplicity, again, this is a very simple exercise demo here. I’m just gonna put it here, and let’s ask that same question again, and see what happens. So, I’m just gonna paste in all these. We sort of saw it already before, what the data is, what our metrics mean, how we report them, and the things that you should know. And I’m gonna save this. So now, instead of two sentences. It’s got the entire section around, the four things that we really care about, and we will… Ask it the same questions again. how many new users did we get last week?
Again, because we cleared context, it’s going to… Redo the calculations and, redo the analysis for us. But this time, it will look through the Cloud.md file first, and so it has all the… it’s primed with all the context already. And certainly, once… once this is done, we should be able to see, what it is. Okay, cool, it’s already done. So, data runs through July 5th, again, That’s, that’s the… the upper… the most recent data that we have in that file. So this is the response. Last week, we got 908 new customers, down 13% from the week before. It didn’t mention signups, because it knows, hey, new customers specifically refer to activations, right?
And then the caveat, the main story here is, most of the drop was in paid social outage, not softening demand. That is, again, part of the what you should know section. That we already primed it with. And so now it’s getting all the right answers and stuff like that. What are we doing on time? 10 minutes, Yeah, I’m gonna… I’m just gonna stop here, so you guys see a… kind of, like, a little bit of an evolution of what this, you know, like, how you can ground Claude in your… in your, in your specific company’s data, in your specific company’s context. And so on and so forth. And you can imagine how to sort of, like, fan this out. We can make it as complicated as we want.
We can make it as elaborate as we want, or we don’t. There’s a billion different ways to expand all of these, but just generally speaking, the… the thing to… The thing to make sure that Claude or any AI is able to understand your business is to write it down somewhere, and write it down as exhaustively as you can, and as, you know, as fresh as possible. Alright, cool. Let me get back to the slides and, and we’ll close it out. Okay, cool. So… Yeah, so we saw the before and after, right? Like, before, when we asked what a new… how many new customers did we get, it was basically a coin flip.
you know, like, in many of the runs I did before, it picked signups because, generally, new people means someone who’s brand new to the platform, or something like that, so it picked signups. In that case, it would be 1664 in terms of numbers. And and then it was very confident that, hey, there’s a tracking issue here, so nothing to worry about, for example. Whereas when we grounded it with with the context and the data, and the quirks and definitions that we have, then it correctly named activations as the new customers, and that the root cause of it was because of this. So, it didn’t… the model did not get smarter, obviously. It’s the same… same model.
It just agreed with us, it just knows what we know, and and takes that into account. So, yeah, a wrong definition can be wrong the same way, every session. So, you know, like, this doesn’t guarantee correctness, but at least sort of like, the, It’s, you know, like, giving it the context that we all have is going to be, in a way, make it a lot more reliable. All right. Cool, let me see, and… I think this is the last slide, so we can get into questions after this. We have some free lessons coming up next. We have one around, analytics interviews in a couple weeks with AI, so pretty, should be really fun. Chavez is going to be running that.
next week, I think, Shane, we’re gonna do a chat GPT, data agents?
Shane Butler: Yeah, I just made the… Hold on. I just made the session for it, so if you want to sign up for this. I just dropped it in the chat. I just made this today. So last… or, like, 5 or 6 days ago, OpenAI released a data agent that’s embedded in ChatGP to work. And it’s not just like, hey, it answers your questions, it’s like, you can, like, create interactive dashboards, you can edit the charts right there, like you would in some, like, data tools. It’s pretty cool. We’re not experts on it. No one is, unless you probably work at OpenAI. So we are going to have… this will be a very different… this isn’t going to be a demo or a lecture.
We want to do, like, we have, like, labs in our live courses, where we’re obviously working with smaller cohorts of people, but we’re going to treat this like a live lab. Where we’ll provide you with some data, some setup guides beforehand. You can also bring your own company data, and we’re all gonna work together to try and, like, stress test this thing, see, like, if we hit it with, like, real-world questions, not just, like, the kind of, like, demo-y marketing questions the OpenAI marketing team shows you. what actually happened, so I think it’ll be pretty fun. Would love… would love for you to join. But yeah, I’ll be next Wednesday at this time.
Hai Guan: Nice. Yeah, that’ll be… that’ll be really fun. You know, we’ll all be… we’ll all be hammering it for the first time. No one has really done too much, too deeply, with those yet. And I mentioned earlier that we have, or we’ve been mentioning that we also have five-week courses that we teach. The one that’s really relevant, for example, if you wanted to get deeper, go further in the learning journey around, context management, eval. like, AI evaluations on, the agent’s answers, things like that. We are… we have a 5-week course around Agentic analytics, how to build an Agent… an AI, or an Agentic analytics system.
And, kind of, like, you know, like, the ways to structure things, the ways to evaluate, the ways to improve context, and also, like, how to use different models to, to kind of, like, review each other’s work, and so on and so forth. So, our current cohort is ongoing. The next cohort is going to be in November, so if you’re interested. Definitely, Definitely join us, on that.
Shane Butler: And then Attendee asked about, actually, how do you know if this stuff’s right? How do you know, if this stuff is helping or not? We have, yeah, we refresh the… we spend an entire week on doing AI emails, specifically in the, domain of Agentic Analytics, then we spend an entire week that really pairs well with this in week four. We go into context engineering management, and we see how… engineering, creating context, but also managing it and putting it in different ways, affects the evals, because we build, like, a system of measuring the accuracy and reliability of it. And then you had a question around ever-expanding.
As more… use cases and more novel questions come in to the analysts, you just have to have, new, kind of, evals around each of those, so that’s what I meant by ever-expanding. We… so we have spent a whole… we spent two weeks, basically, on… on that. In this course?
Hai Guan: Cool, yep. Yeah, we’d love to, if anyone’s interested, we’d love to have you join us in the next cohort, and then, on, in parallel, we also have another 5-week program on, 5-week and also async program on what we call AI Analytics for Everyone, where we teach the analytical foundations for, all the… with all the best practices around, how do you frame a, a stakeholder question? How do you, make sure that, you’re working on the most important things when you do certain You know, like, how do you kind of, like, influence stakeholders? How do you do… what are the best practices and the different, frameworks for conducting causal… causal analysis, or root cause analysis, so on and so forth.
And then so, like, preserving the thinking and the foundations of what makes an analysis robust and trustworthy. And also, while delegating all the technical execution components to AI, that’s sort of, like, how we, how we think about that. Those are much more evergreen in terms of, in terms of concepts that, I would argue in the age of AI is becoming even more important. Like, if you’re, you know, like, if previously. a lot of people are just good at, for example, like, writing a sequel, and that would be their entire job. Now, more so than ever, judgment, taste.
foundation, and all of these, like, irreplaceable skills are amplified in terms of their importance in how successful a person can be in their career. So, we emphasize on that, so feel free to check that out as well. And yeah, for everything else, you can check us out on maven.com slash dataneighbor, or, go to AIAnalistlab.ai. We have our previous courses, lessons, free resources. We offer a ton of those on, on there as well. And feel free to ping us directly as well on LinkedIn, LinkedIn or, or, or Slack. For those of you who, who are already in our community.
Shane Butler: Couple… couple things. Hai mentioned this, so the Agentic Analytics one, next cohort is in November. There’s probably gonna be… I’m just being… being real, there’s probably gonna be a price increase before that. If you… if you register sooner, you’ll lock in the current price. And then… but the AI analytics for everyone. That next cohort’s in October, the live cohort, but we offer a self-paced version of that now, like Hai mentioned, so you could actually go and start that today if you wanted. You just don’t have access to, like. live office hours is basically the main difference between the two of those.
And then there was a question from Attendee around, they don’t… their company doesn’t allow Claude… co-work to connect to their data warehouse and BI tools. So the, Agentic Analytics, like, we used, Claude Code in that course. But the kind of, like. foundations we teach, go across any sort of Agentic system. I actually use Codex primarily. We talked to, one of our students earlier, today in office hours this morning about, doing it with, like, Microsoft Copilot, In week 5 of that course, we actually do… we go beyond Claude, and we work with open weight models, we work with codecs. I’m gonna add something around there, Copilot.
So I think those… I would say look at the syllabus, look at the kind of material, or reach out to us, we’re happy to talk about it more. But, I mean, if you’re trying to get into Agentic Analytics. even if you’re using a kind of enterprise Agentic analytics solution, that learning’s gonna be relevant to you. And then… In terms of actually being able to do the course, like… We will be working with… with Claude Code, so you can… you can get a personal license for… You know, a $20… It’s for that month, and then we supply, like, synthetic data for the course.
So there’s kind of two tracks to every exercise we do in the homeworks, where you can use our data set we provide, or you can connect, to your companies. I saw a couple other questions up here. There’s a really good one from Attendee. Yeah, they’re still here. What’s the best way to get hallway insights or information from the team? Hi.
Hai Guan: Good question. So, one useful way would be to mine Slack, for example, or, like, whatever internal communication platforms that you guys might have. Like, you know, there’s certain always gold mine of information, previous analyses, for example, or, like, previous meetings and things like that. and if there’s, like, I don’t know, like, weekly review meetings, or weekly something, metrics, something, something, like, there’s probably a trove of information that is being communicated to different, different people, and, you know. So there, there’s a… There’s a lot of that. I think the really important thing is, like, you know, going forward.
how do you set up a system, or set up some process for logging them? So, like, you know, what’s done in the past is done, so you can only try to, you know, like, scrape together, hopefully, the pieces of things that people know about. But going forward, for example, whether it be, hey, I have a Google Sheet that logs all these incidents, or every time when there’s a, you know, bug report of some sort, it goes to like, it triggers some events that, then, that then gets, stored somewhere. Like, all of that is, is, I would say, more important. As the, kind of, as time passes. I don’t know if you have any thoughts around that, John.
Shane Butler: I think Attendee had a good idea, like, some sort of questionnaire. I think even, like, better than a questionnaire is, like, you could totally make a sort of workshop out of it that you host for the team you’re working with, if you wanted to, like, try and, like, consolidate a bunch of stuff. at one point. This wouldn’t be, like, an ongoing thing. I’m just thinking about in terms of when I tried to get information around, like. I’m trying to define the metrics and get everyone aligned on the team.
We actually hosted a series of workshops where a couple of them were, like, virtual, where we walked through, like, trying to connect What we’re actually trying to build to the user value to, like, potential, like, driver metrics and North Star metrics. And then we actually got in person later on, just because it worked out with our company off-site, and talked about a lot of that, and so that created a lot of. Artifacts and, like, the transcripts from those calls were… I was able to pull those in, And so we actually created those for, like, for this… for the reason of, like, trying to collect it.
That wouldn’t be, like… you know, if you’re just trying to get, like, day-to-day insights of what happened at the company, that’s not gonna be helpful, but I think for, like. metric definition alignment, just actually hosting some structured way of doing it with the team is helpful. If you want to see… people are… it’s pretty funny to see people argue, too. You know, people are very passionate about their… this should be 3 weeks, no, this should be in a 2-week window. It’s like, geez, I don’t think it matters that much, let’s just pick one and go.
So you have to do… you gotta, like, time-bound it, too, otherwise, like, people will really get stuck on some minute details that don’t really matter.
Hai Guan: Totally. Alright, do we have it.
Shane Butler: One more question from Attendee at the bottom here. Hi, maybe we can close it out.
Hai Guan: Yeah, how much of what you write for this context? Do you read line by line versus not? Do you mean, Attendee, do you mean, like, verifying what’s being written here to make sure it’s correct? Okay. Got it. Yeah, I mean… I think it’s always good practice to just double-check and verify. I think it depends on how you write down the context, right? If you wrote it down by hand, for example, or if it comes from some sort of summary of a you know, like a… like a ver… like a… like a transcript.
Like, the thing to make sure of is just, like, you know, it… it was… it accurately captures what really happened, and there is no kind of, like, estimate… estimations or, again, like, back to, you know, like, LLMs are non-deterministic. If you have it summarize things for you, you wanted to make sure that it’s Capturing that correctly, and not… You know, in a nuanced way that could be misinterpreted, for example. writing that context slowly over time, making sure you keep it high quality. Yeah, totally. Yeah, absolutely. Yeah, that’s what I was alluding to, in terms of the… you know, like, a process going forward to log it.
You would probably want something that is lightweight enough, so people would still want to do it. Like, it’s not about… it doesn’t bog people down for, you know, hey, crap, I gotta write this thing, it’s gonna take me, you know, like, high friction to start. It is not gonna be great. But you also want to make sure that, it’s also done in a way that you know it’s gonna be high quality. So it’s not just, like, spitballing and giving it a bunch of, just random, random crap.
All right, thank you everyone for joining and for staying a little bit over, and I will send out some materials from this, from this session here, and we would love to see you in our future Lightning Lessons, sessions, or, or any, any other, any other forums. Thanks, everyone.
Shane Butler: Thanks, everyone. See ya!
Hai Guan: Right.
Shane Butler: Thanks, Hai.