Hai Guan: Bear this.
Shane Butler: Hey, everyone! Welcome, welcome.
Hai Guan: Hello?
Shane Butler: Can everyone see… the screen. Actually, if you can see the screen, drop in the chat where you’re from. Then we can kinda… Open up the chat. And get a check on screen. San Francisco. Another San Francisco, Oregon… North Carolina.
Hai Guan: Ottawa.
Shane Butler: DC Seattle, Ottawa.
Hai Guan: Serbia. Oh, Keshkaias. Love Kushkais. Attendee, if I pronounce your name correctly.
Shane Butler: Illinois… Nice… We’re just waiting for people to join. We’ll get started in just a minute here. Maybe, like, 1202 PT or 1203.
Hai Guan: Yeah, give it a couple minutes.
Shane Butler: Another San Francisco… High is located in the Bay Area. And then I’m not too far from the San Francisco folks, I’m in Tahoe. Berlin, Calgary. Nice. Really like Calgary, Attendee. Spent a few seasons in the… Danmort, outside of Calgary. It’s really beautiful.
Hai Guan: And what are your roles, if you guys want to put it in chat as well?
Shane Butler: Yeah, it’d be nice to see, kind of. What kind of spread we have? I’m a data scientist, bank treasury. Oh, hey, Attendee. Good to see you again.
Hai Guan: Yeah, hey, Attendee. Product manager, great.
Shane Butler: Marketing analytics, IT Ops, CPG Consulting. Oh, hey, Attendee, good to see you again, too. Nice, lots of familiar faces in here today.
Hai Guan: Yeah.
Attendee: Oh, here it is.
Shane Butler: Oh, boy.
Attendee: vote.
Shane Butler: Hey. Unemployed pivoter, nice, I like it.
Hai Guan: Where are you pivoting to, Attendee?
Attendee: DBD! I have a background in quality. I think that it has a lot of, You know, tracks in a lot of different… In, places.
Shane Butler: Absolutely.
Hai Guan: Yeah, very cool.
Shane Butler: What do you think, hi? Should we get started? I’ll keep admitting people.
Hai Guan: Yeah, I think we can get started, and then as people roll in, hopefully they don’t miss too much. Yeah, cool. Let’s see. So, yeah, let’s get… let’s get going. Welcome, everyone! Welcome to a Wednesday lesson around root cause analysis in Cloud Code. Thank you for joining us. Today’s agenda is going to be roughly 40 minutes or so of me talking, so, combination of slides, and then also demoing some stuff, some workflows in Cloud Code. That, that hopefully will, will be helpful, on this, on this… on this very specific topic around root cause analysis. Now, before we dive in, so just a quick intro for us here. My name is Hai. I am part of the AI Analyst Lab.
We do a lot of, Courses, free workshops, and also email series, free email courses, on our website if you want to take a look. We’ve been doing this for a couple months now, or a few months now. We open-sourced a AI analyst repo. Probably many of you have heard it or have used it. If not, we’ll get into a little bit of a preview on that one. And I’m currently the head of data at ENTRE at a legal tech company called ENTRE, previously at different consumer tech companies like LinkedIn, Pinterest, Meta, Nextdoor. That’s where I met, Shane and Shravia, who’s, part of this crew. So, yeah, nice to meet everyone, for those we haven’t met, and nice to see folks that That, that’s been with us before.
Shane, over to you.
Shane Butler: Hey everyone, I’m Shane. Yeah, as I mentioned, I lead the AI Analyst Lab with him and Sravya, a principal data scientist. Same, legal tech company high as that. Been in… data for a little over a decade, most… most of it in product, but across B2B, as well as, kind of, you know, direct consumer products. Yeah, most recently, I’d say the last two years, been focused on AI evaluation and, degeneric analytics. Prior to that was, like, kind of more classic product data science.
And then our, our third member of our team, Stravia, I think she’s gonna be joining us towards the end of the session, she’s currently on a very relaxing, a retreat somewhere in North Carolina, I think, so she’s probably at the spa right now, or… We’re getting a sound bath, or something like that. Whatever they do with those things. But, yeah, glad to be here. Thanks for spending the hour with us.
Hai Guan: Cool. All right, so, we’ll get into this, right now. Just before we get into the actual content, does anyone… if you were to describe what root cause analysis is, how would you describe it? Put it in chat. What’s one thing that jumped to your mind? When you hear this… these combinations of words.
Shane Butler: The five whys. Oh, that’s such a… oh, two people said it. Yeah, so good. Decomposition analysis, yep, volume versus grade effect. Not my problem. Bruce Principles.
Hai Guan: Principles, structured… okay. Cool. Yeah, it’s good to… good to hear, good to see, we’re… what… what people… how people would describe this. So… where’s my button? So, root cause analysis, typically is trying to answer, like, an observation. So, you have a… some sort of metric, right? So, in this example that we have here, probably folks in this room might have gotten very similar questions, or slash, have some… have very similar curiosity when, when… when certain anomalies happen in your day-to-day metric. So, let’s say… you know, this is not uncommon to hear, hey, revenue’s down 12%. What happens? That’s generally almost like a trigger phrase, or some variance of this, would be.
Would be… would trigger. a root cause analysis to diagnose and decompose down to something that is actually useful and actionable. And so, you know, right here in the second line here, you have until end of day go. Typically, that’s, you know, like, if this question comes from an executive or, like, a VP of some sort, generally it’s pretty urgent. You know. coincident, or, surprisingly, or unsurprisingly, when, revenue goes up 12%, it’s less urgent. But, you know, it’s a way to… it’s a way to… pinpoint as precisely as possible, the story behind why a number moves, and what the biggest contribution to that is. So, we’ll get into that in a little bit, in terms of how to actually do it.
Okay, cool. So, when folks get that question, the… There’s, you know, like, in terms of common practice, I mean practically, not theoretically, or not best practices, three things, just being a little bit, you know, sarcastic here. Number one is probably panic. It’s, hey, down 12%, sounds like, sounds like pretty urgent, but, like, where do we start? There’s so many ways to slice and dice. no clue how to… how to act. You know, that’s pretty common, especially if, for example, a metric is new, brand new, you’ve never seen it move before.
probably for something like revenue, it is well understood, but, you know, there could be a billion other metrics that might still be important for the business, and if you haven’t seen it before, generally it’s not as straightforward, perhaps. So, you know, like, there could be merit to panicking. The second one here is, just, you know, pick a dimension that you have a dashboard for, and then see, you know, if something moves, and just be like, hey, that’s it, that’s the cost. And then just tell a story off of the only dimension that you have access to, for example. That’s pretty common as well. And then the third one is, go super deep into certain, you know, like, paths.
So, for example, it’s like, oh, wow, like, this looks like it’s, you know, went up a lot, so let’s look in, look in there, and then, you know, the more that you drill deep into, the less you lose the specificity of. hey, exactly how much of contribution, when I go this deep, is it actually affecting the top line, and how much is it actually explaining it? So, this is kind of like what it’s referring to, drilling to nowhere. You know, like, if it’s unstructured, if it’s not very specific, you get to a lot of Motions to try to get to a path that may not actually lead you to the right… to the right direction.
So, you know, like, I’ve seen, and Shane’s seen this in the past, I’ve been guilty of it myself when I first, you know, like. either get onboarded to a new area, or, have done this kind of, like, without a lot of reps in the past. It is a lot of, you know, like, hey, see what we can find, and, hope for the best. Does this resonate with us? Hopefully. I saw Shane’s comment throwing spaghetti on the wall, yeah?
Shane Butler: Yeah, I mean, the drill of nowhere thing is pretty dangerous, too, because it’s also just, like, you can also see all these totally unrelated, like, spurious correlations, that actually have no… nothing to do that can, like, mislead you. Like, red herrings, basically, right? That can just totally mislead you around the cause when something else is going on. When it’s not coming from… Well, I’m sure you’ll get into, like, the right way to do it, so I won’t spoil it.
Hai Guan: Yeah, okay, cool. Sorry, my computer applications are… Going crazy here. Okay, cool. So, we’re gonna go through a pretty simple framework to understand, or to ground things in as specific of a way as possible. And, this framework actually lives in our open source repo. That, Shane could share the link to. This is the AI Analyst system. It is open source on GitHub. Anybody can clone it, everybody can use it, everyone can build on top of it. It’s got many different skills, agents. Effectively, this is a agentic system that mimics what a senior product data scientist would behave like in the analytics workflow. So, think of it as, the AI analyst system plus clock code.
You have a very, very capable intern, or a data scientist partner, to help do analysis and stuff. And obviously, within that, there’s a bunch of skills agents and, one of One of those is Root Cause Investigation, which bakes in the the framework that we’re gonna walk through, and it will know what to do, on its own, as a… as kind of like a AI system. All right, cool. So, what do we actually do when we are encountering a, a request to, or encountering the need to understand what is causing the thing that we’re looking at.
So, we call it, I guess, the level zero to end decomposition, and meaning there could be multiple different… multiple different dimensions, multiple different channels, multiple different dimensions that you can slice and dice by. But there is a way to think about how to do… how to exhaust your possibilities in each one of these. So, the… the… sort of, like, the idea here is, we have a level zero, you go from… you go actually state the metric, and then, making sure that you’re clear what is needed to… what is being looked at.
Level 1 is pick a dimension, so something that you have a hunch on, and then level two is, within the or I guess back to level one is find the segment that explains the movement, and then level two is within that segment, pick another dimension, break that again, and then you keep going until you kind of, like, figure out something that Holistically explain the majority, if if not the entire, kind of, like, movement in the metric that you observe. So, start at the whole, break by one dimension, and then drill into the segment, and then keep repeating. And just being very systematic about it, so that it’s not like a, wild turkey chase. Let’s see… okay, cool. So… Level zero.
So, you know, obviously, in order to actually diagnose something, you have to define it. So, define the metric precisely. So, the… kind of, like, the problem statement we saw, revenue was down 12%, what happened? That was not exactly super precise, right? Like, revenue over which period, you know, how do you count revenue? So on and so forth. You can imagine, you know, like, if people say something like, hey, engagement is down, go figure it out, like, down in which in which, like, what does that mean? What is engagement? How do you measure it? What time window? What is the grain? So on and so forth.
So, first, most important thing is to be precise around what that metric is and where the observation comes from. The second one is the, quantifying the… the change. So, is it absolute? Is it relative? The comparison window, so down relative to what? That kind of stuff. So, is it down 12% for the revenue example that we gave? Is it prior to a 4-week average? Is it week-over-week? Is it year-over-year? What is it? So, be very clear on that. And then, setting the bar. So, you know, like, maybe something is not worth investigating, for example.
So, you know, it may be down 12% versus, like, a week ago that had an explained, Increase of… say, whatever, like, 12%, and you’re just back down to baseline, so then there should be no panic on that. So, just making sure that stuff becomes precise, and it’s not just a, hey, it’s down 12%, let me start firing up all my dashboards, let’s… let me start firing up all my reports and stuff like that to go off on the adventure. Does that make sense? Okay, cool. Now, once you are able to Pick the, like, you know, hey, perhaps you’re, you’re being, you’re, you’re being extremely precise, you know exactly what you’re comparing against, you know the exact metric, and, and, and stuff like that.
Now you go into kind of, like, the, the next step here, which is level 1. So, picking a dimension that explains the most variance. So… the… the, the, you know, the easiest way would be reach for a dashboard, for example, to look at, you know, like, things that you have, that’s already charted. Like, that’s… that’s the easiest part. That’s… typically pretty, pretty simple to do, and then, you know, like, you eyeball it and, and, and, and, and stuff like that. The idea here is Try to be as exhaustive as possible, so, like, if you know that, you know, like.
the segments that you care about for your business, for example, is the demographics, the, the regions, the, the, the, the acquisition channel, the, you know, whatever else, like, that pertains to your business that you know are pretty important, and that you, you’re, that, that typically is, is something that you know your business is sensitive to. Just make sure that you check all those dimensions. Understand, kind of, like, across these most common dimensions, which one explains the biggest the biggest contribution to the actual change.
So, let’s say, like, if your revenue is down 12%, and that amounts to, for example, like, $5 million, and then you look at these different dimensions, and you saw, maybe, like, region is, you know, like, is, is, APAC is down, let’s say, like, $4.5 million, that’s your, that’s your direction. Like, that’s where you should go first, even if, let’s say, like, another dimension says, hey, there’s this, I don’t know, like, TikTok acquisition channel is down, you know, like. 80%. Maybe that’s still worth investigating, but it doesn’t explain, kind of, like, your… your, your… the… the biggest… the biggest proportion of your pie, if you will.
So, just making sure you look across, and then kind of, like, figure out the, the one that explains the most. Of the change that you’re observing, because, most likely, that’s gonna keep… that’s gonna point you to the right… the right story, or at least the, the right direction where you can keep slicing. Cool, so which one has the most uneven impact? Okay, and then you just keep going with this. So, within that segment, or within that dimension, for example, then what is… the next segment, that’s… that… that’s, that’s helpful to… to keep going.
So, almost every, piece of, kind of, like, data, like, you know, region, it might… you might be able to break it down by devices, for example, or you might be able to. break it down by marketing mix, or marketing channels, so on, so on and so forth. So, here is where If you could keep slicing, and if the contribution continues to remain kind of like a big basket of the proportion of the total change, then you can keep looking at what is the most granular way, and that typically is where your root cause lies. So the specific example here is revenues down in EMEA, that is Typically too broad, and like, you know, like, that’s an… that’s a good observation that points you to a direction.
very unlikely that something is in EMEA that is, like, that is only affecting it. Maybe it’s seasonality, maybe that’s true, or, you know, some external events and stuff. Internally, it is… it… it… there might be more of a story to it than just, you know, simply this big piece of the pie. And then, you know, like, when you start kind of, like, drilling it in, in deeper, for example, in the example that we’re gonna show in the Cloud Code demo, we’re gonna use a fictional dataset that, that we have in the… in the AI Analyst repo, where… just think of it as, we have a fictional company, e-commerce company, called Nova Mart, think of Amazon, so exactly how… how… how it works.
And, perhaps, you know, like, there’s membership people, like, there’s a cut around, whether a… whether a shopper is a member or not of Nova Mart, and so in this example here, this is getting closer. Maybe it’s the, the members the paid members in Germany or UK on iOS is contributing to most of that decline. Now, that’s getting closer, and you know exactly the isolated area within the region that’s explaining, kind of, like, the top line. Now, you can keep going as well, where, you know, hey, why is… iOS down, it could be because, it’s a particular channel that they’re landing on that is… that is, that is different. And so, you know, like.
as you keep going, you’re gonna start seeing the… the specific cause of something, if it indeed is an anomaly. Hopefully, hopefully that helps. Oops. Okay, cool. So, the whole idea, really, is like, hey, if you have a dimension, you go look at what’s the biggest contribution, and then you slice it again by the other dimension types that you have, or segment types, and then you keep going.
Where, you know, like, the more that you do it, and the more that you maintain your perspective around, hey, how much movement this actually explains the top line, then hopefully, at the end of that exercise, or by the, you know, like, nth cut, you’re able to see, like, you know, hey, this is probably the anomaly that I’m… that I’m seeing that explains the whole thing, and then you can start correlating against other. other things. Here’s a reusable prompt for, you know, what we just talked about. Where, you know, you can run it in any of the AI chatbots or any, your favorite ChatGBT, or Quad, or whatever, that would pretty much mimic that, that, that, that, that whole process.
And you just have to give it the context, for example. Okay, cool.
Shane Butler: Maybe…
Hai Guan: That’s in.
Shane Butler: Maybe just to pop in here for a second, since, like… so we’ve been talking a lot about, like, just the, you know, the methodology and the framework of just root cause analysis in general. I’m sure a lot of people here have done root cause analyses. This… Obviously, this lightning lesson and a lot of the stuff we do at the lab is more around, like, how do we use agentic systems to do this? And, I mean, I think, you know, Hai just shared that prompt, and he’s gonna go and demo some stuff in Cloud Code in a bit.
But I think one of the things I just want to call out here, why it’s really helpful to have agentic systems, Versus just us do it, is that all this stuff can be done at a level of robustness and depth that we just do not have time to do. So, there’s… if we think about, like, the steps we just went through, right? It’s like, okay, is this… is this real? Like, align on the metrics and stuff, is this a real problem based on the baseline? Then we form all these hypotheses around all of these dimensions we want to drill into. We don’t just want to, like, throw spaghetti at a wall and look at everything, we want to have, like, strong hypotheses around them.
Agentix systems like this is gonna be, like, your co-pilot to think of, like, hey, are these really, like, solid hypotheses around, like, dimensions we should check? And then they’re gonna be able to go down and drill down into that one-to-end space that Hai mentioned at, like, in parallel at a speed we just wouldn’t be able to do. So, like, in my job. at any company, like, I’ve spent… I remember a couple times when I was at Nextdoor that I spent 2 months on some, like, code… red, code yellow, root cause analysis, trying to understand why our sessions were down.
And for me, just as, like, a human, like, I’m, like, going through all these different dimensions, even in a structured way, and I would have to keep circling back to new ones as more data came in. It’s hard for me, even when documenting it, to kind of, like, keep track of all the threads and what links together. And so when you have a system like this, where it can literally have sub-agents do it all in parallel, and then write all its finding to, like, the same knowledge base, you’re just able to manage and store, like. That… that context of the problem so much easier, and expedite the, you know, the speed of the analysis of the root cause analysis, but also the depth as well.
So that’s why… how we found it, like, really powerful, here. And it’s nothing, you know, it’s not… you can add some, like, more fancy, like, robust causal inference methods to this to make sure, like, hey, this is, like. You know, the truly isolated driver, but even just at the, like, most naive level of segmentation, and attribution, which is, like, it’s not gonna screw that up, it’s extremely powerful. So, just kind of want to highlight that, since that’s more what this This session’s gonna be about, and obviously what we focus on in, like, our boot camps and our full week course. But yeah, back to you, hi.
Hai Guan: Cool. Yeah, great points. Yeah, I remember it used to take, yeah, hours to just cut… cut one document and go on and, like, just keep going. So, with the help of Claude, for example, and the Agentic system, you’ll… we’ll… we’ll see it… we’ll see it run. Okay, cool. So, back to this, scenario. So, again, the whole e-commerce fictional company that we talked about, Nova Mart, that’s what we’re gonna run some… some, some prompts against. So the setup here is… We know that in June 2024, we have a ticket spike, so tickets would be, you know, call center supports around people calling in to To… to get, customer supports.
And so, tickets were way up in June 2024, versus May baseline, and, and then, you know, we’ll… We’ll just use Cloud Code, with natural language prompts. with the AI analyst system, and you’ll see that we don’t write any code, but it’s gonna do all the drilling, the dimensions, and then we’ll, we’ll see… we’ll see what that looks like. So, let me share my other screen here. Okay. Can people see this? Do you see a black screen here?
Shane Butler: Yeah, maybe just to get a feel for the room, too, in the chat, if you wouldn’t mind dropping, if you’ve never used Cloud Code before. drop a zero in the chat. If you’ve used it, but you haven’t used it for analytics, drop a 1, and then if you’ve used Cloud Code for analytics tasks, drop a 2. Thanks. Okay, cool, yeah, a lot of people have… have either not used it, or have used it, but not for… Yeah, Attendee… Attendee, you’re a 2, man. You’re a 2 now. You’re in it.
Hai Guan: Attendee, you’re.
Shane Butler: 1.5, nice.
Attendee: It will be in a few weeks.
Shane Butler: Yeah, there you go. You will be… after the bootcamp, you can officially.
Hai Guan: You’ll be, you’ll be 3. Cool. So, yeah, so what I’m… what you guys are looking at here, for those who are not familiar, this is… I mean, this is… think… think of this as the AI Analyst, repo that we clone to our… to my… to my, to my… to my computer, so… basically the link that Shane shared with you all, you can use this, I’m in VS Code, so you can clone that entire repo down to your local machine, and then I can work with it via Cloud Code, which is this, this terminal. thing here.
Now, it looks a little bit intimidating, especially for folks who haven’t used terminals in the past, or are not, you know, like, maybe software engineers or technical folks, but it really is just an interface where you just chat with it once you have it set up and installed. So, what I’m gonna do is, I’m gonna start pasting some prompts in, where… I’m gonna say, hey, can you describe what the Nova Mart data is about? What’s the business model? Etc? So, just to warm it up, like, just for us all to understand what this is that we’re looking at.
and this is probably a good practice, you know, anytime that we look at a dataset that we’re not familiar with, what we typically do is, like, explore it a little bit, right? Like, you know… what are we looking at, and stuff like that. And with this company, Nova Mart, we have a lot of rich information about it in the form of the whole context of what this business is, and how they make money, what kind of data we have, that kind of stuff. So, here’s what it responded with. What it is, is a simulated e-commerce business. As we talked about, not a real company. Models the full year of operations from 2024 to 2025. for an online retailer, so on and so forth.
It’s given me the, you know, has 50,000 people, set 47,000 orders, 6.5%, fixed 6.5 million. events that it’s tracking, 500 products. Business model is a direct-to-consumer online marketplace, with a transaction… with transactional, you buy some stuff. with a membership subscription, so, kind of like a $99 Amazon, thing, Amazon Prime. And, it’s… it’s got… we’ve got data around acquisition, engagement, monetization, retention, so on and so forth. So, lots of these information that it just gives me, and I’m just asking it in English. So, I’m gonna do a follow-up prompt here, because we already know the story a little bit. So, how is ticket volume over time look like?
A grammatical mistake, but that’s fine, not an English major. Can you make a chart? So… in here, I’m actually asking it to explore the data a little bit. So, in the setup, we know that as part of this… as part of these… this, dataset, we have… we also track ticket volumes for call centers, and so, you know, I’m interested in what that looks like over time, just to confirm, again, back to our… Back to our, framework, just to confirm that what we’re… what we said in the slide is actually true, and so what’s better way to… to… to see that than to, to actually look at it charting out? So, what it’s doing is, it’s basically Looking at the… the instruction files.
Think of the repo, like the different folders and files here, as instructions for Claude to behave like a senior data scientist. So what it’s doing is, hey, I’ve got a bunch of instructions around here, so I’m just gonna go look at the right ones to make a chart, to explore data, figure out where the data is, that kind of stuff. And then it’s self-healing, so… Anytime that it runs into errors and stuff, it’ll figure it out on its own. I almost never read too much into what these are, because I don’t understand most of it. But it will fix itself, and it will figure out how to actually, you know, keep going.
Until it can’t figure it out, then it will tell me, like, hey, here’s something that I need from you. So, now it’s done. It’s done doing the charting, and it also summarizes Sort of, like, the… the, the findings here. So, here’s the chart. So, I’m just gonna look at this, and… This spike here is something that I’m interested in, sort of, like, taking a look, and… And this is… and it bakes in, sort of, like, the storytelling with data best practices for charting in the… in some of the skills and agents, and so that’s why this looks very different than when it’s charted here, versus if you maybe just prompt Claude to be like, hey, can you graph… can you make me a chart, for example?
Shane Butler: Hey, a couple questions. Hi. Just to, Just to pause for a second, Question, do we need to call out… question around this being synthetic data, and can you use this on real data, in a real company?
Hai Guan: Yeah, yeah, definitely. So, you can connect it to your data warehouse, your data sources, you can give it CSVs, there’s native connectors for, I think, Snowflake, and, DuckDPs, and all the other, and a few, and a few ones. And certainly for the stuff that’s not on here, for example, like Databricks, I don’t think it’s part of it, you can ask you can work with Claude to get that set up, for example. But yes, this is portable to, you know, real data, everything.
Shane Butler: And then, questions around, can you use this… is it just cloud code, or can you use GitHub Copilot? Can you use it with Codex? I just say with GitHub Copilot, we have not developed anything, but we have multiple students from our boot camp and also our five-week course who have extended this, to leverage GitHub Copilot. I know a lot of companies, especially if you’re working in, like, banking or healthcare, you’re limited on what you can use, but… And you probably do use Copilot, at least. So, we haven’t developed anything, but we could probably connect you with some of our students who have. And we’re talking about building at least some kind of frameworks around that that we haven’t yet.
And then, hi, I know you’ve messed around a little bit with codecs on it.
Hai Guan: Yeah, yeah, and I think Shavi’s gonna develop a codex Repo for it.
Sravya Madipalli: Yeah, so, I raised my hand because I wanted to give a quick introduction. Hi, I’m Stravya Madipali, sorry for giving an introduction so late, I joined it right now, I had an appointment. So I, I am one of the AI Analyst Lab founders with Shane, and hi. So great question about the Codex. I’m currently building that right now, and we plan to share it in the Advanced Bootcamp. And great question about the co-pilot, too. That’s something that I want to dive as well. Have AI analysts everywhere, guys. We are in Cloud Code now, we want to get to Codex, we want to get to Copilot, and also open source.
I know there’s a bunch of people that have legal constraints in their companies, so we want to see how can we build systems that could be maybe sitting locally in your service. So, yeah, stay tuned.
Shane Butler: I did tell the… I did tell everyone that you were at the spa, Sravi, and I’ll say, you’re glowing right now, you’re glowing, you look very happy and refreshed, like, you’re not coming out of, like, meetings on, like, your, like, fifth coffee like me, like, you look like you’re having a nice, relaxing day, so thanks for.
Sravya Madipalli: Thanks for jumping in. It’s amazing. Totally recommend this. Along with a bunch of our courses, I recommend this resort as well.
Hai Guan: Wow, we’ll put on a slide, whatever the resort, shopping is. And, yeah. Yeah, running locally would be awesome. Yes, we are looking at that right now, right at this moment. So, okay, so the next prompt here, I’m just, I’m gonna give it is, can you do a root cause analysis on the payment ticket spike in June? And then it’s gonna run off and do the, Do some stuff here… And so, So, what it’s doing is, first, it’s gonna verify that whatever spike I’m giving, it can actually quantify and characterize it, and then it will drill down, and then, it will have the different dimensions that we talked about at the earlier introduction about the dataset.
Prompt, and then it will sort of reason its way to figure out the exact, you know, level 0 to end decomposition. Method, it will go in one, take a look, and then if it’s that dead end, zoom out, keep going, and all of that driven by the, the hypothesis. So, confirmed, let’s see, June payment jumped 41% of all tickets, July fell back down, crucially, attempts grew smoothly. So, it’s now reasoning on its own around, hey, this is the observation, this is my next step, this is… a way for… for us to follow along in terms of what Claude is thinking and what it’s taking in terms of the next steps and the different things that it has in its repertoire.
So again, hitting some errors, and that’s cool, it knows what to do on its own, and so it fixes it. So, seems like it’s already found some root cause, and then so it’s now building some charts around it, and then writing it up, and very soon, we will be able to take a look at what it actually comes up with. Alright, here we go. So, it is now listing some of the findings that it’s got, the root cause, defective iOS app release, version 2.3.0. So, this is pretty deep. Like, this is actually a few layers deep in terms of the, the different levels that we talked about, like, how many… how many times do you slice and dice?
It effectively has done quite a bit of, dimensions and segmentations in the different Tables that it’s got access to in order to arrive at, hey, it looks like this is the isolated incident… incidents here. it says why I’m confident, so it’s got skills to triangulate its own findings, and then figure out that it is not, you know, like, just making things up and stuff like that. It’s got the different ways for it to check itself, and so, you know, hey, it’s like, hey, I’m… here’s why I’m confident. Now, assuming this is all cool, like, we still have to check the work and stuff, the… here’s the so what. So it doesn’t just kind of, like, stop at, hey, here’s the, you know.
I answer your question, that’s it. But it keeps going in terms of, hey, what does this mean in the context of the knowledge that I have about the business that you have, of the data that I’ve already looked at, what usually would mean that we can use this information for, basically. So, so I think the, kind of, like, the… I’ll have it write it out so that we can see it even better than on, on just the screen here. So, I’m just gonna do, can you write this report along with the charts to a report in Google Doc? Actually, it outputs a chart, so I can look at this.
So it basically isolated to a, a version of the, of the iOS release shipped on this, on this date, and then the payment defect was detected off of the tickets, and then something was fixed, and then it was fixed a few days later, and so it dropped down, back to the baseline. So that’s kind of like the, you know, like, one graph that tells them all. Kind of thing, for its conclusion. So, I’m just gonna give it… can you write this report? a long… with the charts to a… I mean, yeah, spitballing, not… nothing, nothing very formal English here. So, it’s gonna… it’s gonna produce a Google Doc, and and then we’ll see, kind of, like, what that… What that looks like once this is done.
hopefully you guys are following along in terms of just, you know, I know this is pretty quick, and you might have a billion different questions around, hey, how does this work? Like, that kind of stuff. If you clone the repo yourself. Like, right now, you would be able to do the exact same thing that we’ve… that we’ve done. Export skills… Now it’s looking at MCPs. So… my… Claude code is connected to Google Docs as well, so that’s why I’m able to have it do the, do the export, and it knows exactly what to do. Now, it’s probably gonna take a little while, but I already have… the thing produced, so I’m gonna show you that. As we let it run, so that we don’t just sit here and wait for it to do.
Wait for it to… to load. Let me see… I’m gonna share the screen here. Can you guys see this? Google Doc?
Shane Butler: Yep.
Hai Guan: Cool. So this is the, kind of like, once it… once it’s done, running its thing, we should be able to see something very similar, which is, hey, here’s the root cause analysis around the June 2024 payment spike. Here’s the conclusion, here’s the impact of it. Here’s the evidence chart, the same spike that we just saw in the in Cloud Code itself, and then it goes through the different evidences of why this is an issue, why this is… why this is real, based off of the data, and why this is confined to iOS only.
It’s giving me all these detailed information around what it’s done, so the different I don’t know, severity issues, order linkage flips, so many different slices and dices that it’s able to do, and that it’s able to summarize. And so… Gave me some judgments around why this matters, recommended next steps, confidence. So, at the very top, there’s the, how confident it is in this report, it says pretty high, and here’s the itself, grading itself, in terms of Based on the stuff that it’s done, the findings that it’s looked at, why it’s… why it rates itself, pretty confident in this, in this analysis. So… You know, and sources, queries, everything is logged, that kind of stuff.
So that’s basically the demo. I know it’s pretty quick, but I just wanted to just show people where, as Shane said a little earlier, if these were done by hand, it would have taken probably quite long, versus if we systematically just prompt, for example, Claude Code, that we’re able to get to the same, you know, like, probably a much more robust and exhaustive understanding of, of what the actual root cause is.
Shane Butler: Yeah, I mean, and it’s not even just, like, the long… the length thing, it’s just, like. we wouldn’t do some of this stuff. Like, there’s just, like, stuff that I would… I just wouldn’t do. I’d be like, I’ve… I can’t stare at this screen anymore. Like, I gotta get outside and touch grass, like, I’m losing my mind doing this root cause thing. Like, what… what am I doing with my life? And now I can have a… agent do more of that. There was a question here, hi, around, If the event is driven by multiple causes, will the agent only surface the one with the biggest contribution?
Hai Guan: It will surface everything. So, things that would contribute to, like, if there are multiple causes, it doesn’t just go to the one and only, or the biggest, or anything. It would flag that, you know, hey, I’m seeing two anomalies here. Both probably need to be looked at. So it really… it’s not hard-coded in a way that’s like, hey, just find me the deepest. But really reasons itself through the, all the driving factors that could be possible.
Shane Butler: And then there was a question also around, how do we ensure our repo holds up across different models, Sana vs. Opus, as well as model updates. So model updates, we test out… so right now, like, we’re starting to test it out on Codex, with Codex. We’re starting to test it out with, some open source LLMs. We haven’t tested it out with any of the Gemini stuff yet. with all of the Anthropic models, we basically test out every model update. Like, even in our last 5-week course, Opus 4.7 came out a week before our course started, and we actually tested some stuff out, and we re… We changed some things in the repo itself.
We have, like, a plus version of this repo that our students get access to, that we updated for 4.7 because it was going off the rails a little bit, and we also re-recorded some content. So we actually don’t only update the repos, we update our course curriculum that we run every month. for the new model updates, and we’ll continue expanding that to other Frontier models and open source models as well, and maybe even something around, like, GitHub Copilot. In terms of Sonnet versus Opus, this does not work well with Sonnet. There’s probably some tasks that do work well with Sonnet, but we’ve been kind of working at this for, I don’t know, like.
maybe, like, 8 months at this point, we were, like. trying it, and then, like, it didn’t really, really click until February with Opus 4.6, where we were pretty confident. We’re like, okay, we should, like, release something open source, because it works quite well, but… Prior to that, like, 4.5 didn’t work great with Opus. Saunnet doesn’t really work that well for a lot of the stuff. I mean, like, it just works well with some things, but it’s, like, very confidently wrong. So that’s why I don’t like using it with Sonnet.
Sravya Madipalli: What? Yeah. Recurring Sonnet, what I wanted to share is, if you’re sure, like, it’s purely execution, there’s no, like, analytic… analytical thinking or reasoning, and it’s, like, executing the exact, you know, rules that you’ve put in skills, you could try using Sonnet. It’s not, like, it’s not going to basically think by itself. But for things like this, the root cause analysis, where it has to think through and, you know, question it and also, like, come up with some validation loops, you’d rather do OPUS.
Shane Butler: Yeah, anything with reasoning, yeah. Yeah.
Hai Guan: Yep.
Shane Butler: We should probably do something… I mean, we just… there’s so much… everything’s moving so fast, I’m sure everyone here always feels like they’re constantly behind. I definitely do, and so, like, there’s such a big backlog of things I want to do in terms of, like, I’d love to be able to go in there and just, like, have For every single, like, most granular task, have it assigned to, like, the cheapest model that does it. To the highest quality, but, you know, obviously haven’t had time to do that. But if anyone else wants to, and share it with us, then that’d be an awesome contribution.
Hai Guan: Yeah, that would be so cool. Alright, so just to wrap things up, you’ve seen a bit of the AI Analyst system. And we have an upcoming bootcamp just this weekend, so in 3 days, we’re gonna have a 2-day bootcamp that goes through building your own AI analyst. So, many of you may have questions around, hey, how was this built? How can I build it myself? How do I use it in my company? How do I do X, Y, and Z? How do I build on top of it? That’s exactly what we’re gonna go over. like, you can use the repo yourself, like, completely open source, free, you can just use it, figure it out on your own.
We also have a bunch of free content as well around it, but if you want to build with us to learn the… how do we design it? How do you actually do skills and agents here? How do you connect to the different tools that it could have, connect to different, context in your company, and stuff like that. Two-day bootcamp, 4 hours each day, so Saturday morning, Pacific time, Sunday morning, Pacific time, 4 hours each. You’re gonna be in a room with many different other folks who are doing this, too, to, to understand how to actually build agentic systems, and the patterns associated with it.
For attending this lightning lesson, we have a 20% off coupon code, or discount code, RCA20, and this expires tomorrow, end of day, and you know, we’ll paste in the link here as well to the actual bootcamp that you can sign up, and, you know, ping us with any questions that you may have, or anything. We would love to have you there.
Shane Butler: Yeah, and then just a couple other things. There was a question around… Unable to attend the weekend boot camps. Can I access the content outside the bootcamp? So, couple options here, and yeah, totally understand people can’t go on weekend boot camp, especially people with, like, kids. I don’t know how Savia teaches these bootcamps. She’s… she’s nuts. But, like, I have no life, like, this is just all I do, so it’s fine, like, my dog, it’s fine. But, the… We do have some people who could not make the bootcamp, but everything is recorded during the bootcamp. We release the recording on Saturday and Sunday about 2-3 hours after the session ends.
once Zoom, like, processes everything, as well as all the kind of material and resources covered. So we had a few people in our boot camp, last month just go through that async recordings, and it went really well for them. Something we’ve talked about doing is, like, doing another office hours during the weekday, following the boot camp, For people who couldn’t make it live.
Sravya Madipalli: I get access to Slack, right? Shane?
Shane Butler: Oh, yeah, yeah.
Sravya Madipalli: Yeah.
Shane Butler: Yeah, yeah, yeah.
Sravya Madipalli: We have a Slack channel for every course, and so even though you’re going to do it async, you’re going to be added to the Slack channel. You look at all questions people had, all discussions, all outputs, and learn from where people, like, you know, had questions and outputs from, and you could ask your questions there too, and we’ll reply, but the most important thing about these bootcamps, and that we are actually really excited about is the community aspect. We have people from multiple levels, from VPs of data to, like, you know, staff ICs, and they learn so much from each other by sharing with each other. So, those Slack communities and all are going to be really helpful, too.
Even though you’re async, you could just, you know, be… have a conversation there.
Shane Butler: Yeah, and then, Maybe at some point we’ll do them on weekdays. We just haven’t… haven’t done that yet, but we will definitely consider doing that, in the future as well. And then the only other thing I wanted to add, because I know, like, half the room… when we asked, like, the 012, have you never used Cloud Code before, have you used it, have you used it with analytics? There was a lot of, zeros. So, for people who, So you can join the bootcamp and never use Cloud Code before, like, that’s totally good, but if you’re, like, not ready to do, like.
a full-on boot camp, and you just kind of want to, like, get started, get it installed, because you’ve never touched it before, or you’re not familiar with the terminal, we do have, like. a pretty reduced price, like, 3-hour intro workshop. We just ran it for the first time this past weekend. Actually, there’s some people in here, like Attendee and Attendee did it, last weekend with us. It’s 50 bucks. We’re gonna do that again. We’re gonna keep doing that monthly, I think, so we’ll do that in a few weeks. I think on June 10th. So, Wednesday morning. So… it’s like 7 to 10 a.m. PT. Yeah, thanks, Attendee, for the… for the shout-out. So, yeah, it’s like a very mini boot camp.
It’s, we basically just, like. walk you through live, how do you install Cloud Code, how do you clone a repo, how do you download the data, how do you run your first, analysis? That’s pretty introductory, but if you’re kind of in the… in the… haven’t used this before and you just want to get set up, then that’s a really nice first step. Yeah, Attendee, yeah, that’d be a good one, I think, probably for you, since it’s coming up.
Hai Guan: Yep, and we’ll paste a link here as well.
Shane Butler: We were talking about?
Sravya Madipalli: what we’ll be covering in the bootcamp. We’ll also have everything that Shane shared in the, like, what we did in the intro bootcamp, so that you get everything set up, and we’ll also share with you how we built that rip-out.
Because when we ran our first bootcamp, that was the most hottest question, was like, can you help us understand what went into the code, and how you built it, and, you know, all of that, because I understand that most of you wants to take this and, you know, share it with your team, or start, you know, building and, like, you know, 10x-ing your work output, and knowing the brain behind how we got there would probably benefit you, and if you share with your teams, benefit them. So, that’s the intent of the bootcamp that’s coming this weekend.
It’s actually pretty exciting even for us, because this is our first first time sharing with the world, we actually have this information not shared with anyone yet, yeah.
Shane Butler: Yeah, so it’d be, like, walk through, like, hey, what’s the different components of a Gentic system? In the context of the analytics, what is the, like, design approach that you take to, Build each of those components, then, like, purposefully try to break it. And then identify why it broke, and then apply fixes to it. And then we’ll also do some… spend some time on, like, connecting to different MCPs, like Google and Notion and stuff like that.
Yeah, should be… it should be pretty… and of course, we’ll run… we’ll run analysis and walk through the repo itself, but I think the best value is from… People not just, like, taking someone else’s tool and using it, but knowing how to, like, build it theirself as we kind of, like, enter this… World where, like, building and execution of code becomes really cheap.
Sravya Madipalli: Attendee has a very good question. Do you want to take that, Shane?
Shane Butler: Oh, yeah, definitely not just for technical… Folks, there’s no coding, it’s all just natural language. we are gonna have different kind of, breakout rooms in there as well, so if someone is, like, more technical and is more familiar with stuff, then they could be in that breakout room. If you’re, like. Totally, a non-technical kind of person, or less technical or intermediate, there’ll be a breakout room for that, to make it approachable. And then if you’re… if you’re already building agentic systems and stuff, and, like, this is too kind of, like, intro vanilla for you, there’s an advanced bootcamp at Adichie, too.
Hai Guan: Yeah, I mean, you guys saw that there was nothing coding-related in the demo, and a lot of the pattern is gonna be like that, where once you get it set up, once you get an understanding of it, you’re just chatting with it.
Shane Butler: Hey, hi, do you want to answer Attendee’s question around, I think his question’s more around, like.
Hai Guan: Oh, fragment.
Shane Butler: Maybe the, maybe the 5-week course kind of thing.
Hai Guan: Yeah, great question, Attendee. So, bootcamps tailored towards more of building your own systems, understanding the AI components of, I would call it kind of, like, at the forefront of doing AI analytics. We have another that we didn’t really talk about here. We have another course around, a 5-week course around AI analytics… it’s called AI Analytics for Builders.
I think Shravia pasted the link here, where we really help people to become analytically independent, meaning even if you’re Maybe if you’re a data professional, or if you’re not a data professional, we give you all the foundational knowledge, the frameworks, the… best practices for how to think about different aspects of the analytical workflow, just like a root cause analysis being one of those lessons, for example, out of, I don’t know, like, 80 or 90, that we would equip everybody who takes the course With the best practices that accumulated over our collective history of being in the field, of what good looks like and what solid means in terms of analytics, and and then go through them systematically.
while delegating the actual execution to AI, just like clock code that you see. So that is more of a 5-week course where we distill as much as possible for all the different framework and things. Such that everybody can absorb and learn those, and then become really, really self-sufficient in their analytics journey. So, next time when you ask a question, or when you think of, you know, like a data… Question or something. You already know, kind of, like, what… how do you… how to best do it to make the most informed decisions.
Shane Butler: Yeah, the way I think about it, there’s always been boot camps around how do you do data science or analytics. This is your, kind of, end-to-end data science… analytical framework bootcamp, but instead of the execution layer being taught in, like, Python or RSQL, it’s being taught. through clon code.
Attendee: Thank you.
Shane Butler: I can stay over a bit. If people got more questions, I know we’ve been asking questions at the bootcamp, but also questions about root cause analysis, or the repo, or can be anything around Agentic Analytics. And if people don’t have questions, that’s fine too, but happy to stay over a little bit.
Hai Guan: Yeah. It’s the overlook, if folks are interested.
Shane Butler: They’re speechless. Alright. No questions is fine, too. But again, hit us up on, on, LinkedIn, or, feel free… if you’re in the Slack community, if you want to join the Slack community, hit us up there. Intro to optimization stuff. We’re gonna kinda go over that in our advanced… This is… this is referring to Attendee’s, question in chat. We’re gonna kind of go over that a little bit in the advanced stuff, like, some of the kind of, like. code optimization, Loop stuff, where we… where you build, like, some kind of… Evaluation scores, and then can’t have, like, the system itself kind of, like.
Optimize, what’s going on, but… It’ll probably be, like, a couple hours of that course, not a full course itself, which I think it could be. Maybe sometime in the future, though, I think it’s a good idea. I’ve been doing a lot of that at my work. But, we haven’t been teaching much of it. I do feel like with just the agents and how… how good everything’s getting, like, optimization and simulation, it’s gonna have, like. A crazy, moment in terms of, like, analytics.
Hai Guan: Globally. Okay, cool, great. Join our Slack community as well. That’s where you can find us. You can also find us on LinkedIn, you can DM us on Slack. And, yeah. Otherwise…
Shane Butler: Yeah, let me drop the… I’ll drop the invite link to Slack right now.
Hai Guan: Okay, cool. We’ll give it… We’ll go to 15 seconds, and then we’ll…
Shane Butler: Yeah, yeah, if you want to join the site who’s not in there yet? Cool.
Hai Guan: Alright, thank you everybody for spending your time with us, and hopefully we’ll see you again in a future session, or in the upcoming bootcamp.
Shane Butler: Yeah, we have another free session this Friday on insights to Action in Cloud Code, so should be a really good one that Hai is leading. As soon as we see you there.
Hai Guan: You’ll see me again on Friday. You sign up.
Shane Butler: Alright. Later, everyone.
Attendee: Thank you.
Shane Butler: Thanks.