Shane Butler: Yeah, just gave it to ya. I’m gonna… I’m gonna admit… And then… Yes, let’s start admitting. Hey, everyone! Welcome. Welcome to another, lightning lesson. So, before I get started, maybe if people don’t mind dropping in the chat… Where you’re calling in from. I’m coming in from… South Lake Tahoe, let’s look up the chat a bit. And see who we’ve got in the room. Berlin, Rhode Island. Berkshire… London. Okay, India. Nice. LA. Bay Area, Sharavia’s located in the Bay Area, too. Memphis. We’ll give it a minute while people roll in. London, Canada, okay. Budapest. I think U.S. is in the minority on this one, actually. We got people from our… Nairobi. I don’t know if I’ve been from Nairobi before.
Milan? Italy.
Sravya Madipalli: Hello?
Shane Butler: Hey, Savia.
Sravya Madipalli: Okay, sorry, I have an issue with my computer all of a sudden, so I had to…
Shane Butler: No worries.
Sravya Madipalli: Yeah.
Shane Butler: Brazil… Okay, we’ve got quite the spread today. Nice, here comes Hi. Hi is in the room.
Sravya Madipalli: Nice.
Shane Butler: Alright, are we ready? Should we start?
Sravya Madipalli: Yeah, give me one minute, maybe we could just take a couple questions, and… Or… if you can get started on what this is about, I had my entire computer, for some reason, reboot for.
Shane Butler: Yeah, no worries. So, today, and Shravia’s gonna go, through this in more depth. This, today’s session is around, basically cutting wasted analysis time, so, You know, a lot of, I mean, any job, really, but I think especially in the data field, it’s so easy to go down different rabbit holes. It really depends a lot how your data team’s set up. You know, it’s not… It’s not, as easy if you’re, like, embedded in a team, in a product that you’re working with, and so you have a lot of context around the problem you’re solving day to day, like if you’re going to do the daily stand-ups, the weekly planning meetings with, like, your, you know, engineer and PM and designer, everyone.
But a lot of data teams aren’t set up that way. A lot of folks are, you know, understaffed in the data org, so maybe you’re covering multiple teams. Maybe you’re more of a kind of, like, a service-oriented centralized model, and that makes understanding the context of the problem really challenging, and… Questions can come in that actually seem, like, pretty concrete and pretty specific around how you’re gonna answer them, and it’s really easy to spend, you know.
not days, but I mean, like, weeks, doing an analysis, even communicating with your stakeholder that whole time, but it’s like you’re just kind of talking past each other, or, like, speaking different languages, but you feel like you’re actually saying the same thing, and then you come through with the analysis, and you get some coordinates of, like, oh, this is, like. interesting, but this isn’t really what I’m looking for, or like, thank you, like, I’ll take a look at this, but then nothing ever happens. A lot of that can be solved. Upstream, by just, you know, reframing questions, answering the right… asking the right questions, getting the right context.
And something I’ve been finding lately is, you know, there’s a lot of talk, and I’m not just finding it around, like, this kind of reframing questions things, but anything around, like, rigor, around experimentation, around, like… checking myself on if I’m looking at spurious kind of random relationships. There’s all this talk around, like, you know, LLM is, like, being, you know, hallucinating and, kind of being biased, but as humans, like, we’re very biased as well, so I actually do find.
there’s a lot of opportunity when we build Gentex systems ourselves to kind of just, like, put checks and balances on our own bias and our own thinking, so we can have guardrails and make sure, we start our analysis from the very beginning in the right direction. So, strawberries, I’m talking about that a bit today.
Sravya Madipalli: Anyone see my screen?
Shane Butler: Yeah, we can see your… I can see your screen. Can you all… can you maybe drop a yes or no in the, chat, but I can see it. Okay. Nice, we got some thumbs up.
Sravya Madipalli: Yeah, I’m on this new computer, and I’m currently doing Zoom on web browser. I realized I didn’t have the app until, like, last minute, so… Sorry, folks, if there’s some technical difficulties, but thank you so much, Shane. We could do a… I could do a quick intro about myself. Shane did go over a bunch of it, but Stravio Madipali, I, am the co-founder of this, along with Shane, and hi. We lead the AI Analyst Lab here. I have experience of around 14, 15 years now in data science, starting with Microsoft, And most recently, it’s Superhuman, which is Grammarly. So, have been into data science in Silicon Valley for a long, long time right now.
Okay, so, like Shane said, we are basically going to look into the most I would say the first question, the first thing that we want to do when data or analysis comes into place, right? So, we run, like, a bunch of courses. We run Cloud Code Analytics Bootcamp course, we run, 5-week frameworks of data science course called AI Analytics for Builders. And the first thing that we teach in that… that we taught in that course, we had one cohort out, and the first thing we taught in that course is about framing your questions right, because if that’s not correct, guys.
If you’re going after the wrong problem, the entire work that you’re gonna, that you spent your bandwidth on is not going to be held, like, you know, it’s just utter waste. So, and we knew about the importance of this, but to be honest, until our students actually picked, that’s the biggest thing that they learned, we were… You know, like, super surprised. So, this was something that we did as part of our course cohort discussions, where we asked people what was the single most valuable thing they learned? The most popular answers was, that, you know, pushback on the question, and actually reframing the question.
Because… UC, especially, it happened with me as well, when I started my work in data, we are used… used to getting questions from so many different stakeholders for so many different reasons, and do… this is, I would say, more relationship, more about data, like, it’s a combination of so many skill sets, not purely data skill set, and But that totally affects the impact of you as an individual that you could bring to the, you know, to the company, wherever you are. So, that’s the reason we believe this is one of the most important skill sets. This is beyond technical. This is, I would say, in the frameworks and thinking, analytical thought process, and also your personality type.
Like, we are so used to saying yes for everything. Now, this I hopefully reframes your thinking like it did to most of our students. And, like, trust me, they all had, like, AI analytics, like, a repo of 60 skills and agents, like, they had all of that with them. That was also very important that, that people really liked that, but This, which is, like, the most basic thing, came out as, like, the top one that people write. Okay, so where are we? Let’s talk about, like. making this whole thing concrete, right? So, let’s say we have same AI, same data, but we have two tabs, right? So, we have a vague question and a sharp question, and what is the difference between both of them, right?
So, if you look at a vague question, analyze user engagement. You know, these are some things that you probably keep seeing a bunch of people, like. your stakeholders come and ask you, hey, what’s the user engagement, you know? And this is basically… could be so many things when we talk about user engagement, right? So, it… it gives everything, but also nothing. It basically is, like. three paragraphs of generic observations, because, like, DAO is related to stable, but some segments are worth monitoring, nothing you can act on, so this is, like, what a vague question can, you know, encompass. It could mean so many things. Versus what a sharp question looks like.
Which segment drove the engagement drop last month, so we can decide where to focus? So you can actually see there are particular frame, like, you know, segments of this question that you can divide and understand what each of them mean. For example, there’s a named segment. Right? Because it’s talking about a specific area, a specific user group, where you might have seen the drop, and then it is also talking about it being a quantified driver, like, understanding the what behind it, and it also has, like, a clear recommendation. For example. what would we do if we know the answer for it? Which is THE most important thing.
that I would say, especially people who are, you know, just starting out, or people who are used to the surveys based of data types of, you know, data, service data-based orgs. we are not used to this setup at all, where we are, where we basically take a wake questioning, do all of the, you know, the complicated backing ourselves, and finally end up at a place that we could have gotten to far more, like, you know, in a far more shorter time, with far more clarity. But, you know, we basically didn’t start there, and hence… It took so much. So, so, so, so long than when compared to, like, without it. And guess what? This is something that’s not just at work.
This is something that you’ll see in multiple places. I myself have, like, you know, we just, until now, spoke about how whatever we’re talking about, question framing, like, changes how you work, right? For example, can you look at why signups are down, and then you spend a day on the charts, and then, hey, this is not what I meant, I meant something else, you know? This is something that we see on our day-to-day work. But guess what, where this is more important? This is actually, one of our students mentioned this too, on how this question framing helped them, like, crack, you know, get better at interviewing.
Which is, if you go to interviews, like, if you look at what metas, like, data analytics questions are, they’re pretty, they start very vague as well. Like, how would you measure success of this feature? It doesn’t… basically cover any of the specifics, and it’s upon you, as an, you know, interviewee, how you’re taking this question, and how you’re segmenting it down into multiple segments, and how you’re actually actually tying a metric to it, and how you’re actually ensuring that you ask the interviewer the right questions about, hey. what are we trying to understand with answering this question? How are we going to execute on this, right?
So these things are something that interviewers look for as well, not… so this skill set that you’re going to learn today, it’s just going to go way beyond your work. And another very interesting piece, that totally, you know, came together in today’s world is AI. So you might have seen this a lot. This is what, you could call, like, context engineering, you could call so many things, where if you give AI something vague. it has a lot of scope to cook things up, right? You could call it, I mean, if you don’t give it enough information, it’s going to basically assume a bunch of information, right? It’s kind of like, you know, an intern you work with that doesn’t have all the information.
So, ensuring that you’re specific, and you probably have, maybe, already a system, like AI Analyst, system that we have, like a repo full of you know, skills and agents that we built, even our free repo has them, guys. Shane can, like, share the free repo, where we actually build the system where if you ask a question to Clawcore. It actually tells you, hey, this is too vague, you know? Can you get more specific? So you could build an AI system that does that for you, so that you don’t have to be the person that’s, you know, telling it every time to be specific.
If not, you better be specific, because the more vaguer, vague our questions are, the harder it would be for The, you know, foot claw code to actually understand and help you solve the problem in the right way. And this is where we could… we basically talk about the exact specifics of what I’ve been talking about, right? So this is basically a three-check framework. You could call it, like, you know, a 30-second gut check, right? So, every analytical question should pass all these three questions. What are those? The first one is decision time. I would say this is the most important one. Which basically talks about what decision does this question inform.
If you can’t name a decision, you stop right there. this is probably the hardest as well. Imagine if you have a VP of product coming to you and ask a question, and you are like, hey, this is pretty, like, you know, thanks for this question. Could you please tell what will my answer for this question change what you and your team’s working on? Or how do you want to take this answer and implement something? Because at the end of the day, if you’re working on something that doesn’t prove to be impactful, it’s not going to help For anything. It’s not going to help for your work that you do at your place, it’s not going to help you get promoted, it’s not going to help you, you know?
If you want to interview, what did you work, and what’s the impact? You’re not going to have a full closed loop, right? And the second most important thing in this framework is being data-grounded. Can I actually answer this with data that exists? I know this sounds pretty basic, but I’ll be honest, there’s so many questions that come to us for data teams, where the data is not entirely fully present, or half present, or, like, not fully baked, and does it make sense to answer this question with the amount of data that we have? Or do we just wait, go implement, get the infrastructure, talk to engineers, get the tracking implemented, and then come back to it maybe next quarter?
instead of giving half-baked answers that are not going to help you make that decision that you probably, you know, pushed your stakeholder to give you, right? Like, what’s the decision? They’re going to come up with a decision, but guess what? If you don’t have data, or if you have half-baked data, it doesn’t help. And the third most important thing in this framework is being specific enough. How will I know, or how will you know, basically, when you’re done? Is there a metric, a comparison, a time frame? You know? to make sure that you have some sort of a bound. Scope creep is the most important thing, the most, you know, thing that weighs anyone down.
So ensuring that all of this is under the scope is also very important. Yeah. So, before we spend an hour analyzing, spending 30 seconds here, to understand, does my question actually satisfy these three rules? Is the decision tied? Is the data grounded? And is it specific enough? Now, let’s go to the next one. Okay, so let’s actually watch one get sharp, right? So let’s start big. Why are signups done? No decision down from what? No, we don’t have for whom? We basically have it really vague around saying, no, like, you know, no decision, like, we can’t basically make out anything from this question, right? So, how can we make it slightly more specific? Why did signups drop 12% last month?
So it’s better, better than the last time, the first question, right? We have a number now, but what would you actually do with the answer? So the decision is still unclear, right? Now, we can make it even more sharper. Did the pricing page redesign cause the 12% sign-up drop in the two weeks post-launch? And the decision comes. Should we roll it back, right? So this basically has the decision, which is rolling back if it’s not great. Yes, we got that frame… that part of thing figured out. Now, what about data? The data, is a data grounder. It is, because we are looking for sign-ups, plus… and we also have a launch date, and we have all the specific things. So, that’s data groundedness.
And what about the specificity? We have things specifically saying that attribution, and we also have recommendation. So, this is basically, I would say, the most minimal thing where we start, to ensure that we go from a vague question to actually a sharper question. Awesome. So, do you… so that’s… this is something that you would probably have something at work, but what about… like, I have an example for interviews as well, right? So… this let’s say an interviewer asks this. How would you measure success of the new search feature? this is a pretty common question, guys.
Like, if you go to any of data scientists, or, like, product analytics, or maybe even data product analyst questions, you would find a metric or a measuring success type of question, right? And most candidates, especially, like, from my experience, the more junior ones even more, they immediately start listing the metrics, saying, hey, DAO, retention, engagement, I’ll probably look into this, that, and that, right? So, yeah, that maybe feels productive. But it’s actually the trap. The candidates who get offers, and actually offers for higher, you know, level roles, they actually pause and ask clarifying questions first. Success for whom, right?
Because we are talking about success for this usage field, but who is it? Are we talking about? And ensuring that, is it power users or everyone? So, we need to have a Tier 1 set of questions, and then a Tire 2 set of questions under it. So, question like, success for whom, is a Tier 1 question. And a Tier 2 question under that is, what segment of users? Is it power users or for everyone. And then, another set of questions that you could ask is, measured against what is the baseline, right?
So this basically also… this framework around ensuring that the specificity, and ensuring that you have data groundedness, and ensuring that there’s a decision tied to it, it’s also going to super, like, you know, it’s going to be super helpful for you for interviews as well. So, the question that could be pretty, you know, targeted, that you could start work on the interview, is once you get, the clarifying questions answered, and get to a question, something like like what it says at the bottom. For search targeting power users, what’s the primary success metric, the guardrail, and what would be good enough to keep investing? Like, you know, a look-like at the end of 30 days, right?
Awesome. So, a sharp question usually carries a hunch. It has a hypothesis. The most important thing that we need as data people. Irrespective of, you know, it doesn’t… you don’t have to be a data person. Irrespective of, your role, if you work with data, you need to come with a hypothesis, because all we’re trying to do is either validate or invalidate the hypothesis, and we have multiple ways to do that, right? We have A-B testing, we have causal inference, we have correlation analysis we could You know, validate to some extent. But what we need is ensuring that we need to have that hypothesis.
And the move that separates rigorous analysts from everyone else is that you basically have a rejection condition as well. What would confirm the hypothesis, and what would reject the hypothesis, right? So before you touch the data, you need to basically say it out loud what result would prove you wrong. Here’s the mid-market churn, you know, is driven by failed onboarding, not pricing. what would confirm it? You could say something like, hey, John clustered in accounts that never finished setup. You know, pricing absent from exit interviews. that these are some things that would confirm it, but what would reject it?
Chance spread evenly across onboarding states, pricing dominates exit interviews. So, this is something that would reject this hypothesis. So, it is very important, to ensure that you have a hypothesis and you have pre-written rules on what confirms it and what rejects it, right? Now, another very important thing, is not every question deserves the same effort. So score each one on two axes. you have impact and you have feasibility, right? So, top left is basically, something that you do first. It basically is a question that clearly drives a decision, and the data already exists. That’s where you start, every single time. But what about the top right, the second part of the metrics?
It’s like, basically, you plan for it. High impact, but you’re missing the data, or you’re missing instrumentation. So you probably need to, you know, stake it out for the next quarter. Okay? And what is the, like, you know, leftmost one? This is something that you would use it for buffers. You basically do if you have time. This is basically something that are quick wins, and that are dashboards. You basically ensure that when your bigger work is blocked, you have something to fill your bandwidth with. Right? And the most important one of all is the skip one, the bottom, you know, rightmost corner in the bottom, right?
This is the most important thing, because a lot of asks, so many data scientists and analysts I’ve worked with, that needs to be skipped actually falls under one of the other categories that you don’t want it to happen, right? So, it’s basically hard and uninteresting. This is the graveyard of half you know, finish side quests, right? Ensuring that you say no early, And ensuring that you pinpoint to why you’re saying now, right? That it’s low impact, it’s low feasible, you don’t have data grounded, there’s no specific decision you could make.
And these are some things that you need to ensure that you write down, and talk with the stakeholder, and share with the stakeholder, and give them a reason for why you’re skipping it. But If they don’t want you to skip it, they better come with those answers related to the type of decision they’re going to make, if you give the answer, and they… let’s say it’s a product manager, they ensure that the data groundedness happens, they work with the engineers to have the tracking ready, and, you know, get everything done. Without it, all of those go to skip requests. Okay, so… where are we at time?
So, we probably… I have a few notes as well, I’ll share this, with you, but I’d probably go to the demo instead of going through this, because I have a good… something good to share in the demo. Any, any questions, any discussion so far, before I go to the demo?
Shane Butler: I think a couple questions. One’s around, do you define the metric yourself, or do you rely on the stakeholder? And then there’s another one that, maybe Hai can jump in on, because you can answer it in the chat, but what about in the case of ETA-type analysis?
Sravya Madipalli: Okay. I mean, the metric question, guys, it’s… defining metrics and coming up with… we actually have… Kai actually shared, how do you define metric? We have, like, an entire segment in our course that talks about it, but it’s the most important part as well, because all these steps that come at the start of your work If they’re defined wrong, then your entire work goes, you know, haywire. To ensure that you come up with a framework of what a metric means, and what is the type of metric you want to, like, you know, how do you want to define it?
And you work with the product partners, or the decision makers, whoever it is, into understand the goal of the metric, why you want the metric, and what is the decisions that it’s going to help change in the product. Once you get those decisions from the product partner, I think it’s the data person that comes up. And accepts and, you know, verifies and, like, validates the metric. But you want really tight partnership with your decision makers and product partners to ensure that they are on board. And they know what they’re going against, because you don’t want to be vague.
You need to be very sure about the specifics, and as a data person, you need to come in with the, you know, the caveats and frameworks around it, because you don’t want the metric to be too specific in a certain area. You want… also, it doesn’t have to be too vague, so high up, that it doesn’t help make any decisions, right? So as a data person, you bring that perspective. And as a product, or finance, or whoever is the partner that you’re working with, they need to bring their perspective of the goals that they are trying to achieve with that metric. So, that would be my answer for the metric one, but yeah. And hi!
Shane Butler: We have a… we have a session next week, actually, that’s free. It’s called Developing North Star Metrics with AI. So if you wanna… if anyone wants to join that, I’ve just dropped a link in the chat, and if you can’t make it, but if you sign up, we’ll get you the recording that for free, as well. There’s actually another question in here. This is a little off-topic, but not off-topic. Are you guys already building agentic systems for self-serve analytics at your companies? what is the expectation now from analytics folks? So I think there’s two directions for this.
There’s the, Agentic Analytics, where the data team is leveraging… is building their own Agentic systems to, either automate out their work in such a way that is more robust, or, faster, so increasing breadth and depth of the analysis, but the agentic system, the analyses… the analyses are still done by the data team, but it’s just through, you know, the agentic systems they build more than just, like. direct SQL, Python, R, whatever.
That’s something that we’ve, been doing for several months, that all of our students have been doing for, several months, and we’re seeing this as kind of, like, the new, I would say, expectation for data professionals is moving from doing analysis yourself to building agentic systems that Do a lot of that work for you, where it makes sense, and then validating, that analyses. The other side of the coin is, like, building self-serve.
analytics systems for, you know, the rest of the company, other orgs, where they’re actually, like, a, you know, like a… someone from the marketing department has a question, they’re able to type it in in some sort of centralized, authentic tool, and have the analysis done themselves. There’s a ton of enterprise solutions working on this, right? You know, companies that have, like, hundreds of millions of dollars in funding trying to solve This problem… this is a much more complicated problem because they cannot validate the answer themselves.
In the case of, you know, a data professional building out a Gentic system and running the analysis, they can look at the numbers, they can look at the code, and they can say, like, yeah, this is right or wrong, they can review it, just like a software engineer would review the code that, Like, just a coding tool would output. So we focus on the former and less on the latter, although we are kind of in talks with a few partnerships with some of those enterprise solutions, and we’re happy to talk more about that offline or one of our courses.
Sravya Madipalli: Awesome.
Shane Butler: Maybe I’ll let you… there’s some other questions here, but I’ll let you… maybe, like, if you want to keep going, then we can get to these.
Sravya Madipalli: Yeah, we can get at the end, sure, awesome. Okay, so I’m gonna give you a bunch of, like, because you signed up, guys, because you signed up for this, yeah, I’m gonna give you a bunch of, you know. free PDF resources as well, that contains all this information that I shared and a lot more. For example, like, how do you come up with the right question, give you a bunch of examples of, some questions that you could ask in your like, you know, interviews, if there’s a question that comes up, what is the type of example that you need to, you know, ask and stuff? So, for example, yeah. So, definitely check them out. I’m gonna send them over an email. Okay, what are we doing right now?
I’m gonna have two demos, hopefully time permits. So, the first one is going to be in Cloud Web UI. I’m sure, if you join our… a bunch of other courses, we… like, the Intro to Cloud Code, we… you could do all of this in Cloud Code. My second demo would look at that, but since, if you don’t have Cloud Code, or if you, have not used it with our April, this is something what you could do with Cloud Web UI, okay? I’m sure this is something that you could do even with ChatGPT as well. Okay? So, this is a prompt I gave it here, and what does it say? It basically tells that you are a question quality coach. Your job is not to answer questions or do analysis.
Your job is to help me turn a vague question into a sharp, answerable. And, you know, something but fast, but coaching me through the few targeted questions, right? I gave it 3 modes, I gave it a work mode. Basically, if this is a question related to your work, and I gave it an interview mode, let’s say if this is a question that you were asked in an interview, and you’re prepping for interviews, right? This is something that the interview mode would help you, and there’s a rank mode. Let’s say you have a bunch of options, and there’s, like, a vague question. What… how do you rank all different questions?
And then, it basically has step one, running the three-check framework that I just shared with you, and step two is something that talks about how does this framework work for interviews, and then how does the rank mode works, right? And then, it talks about synthesizing it in step four. So, begin… and look at this. Begin by asking for my way question, request or problem, and whether it’s a work situation or interview situation. and the problem you should rank framings for. So this is all the entire input, I’m giving it to Cloud Web UI, and let me ask it a question and see what it gives as an answer, okay?
Shane Butler: Can you drop the prompt in the chat, too, Stravi, if people are asking?
Sravya Madipalli: Okay, let me just do that, so that…
Shane Butler: And someone asked if they can transform this into a skill.
Sravya Madipalli: Oh, yeah. That’s a wonderful question. This is something that, you know, we have a free repo. I’m sure Shane might have shared it, we’ll share it again in the email, but you could look through our free repo, and look through, like, the skills and agents that we already built, right? But if that’s something that you need help with, we actually have the Intro to Cloud Core Analytics workshop where you could, you know. work with us. But yes, these are some things that you can definitely build skills and agents for. Cool. So, I gave you that, and I’m giving you my simpler question as well. Okay? Yeah, so what’s my question?
Now that I gave it the entire prompt, I’m asking it, it’s a work situation. My VP just slacked me, hey, can you look into our chore numbers? They seem high, and leadership is asking about it. Seems familiar, at least familiar for me. There’s so many of these type of questions that keep coming, right? So what does it say? It says, running the 3-check framework on looking into our churn numbers. Guess what does it say? Decision type. No, there’s no decision in this question, so it crosses it out. Is it data grounded? Mmm, kinda. Churn is measurable, but high versus what? So it’s not also fully data-grounded. So that’s kinda a maybe. It’s not a fully now, but it’s almost a maybe.
And what about specific enough? No metric, no segment, no time frame. So, and also, there is no definition of what done means, because you could just keep answering this question for weeks and weeks, and still not get to the answer that The requester is asking for, right? So, this fails, 2 out of the three, let’s… and it talks about, let’s close the biggest gap first. So, it is asking, if you got a clear answer on churn, what would leadership actually do differently? And it also asks for examples. Fund a retention initiative? Does it change pricing? Reallocate to the growth team? just report a number upward. So, what is this decision feeding, right?
So, it’s asking me the question, and let’s give it an answer. So… I just gave it. You could… guys, this is something that I’m doing… you could have something entirely different that you could work with, okay? So, what I’m giving it is figuring out whether it’s one segment we need to fix, or it’s a broad problem. Basically, where do I point the retention team? So yeah, there is a decision now. You could try the same thing with another answer and see what you get, and you know, you could share that with us. But I gave it this, to demo something. Okay, so it says, good, this is a real decision. Where to point a retention team that makes this a resource allocation question, right?
Not a reporting exercise, which is pretty good. Now, it is talking about the next gap being data. So, the first gap, the decision tide is done, now we are looking at data. Which of these exists cleanly? Plant tier, acquisition channel, tenure, cohort, oh my god, we could have so many things, right? So, let me give it a few things, let’s say, that I kind of have. I’m saying, I’ve got plan type, region, tenure, monthly revenue, and last login per account. Now it says it’s a solid one, right? So it’s basically asking, how do you know you’re done, right? The final one.
And let’s say I’m gonna give it, I basically, end this when I get a ranked view of which segment is churning the worst, so I know where to focus. So that’s where I’m gonna end it. Awesome. So, Heath is now giving me the sharpened versions of the same question. Look at this. Because you could have multiple styles, right? And it’s giving me all of those styles. It’s giving me a concentrated view, and it’s recommending that. It’s basically giving me which segments, the plan, tenure, region, and stuff that I could look at. It is giving me a rate versus volume view. It is giving me a trend view. And these are a bunch of options.
So it’s asking if, like, you know, what is something that, you want out of these things, right? I mean, I could keep going until I get, but I want to get to the Cloud Code demo as well. Or maybe let’s do one more thing. So, you could go on with this until you get to a specific question, but I want to try one more thing for the interview question, and then go to Claude Code, okay? Okay, let me just say. Okay, let’s… Move on to… Interview question. Okay? And this one’s an interview question. How would you measure the impact of a new recommendation algorithm? Right?
So, it is basically doing the same thing, it goes to the interview mode, it is not in the, you know, it is not in your, the work mode. And look at this, it is asking about clarifying metric, understanding context, planning the cuts, confirming the decision. trust me, guys, I’ve done, like, at least hundreds of interviews, and… you would expect people to do this, but no, people actually don’t do this at all, especially confirming the decision. So, you want to ensure that you do this 2-minute decomposition of this question live with the interviewer, and keep asking them for, do they agree? Do they want you to go down this direction or not?
And hopefully this prompt helps you to do some mock interviews yourself. Okay, so let me move on to Cloud Code, and I’d like to share you a Cloud Code demo. Okay? Let’s try to do it in the next 5 minutes, so that we’ll have around 15 minutes to… for answering any questions that you have. Okay, so where am I? What are you looking at? You’re looking at Claude Code, and this is visual code, VS Code. We actually teach you, in one of our courses, how to, you know, install all of this yourself. And, you know, have, how do you have the repo? So, the repo that I have on my left is the… is our entire repo that we built all of this.
For example, if you look at the repo, for the bootcamp, this is the repo that you have for bootcamp. These are all the skills that we have, analysis design, you know, design spec, and, you know, all of that stuff, but… I’ll get into the detail later. Let’s jump into the Clark Core prompt. Okay, so, what am I asking here? I’m asking it, you are the question framing layer of an AI analyst system. Like I told you, we have so many of these skills, the analysis design, you know, archive analysis, like, you know, so many of these, and I’m asking it.
Instead, if I have a vague request, like, look into our churn numbers, they seem high, and leadership is asking, what is the process that the Cloud Code repo that we built is going to go through, right? And this is an ASCII diagram of everything that Cloud Code runs for you. Let’s say you have this question, you have this repo at your hand, right? Look at this. Gate 0. This is, like, the first gate that it looks at. Three checks. Decision tied, fail. Nope, it’s not great, like we said. Is it data grounded? Fail. The churn numbers? Which table? Which churn? No metric is named. And what about the specific enough? Again, a failure, right?
So, it basically is giving me a score of 0 by 3, and it basically is telling me that you have to… I’m rejecting it, and we are going to, you know, you need sharpening it, right? And in case… I asked for the entire ASCII diagram for you to understand what all things the system is capable of doing, so that you’d understand to get to a specific question. Let’s say we get to a better place, and what does the skill pipeline look like, right? So, this is invoked to sh… and there’s another thing. All of these get invoked to sharpen the question too, right? So, question framing, sharpening. So, it basically helps you to get to the right question. And what does it ask you?
It basically asks you, hey, what is your goal? Is your goal retention? Is your goal revenue? Or is it just a broad narrative, right? And what is the decision you want to do? What changes If churn is high. And what does high mean? Is it related to the target, trend, or a cohort? And who decides and by when? This skill is the most important skill, the question framing skill that goes through the three-spec framework that we have. And ensuring that the question ladder gets done, right? And once that’s done, it basically goes to the metric spec. This is basically the data groundedness, right? So what is the churn type? I know, what are the multiple steps here? Within the… what’s the denominator?
Who is at risk? Is it, like, a voluntary one? Is it an involuntary one? Basically, all multiple segments that you get to within, like, understanding what the churn metric actually means. And what are the multiple segments you can divide this churn metric by, right? And then, there is data audit. You want to make sure that that particular data is actually available to query, right? And you want to make sure that, you know, the data is clean, that everything is reliable, and you can actually go and query it. Because specificity is not the only thing you need. You actually need data that you can rely on to actually go and execute, right?
And then, another important thing is actually, now that you know all of these things, how do you want to prioritize your question? What are the other things that are at stake? What is the amount of dollars that’s at stake? What is the, you know, like, you know, what is the stop rule? When do you think you need… you’ll stop this? And is it worth the return on investment? Like, the amount of return that you’d get on this investment, is it worth your time or not? Right? And these are basically the forks that go… that we go through, like the metric, baseline, window, segmentation, the ranking logic, the threshold, the unit, the output decision.
All of these is baked in into the skills and agents in the system. Most of this, I would say, is already part of the free repo, but our AI Analyst Plus repo that we share as part of our courses has even more detailed description into, like, what, like, one… if you ask it a vague question, how can the repo help you get to a better place? Yeah, so this is how it looks like, and one final thing I want to share with you is, I want to ask something. Oh, where are we? Maybe I can skip it so that we’ll have 15 minutes. Is that… What do you want to do, Shane?
Shane Butler: I’ll go ahead and ask your question, and then we can… we can see if people have questions. If people have questions, feel free to drop them in the chat now, and we’ll get to them after… Yeah. Soravia goes through this.
Sravya Madipalli: So, it basically went through, the question, right? And it basically gave the exact sharpened question that, I know we didn’t give all these answers, but because for the demo’s purpose, I basically took something based on a few assumptions, and basically has a sharpened question for us, right? Has monthly voluntary logo churn risen about 3% target in the last two quarters, and if so, which customer segment is it? by plan tier and tenure. And is that driving the increase? And so, leadership can decide whether to fund a retention intervention this quarter or not. So, this is the sharp-end question that solves all these forks.
Like you see here, the metric, the baseline, the window, the segmentation, the ranking logic, the threshold, the unit, which is customers. like, you know, accounts, seats, and, you know, whatnot, and the output or decision, right? And it also ensures that all the checks are done, the decision type, the data groundedness is specific enough. And the most important thing, the rejection condition, if you remember, we talked about being… having a hypothesis, and ensuring that when do we reject the hypothesis should also be, you know, like, documented and tracked and thought of.
And if voluntary logo churn is less than 3%, the target across both quarters and no single segment, that’s when you say that the report is not elevated, and recommend no spend, and we stop. So we have a rejection condition here, which is very important, right? And then, we have the data scope note. What is the source? What it excludes? What is the grain, and what is… do we do if… if, you know, if you’re blocked? Yeah. So it goes through all of this, and it gives you this. And this, guys, is just… just the first part of the system. If, maybe I have something here to share with you. Oh, so, if you want to understand the entire system.
Of what actually happens, now that you have a question that we basically share in our courses, I can briefly share with you while we… we could probably keep taking questions while I run this, maybe. Shane.
Shane Butler: Yeah, I think something to emphasize about this is, like, like, reframing questions, like… It sounds easy. It’s just hard. There’s two, kind of. there’s obvious things when, like, a vague question comes in, you’re like, oh, that’s super vague.
And then there’s, like, questions that come in where, like, you kind of don’t realize it’s vague, but it actually is, because it doesn’t have everything here, like, it doesn’t have… you know, I’ve started plenty of analysis in the past, and be like, oh, I have biometric, I have, like, the kind of decision that’s gonna be made from it, but then, like, there’s all these other components that are on, like, oh, like, actually, what window are we talking about? Oh, what are the filters that need to be applied? be applied? What’s, like, the threshold we care about? And I go back to my product manager, or designer, engineer, and everyone’s like, oh yeah, I don’t know, what about… what do you think?
What about this? And, like, this just, like, gets everything done. Upfront for you. And it’s not that it’s, like. You know, like, there’s no math here. There’s no, like, crazy, like, sophisticated, like, thinking. It’s just, like, all these things that we don’t have time to do, or kind of, like, forget to do, and now it’s, like, every single analysis you do, every single little question, you can run it through this system. It’s gonna take a few minutes, and you’re, like. Question is gonna be… become so much more powerful, and you’re gonna get a, like, like, I think Shrabi was showing the slides earlier, like, a decision around, like, hey, do we kick the analysis off?
Do we need to, like, plan more? Do we just, like, put this in, like, a backlog to do later? Or do we, like, politely decline the analysis? So, it kind of guarantees up front that, you know, you go from Just, like, these are fake numbers, but, like, imagine, like, 80% of the analyses you do, don’t actually have action come from it. By adding these gates at the very beginning, that becomes, like, drastically smaller, and you get a much larger percent of your analyses, and the time that you spend actually being, actioned on in the company.
And that means… that means that, like, yeah, the company’s doing better, that means you’re invited to more… more important rooms around, like, how do we make decisions around the company? That means you get the promotion, right? So, it’s a really, like. not crazy challenging skill to learn, but it’s so important, and so many people miss it, and it’s not just junior people. I know senior managers of data science who suck at this, who are just like, oh yeah, oh, we should track WOW, and it’s just like. No, dude.
Like, people who, like, who focus their whole thing around, like, reporting a metric and not actually relating the metric back to, like, the actual reason we’re here to build some sort of user value of a product. I know CEOs who suck at this, I know, like, leaders in product who suck at this, so it’s not just, like, a junior thing, also. And now you can have systems where you just, like, build it in to, like, automate out that, like, those errors. Yeah, so Shravia, you’re sharing here, like, the full system here, and you can see, like.
The sharpening the question is just, like, the very first part of this kind of end-to-end, the GenTech system that we’ve built, and that our, like, our students and clients have, like, built on top of, like, way more than us.
Sravya Madipalli: And then you start with input, and then there’s a stage one, you basically scope and plan, and there’s a data audit that happens, and then you actual… now that… now execution starts. you basically measure, and what exactly happens there, and you diagnose, right? If, let’s say, you find something in the measurement, how do you diagnose that? And then you stress test it, and then you close the loop. There’s so many other things as well. I tried to, you know, get it to… so this is what happens.
So, it basically, if you run, like, run analysis pipeline, these are… a bunch of things that get, you know, called, the agents and skills that get called as well, which question framing, hypothesis, data explorer, you know, source tie out. The descriptive analytics, the root cause investigation, so you’ll see all of them here on the side.
These are my skills, these are the repo skills that we have here, the deck critic, if you look at it, the data quality check, comparing data sets, closing the loop, so we have… Basically, you could invoke all of these skills by themselves if you want to stay in control, and if you want to ensure that what is the output of this particular stage of my data analysis, which I would recommend 80% of the time. But let’s say you’ve done this loop multiple times, and you know the type of question you have and the output you need, then in those cases, I just run the end-to-end pipeline that basically goes through all of these skills, one after the other, and gives me an entire pipeline.
And since I’ve done this so many times and validated every step already, I’m confident that my final end-to-end pipeline is a good one. So, just wanted to share with you the capability of the entire system here. And it, to be honest, doesn’t cover other areas as well, like experimentation and tracking. That’s going to be a different set of systems and skills that get generated. But yeah, I’ll stop here.
Shane Butler: Yeah, we can pop out a few questions, so if you got questions, drop them in the chat, We can raise your hand and ask them as well. Two questions that are basically the same question here, So, Attendee asked, like, what is the difference, between our three courses? Yeah, they said, they’re working as a product analytics in their current org, their goal is to fully integrate agentic AI workflows and tools into their work, and bonus if it helps in interviews. And then another person asked. hey, the repo looks really great, what would we learn from the paid sessions? I can answer these. So we have 3… we have, 4 courses, actually, technically. So we have… and they’re all coming up really soon.
In the next few weeks, a bunch are kicking off. They’re recently ranked, Maven’s top 100 courses, also, and so Maven is running a site-wide sale on their top courses, where it’s 25% off, I think, throughout… through the end of, this week, so that stops on Sunday. That’s better than any deal we get. We usually get, like, 20% or 10% off, but, just something to flag. So there’s four courses, there’s one. I’ll say what they are, and then I’ll go through them. Intro to Cloud Code Workshop, Cloud Code Analytics Workshop, there’s Cloud Code Analytics Bootcamp. Advanced AI Analytics Bootcamp, and then AI Analytics for Builders. So our Intro to Cloud Code Analytics Workshop is super introductory.
If you have used Cloud Code before, you probably don’t need to take this. If you’ve never… this is for people who really have never used Cloud Code before, It’s like… Our, like, introductory course, it’s, like, it’s 50 bucks, and we spend 3 hours, and we just walk through, step by step, how to install Cloud Code in Terminal, install get VS Code going, you clone the repo, and then you run your first analysis end-to-end on some synthetic data using the repo.
So, this is open for anyone, there’s no prerequisites, like, you can never have coded before in your life, but we do find that, like, that setup stage is challenging for some people, and once you get past that first step, you’re kind of off to the races, like, you’re all, like, curious, smart, driven people, so we’re… this is, like, just our goal, like, hey, let’s get people setting up… set up and running their first announced end. So that’s coming up next Wednesday, on June 10th. 7 a.m. to 10 a.m. Pacific time. And then we’ll record it also. If you sign up, we’ll send out the recording. We have a bunch of people who just followed along with recording.
And there’s, like, also a step-by-step, like, guide you can walk through as well. So that’s the… that’s the workshop. Super introductory, though. But great for a first step into Cloud Code. Then we run a Cloud Code Analytics Bootcamp. not this weekend, but next weekend we’ll have that. So it’s Saturday and Sunday, and then we have a bit of async material following through the week that you can, as you start working with Cloud Code Analytics at work, and then we have an office hours that’s optional on the following Friday after you’ve ran through the Agentic system at work for a week.
This is about building Agentic analytics systems, so it’s about, you know, learning the foundations of Agentic systems, how they differ from chat. You start by building out your own skills and agents, then you run some analysis with those, then you, implement our, like, full AI Analyst Plus repo, which is kind of our private repo we’re constantly developing on. You, add, you port over skills to that from your… from what you built your own, you build on top of that, you connect to data warehouses, and then we do some, kind of, like, 101 previews into stuff around, like. Context management and, validation. And open source.
But, like, those are, like, more, like, just, like, introductory parts of it. Our advanced bootcamp then goes all in on that, so our advanced bootcamp is focused on how do you build evaluation systems and validation systems to know that the Agent Tech Analytics output is correct, how do you manage context across your team, and organize it even, like. for yourself, how do you… how does this perform against other models? So this Cloud Code Analytics Bootcamp is just for Cloud Code. Advanced AI analytics, we go into codecs, we look at open source models as well. And then… so that’s another two-day boot camp. We might extend it.
We’re experimenting a little bit with the boot camps right now, so we have a two-day with some async, and then office hours for the Cloud Code Analytics Bootcamp. in a weekend from… in a week from now. And then July, we’re actually going to move this and try it on weekdays. So we’ll do, like, just 2 hours a day, Monday to Friday, instead of 4 to 5 hours a day, Saturday and Sunday. So that’s an option for you. We… we reiterate on our courses every time we have a cohort to try and make them better, so that’s some of the feedback we got back. And then our last one is AI Analytics for Builders. That’s a 5-week course, and this is more about, like. end-to-end.
analytical thinking, it’s less about building the agentic system, and more about, like, how do I use, how do I use Agentic systems to do the actual analysis work? And so, you could think of this as kind of like your… classic data science analytics course, except the execution layer is no longer in SQL or in Python directly, it’s in, it’s in Cloud Code or Codex. So, we teach, we start… the first, week is all around Kind of, like, framing questions, The second week is all about, like, setting up your tools and your system. Third week is about, correlative analysis, so think, like, trend analysis, segmentation, root cause analysis. Fourth week is around causals, so think experiment.
Design and analysis and causal inference, and then fifth week is around, kind of, like, getting insight to impact, so it’s about, presentation, managing stakeholder, and that kind of thing. Yes, those are our four courses. And like I mentioned, all of those are 25% off right now on Maven, until Sunday, I think, is when they… when they, And that, but so if you just go to the course. You don’t even… the promo code’s TOP100, I think, or Maven100, It’ll be auto-applied since it’s a site-wide sale at the moment. And then, if you miss it this week and you want to go on later on, usually we offer some, you know, if you need a promo code, DM me.
Usually we give, like, 10-20% off folks who come to the Lightning lessons. Alright, that was a pretty salesy question-answer, so let’s… what are some other… Questions Oh. Attendee was asking, my post about moving from tech into acting. No, that’s just a joke. I’m not going into acting. That’d be pretty good. There’s plenty of people who get into Hollywood later in life, so maybe I’ll do that in my 70s. No, I am leaving my full-time job, though.
I’m currently principal data scientist for a legal AI firm, or a legal AI tech company, and tomorrow’s my last day as we go full into… Agentic analytics research, education, and consulting, so the, you know, leading, co-founding this lab was Shravi and Hai. No acting yet, but who knows? What other questions we got? 4 point, Attendee? Have you guys noticed a difference between 4.7 and 4.8 for analytics? Man, I feel like it kind of topped out at 4.6, and I’ve read a few people who leverage it for more, like, product managing stuff and less for coding stuff. And their opinion, I don’t know if you listen to, like, Clairvo’s podcast, but I think she said something like.
like… as the 4.7 and 4.8 models got better at coding, it got worse at strategic thinking, and I kind of think the same thing for analytics. I often just try and set my model back to 4.6 because it does a great job with analytics, and it’s faster and cheaper. But I don’t know, 4.7, 4.8, I don’t think it, like, deteriorates the analysis, but I’m not getting much of an added benefit. So, whenever a new model comes out, we test them. And then, also, OpenAI, they just released a data analysis, flow this past week, too. I was actually talking to some folks on the Codex team there, and so we’ll be… Investigating that quite a bit.
We don’t want to just… we’ve kind of, like, primarily been focused on cloud code the past few months, but now that we have some more time, we’ll be investing on that. And then also open source LLMs. Hi has been… investigating that, at least I’ll be in our advanced bootcamp. But we’ll probably do some sort of free session on it. Haven’t tried Codex Mobile Attendee. how Tharbright enables self-service data? I’m like, oh yeah, I’ll have to check this out. Thanks for sharing, Attendee.
Sravya Madipalli: Yeah, that’s cool.
Shane Butler: The self-service stuff’s so interesting. I mean, it’s just, like, there’s not a great solution out there for even any of the enterprise stuff around, like, how do you validate that the number’s correct? Like, Snowflake… Cortex is amazing. It’s… it writes great SQL and stuff, but there’s, like, there’s not really a great system of, like, saying, like, hey, is this number correct or not? like, when I talk to my intellect engineers, I’m just like, hey, how do you know this is, like, correct? And their answer is usually like, oh, I think it should be if we build this table out.
And it’s like, man, I’ve built, like, really good semantic views where I’m sure no one can screw up the number, and then I give it to, like, someone in the C-suite, and… they’ve figured out some creative way to screw up the number. I’m like, why did you ask it that? So, like, we focus a lot in our… advanced course on, like, validation. It’s kind of why we’re leaning more into, like, empowering people who, like, are checking the numbers themselves.
Otherwise, you end up in this weird situation where, like, Like, you’re just in this, like, reviewer mode, where, like, say, like, one of your stakeholders runs an analysis and then sends it to you, and then you just have to, like, go through the whole trace of the conversation they had with the agent and check the number anyways, and it would have been just faster if you ran it yourself. We’ve talked about OpenClaw a bit, but we haven’t done anything… anything there. Any other questions? I know we’re over. I have, like. 5-10 more minutes if we have other questions, though.
Sravya Madipalli: Attendee.
Shane Butler: Tomorrow, we’re gonna do… We have some… a bunch of stuff coming up in the next two weeks that’s free. That I can just drop links in for it now. So tomorrow, we have one on Agentec experiment design. and analysis? This is free. We’re actually working on a separate repo called AgentXP. It’s, like, pretty, like, early… stage right now, but it’ll be an open source thing, kind of like the AI analyst, but it’s, like, purely focused on… experimentation, and the goal of it is to remove the bias that the humans have on experimentation, because it’s like, with experiments, I don’t know how many of you have ran experiments. I ran a lot, and it’s kind of like you’re grading your own homework.
like, the people like me who’s building the… building the product, building the feature, is the same people who’s creating the experiment and then reading the results, so it’s like, you really want it to be good. You’re, like, so incentivized to, like, find some part of this experiment that was a success, and so, this… this repo, it’s not necessarily around, like, the instrumentation of the experiment itself, it’s more around, like, how do we set up and automate guardrails so that, like, we’re not biasing the results, and we can, make sure that’s like, oh yeah, this is truly a successor or not. Kind of like how we did the question framing here.
So we’ll have that, and then Savi is leading another really cool one next week called Publish Analysis Everywhere with Cloud Code. This is about using, MCPs to get your analysis output in, like, several different, platforms. This is gonna be really cool. I think that’s one of, like, the really powerful… parts of this system is, like, the MCP, work, and then next… and then I talked about this other one already, which kind of goes hand-in-hand with this question frame one, is develop, Northstar metrics with AI, and so this will be… Well, today was around, like, question framing, this will involve some of that, but it will be around how do you, like. Create really rock-solid metrics.
That actually reflect, like, the user value of what you’re trying to build. So those are all free sessions, check those out, and… what was the reason behind keeping this open source? Yeah, I mean, my philosophy is now, it’s like. Like, there’s a couple reasons. Like, one, like, I think, like, building is so cheap now that it’s kind of… a little pointless to, like, try and, like, build product and put it behind a paywall for some things. It’s, like. you guys… anyone here can go build the AI analyst, like, you know, like, we teach a boot camp on it, where it’s like. we believe in, like, this are a few days, people should have, like, the skills it takes to build it themselves, so it’s like.
That’s one reason. And then the other reason, I think, is just, like. since, like, everything is, like, changing and developing so fast, like, I just think the best way to learn is for… is having, kind of, some sort of open source community where we’re all kind of, like, testing stuff. A lot of the really, like, I don’t know, valuable parts of the bootcamps and the five-week course for me is all the live time with all of everyone who joins, because people take what they learn and then immediately use it at work that week and find out way more better ways to implement it than I would, and then we take that and we share out across the… across everyone.
So it’s like… it’s… we’re entering this field in energetic analytics where it’s not like, hey, like, there’s decades and decades of stats, best practices that you can just kind of read books on, or, like, people just have, like, the expertise and know what it is. It’s like. I don’t know if this is the best way to do it. I don’t know what the best way to do it is going to be in a month or two from now. So I think it’s really important for people to kind of, like, be, like, learning in public. Like, I think you can go, like. fast alone, but, like, way further together, right? So that’s, like, the reason behind open source. Open source is also a nice hedge, right?
If I screw things up, it’s like, hey, man, it’s free. Like, you didn’t pay money for that. What other questions? I got a couple minutes, but we can also close out.
Sravya Madipalli: And I’ll send you, a bunch of PDFs, guys. I’ll be sending you 3 PDFs, with some prompts, and, you know, and later, as soon as we have a recording, the recording as well.
Shane Butler: Cool, yeah, so check out that Maven 100 deal, I think it runs till June 7, whenever Sunday is. they give that 25% off. It’s not just our course, it’s, like, all the top courses on Maven. So, definitely would recommend checking out Maven, in the next 3 days, because there’s other stuff on there, too, if you’re not looking for analytics. That you can get a really good deal on right now. Alright. Thanks, everyone. Hopefully we’ll see some of you tomorrow at the experiment… the Agentic Experiment Design and Analysis One.
Sravya Madipalli: Get your questions there as well. We’ll mostly have, like, the last 10-15 minutes like we did today, okay?
Shane Butler: Yep.
Sravya Madipalli: Thank you so much attending. Bye, guys.
Shane Butler: Zip.