Sravya Madipalli: Okay. Hello, hello!
Hai Guan: Hello, everybody. Welcome, welcome.
Sravya Madipalli: Welcome, everyone!
Hai Guan: Welcome. Welcome to this week’s, exciting… Lightning lesson. Or I don’t know how exciting it is. Not as exciting as Xavi is.
Sravya Madipalli: No, I think it was… it is exciting. So, everyone, as you start coming in, could you please share where you’re joining us from? Would love to know. I’m sharing where I’m joining from. Boston, nice! San Jose! Okay. Yeah.
Hai Guan: Still admitting people to the room, so we’re gonna get started in a little bit. Yeah, if you guys all can just drop in the chat where you’re from. We would love to know. What do I disable? Waiting room now.
Sravya Madipalli: Nice, nice, that’s really cool. Hello, hello! Hello, peoples! Awesome, so many from Canada, Europe, India… California, Texas, Washington, D.C, nice! Paris, wow! What… what’s the time right now? Oh my god, it’s like… It’s pretty late for some of you, so… We’re glad that, you know, you’re coming to us at these odd times.
Hai Guan: Yeah, thank you for… thank you for joining us. Lots of, very international, so we’re gonna get started in just a little bit. Shravia, I made you a co-host, so you can also admit people.
Sravya Madipalli: Yes, yes, I will do that, thank you.
Hai Guan: And then I will get us started here. Let me see… K… Can you all see my screen?
Sravya Madipalli: Yes, yes, we can.
Hai Guan: Cool, okay. Alright. So, yes, welcome everyone, welcome to this, lightning lesson around opportunity sizing in Cloud Code. My name is Hai, I work at a legal tech company called ENTRE, and, I was previously, Worked up. data science teams at, LinkedIn, Nextdoor, Pinterest, Meta, all the big consumer tech companies, and, Nextdoor was actually where I met, my co-hosts, today, Shravia and, and Shane. Shravia, do you want to introduce yourself a little bit?
Sravya Madipalli: Sure. Hello, everyone! I’m Stravia Madipali. I lead the growth data science team at Superhuman. It was previously called Grammarly. I have around 14, 15 years of experience now. I majorly worked at Microsoft for 6 years, moved to the Bay Area, I was in Seattle, Redmond, moved to the Bay, worked at eBay and Nextdoor, and like Kai said. Shane Hai and I met at Nextdoor, and we want to do something in the world of data, and here we are. We started with a podcast, guys. You have to check out the podcast, from the past. We also keep sharing some of these sessions later as well there, so if you want to check out any old sessions, definitely check that out.
And AIAnalystlab.ai is where you would find everything that’s related to current workshops, future workshops, courses, and whatnot. Yeah.
Hai Guan: Yup. Cool, awesome. Shane is on the road today, so it’s just gonna be me and Sravya talking, so if you have any questions, or if in between you have thoughts or comments, we’ll drop links as well in the chat. Feel free to just, you know, like, put things, put things in the chat. Cool, so as Shavya said, we are from AI Analyst Lab, so this is a, almost like an open source community, where maybe many of you have seen, sort of like, we open-sourced.
a repo called AI Analyst, and that’s fully open source, fully free, that, effectively turns Claude and Claude Code into a senior product data scientist, who is very well versed in how to do, sort of, like, analytics in Silicon Valley, let’s call it, but it extends to many different, areas, so… Today, we’re excited to talk about, one component, in the AI Analyst, repo to sort of, like, hit on this topic on opportunity sizing. In parallel, we also run courses, so we have one that’s coming up actually next Monday on AI Analytics for Builders. This is a 5-week course.
where we teach people, on the, kind of, like, the analytical thinking and the foundations of how to become analytically very sufficient in your day-to-day job. So that’s… think of it as almost like a, with execution delegated to AI. So, think of it almost as, like, a, like a condensed version of, all our years of experience working in this field. between myself, Shravia, and Shane. So, combined to be probably over 50 years of, of… of experience here. Yeah, that sounds kind of crazy, but that’s, that’s what we distilled into. And then we also run a bootcamp, like a weekend boot camp on, building AI analysts yourself.
So, we ran our first cohort just a few… Just a few, no, a couple weeks ago, or a week ago, something like that. Very, very soon, and, or, very recently, and, I think maybe some of you have taken it, and it was really fun. So the idea was really to get people to actually building, and then, seeing for themselves, kind of, like, what they’re capable of in terms of building an agentic system in the context of, kind of, like, the AI analysts. So if you guys are interested in that, we’ll drop links Throughout, and, let’s get started. Cool, okay. So, has… before I get into this, does everyone know what opportunity sizing even means? Drop it in the chat. Just a quick yes-no.
Sravya Madipalli: Yeah, just a yes-no.
Hai Guan: Okay, Attendee says, I do a lot of time. Nice.
Sravya Madipalli: Awesome. Time to tum.
Hai Guan: Attendee, yup, no. No. Okay.
Sravya Madipalli: Nice.
Hai Guan: Cool, okay.
Sravya Madipalli: That’s a good mix, some of you. Yes? No?
Hai Guan: I gotta bring her remittance. Love it, Attendee, love it. Cool. Yeah, so, yeah, so just to define it a little bit, opportunity sizing really is, like, hey, if we were to work on X, how big of an addressable market are we serving? Meaning, like, how do we translate that into something that, that’s of a common language, usually down to the impact. Basically, it’s like, you know, hey, if I were to do this, what is the impact quantified? So I think the, kind of like, in, on, on, on this slide, there’s a… I think Attendee kinda… kinda hit on it. Most of your opportunity sizing is a… almost like a main-up number, at least for people that I worked with.
I mean, like, you know, different people are different in terms of how they, how they, how rigorous they are, in terms of the sizing opportunity. But many of them is actually a, you know, let’s call it, like, a made-up number, or like a pie-in-the-sky sort of number, on a slide deck. So, on the left, many of the sizings could look like, something like, you know, pick an optimistic assumption. And then multiply it by a very big total addressable market. And then, put it in a slide, and then hope nobody asks, kind of, like, where the numbers come from. Like, nobody asks you to decompose, like, you know, hey, what’s your assumption, and whatever it is.
So typically, you’ll see that in the form of, hey, this is a multi-million dollar opportunity, or this is a $10 million opportunity, something like that, right? On the right, which is where we want to get to in terms of being rigorous and being… thoughtful and thorough in how we’re thinking about these things is that, it should always be a formula that separates, kind of, like, what’s assumed, and then what is the actual math that gets you to, like, a believable number.
And then all you’re doing is checking the inputs, like, figuring out which inputs are grounded, which ones are assumed, that then, you know, you can always debate and, like, have, have judgments around what it should be, and things like that. So… Kind of, like, on the right side is what we’re trying to get to, in this… in this lightning lesson, and that’s part of some of the skills and agents that we encoded into the AI Analyst repo. Cool. Okay, cool. So, when we boil it down to, like, you know, hey, what does that actually even mean? It’s pretty simple, right? Like, it’s, you know, think of it as, like, one formula, just three inputs, and that’s it.
You land at the impact at the very end of this formula. So, the first one is, stated very simply. It’s like, how many users or transactions are actually in scope? that we’re affecting. So, if you were to… Make a decision on doing something, you probably would know, like, what is the, the subset, or what is the audience that you’re hitting that would benefit from the change that you’re thinking about making. So that’s one, just, like, the baseline number for what that, what, what that number is. And generally, this you can calculate, right? Like, you can pull from a user base, you can count how many, how many actions, or whatever it would be, it would be, it would be affected.
And then number two is the rate of improvement. Like, what do you believe could be possible To, to actually, drive the change in that number that you’re looking to move. So, you know, this could be anywhere from, like, 0 to 100%, or, you know, even, even higher. Like, for example, like, hey, I want my… I want to double my conversion rate, or something, on, on some sort of, on some sort of product flows. That’s totally fine. And, you know. mostly, you just have to figure out, kind of like, you know, either based on experience, based on industry benchmarks, based on what’s believable, what that improvement rate could be from a, from a sort of, like, a, you know, like, not a made-up sort of way.
So you can always ground it, As well in prior context. But that’s the second component here. And then number 3 is the value per unit. We call it the value per unit, so… If you could translate what you’re doing into dollars, what would that be, in terms of every single unit that you are able to sort of, like, lift in the improvement? And so, you know, pretty, pretty, pretty straightforward. If you go… if you just multiply through this whole formula, then effectively you have your impact dollar amount.
And then what you’re doing, effectively, is, like, looking at these three components, and then figure out how to get data To kind of, like, plug into this thing, and perhaps even construct a range of possibilities, you know, given, given the movements of each, like, given the range of each one of these. So, impact formula, users affected times improvement rates times value per unit. That’s, hopefully that is, that, that is, that is pretty straightforward to folks. Okay, cool.
So let’s walk through some of, like, the, good versus bad examples of each of these, and then, and then we’ll get into some, some other examples, and then, at the end, or towards the end, we’ll get into kind of, like, the AI component, Cloud Code. So, the first one is users affected. The bad example, which actually I hear a lot myself, perhaps some of you might have heard it in your day-to-day as well, it could sound something like, hey, all our users would benefit, or something like, you know, hey, a big portion of our population would be affected.
I hear that all the time, especially coming from, you know, like, for example, some of, Perhaps marketing teams, or, you know, like, teams that aren’t, used to, sort of, like, thinking in terms of numbers. What a good example could be is, like, hey, we have 12,400 monthly mobile users who started to check out Flow, but don’t complete. Like, that’s really concrete. That is grounded in data. So, like, literally, the difference is extremely vague versus extremely specific. Same thing for the second row here, if you look at it, improvement rates. I… the bad example here is, we think conversion will double, like, I mean, you know, totally fine, like, we can… we can think what we think.
So, you know, that’s never wrong. The better believable part of it is something like, hey. mobile checkout conversion rate is 33%. Desktop is 52%. So, do you think it’s plausible to close half of that gap? Which would get us to 42%. Like, that is a lot more, concrete in terms of what the range of possibility is, and, like, if you pick a number between this, that would be way more believable than, let’s say, like, hey, conversion rate will just double, so it’s, like, 100% or whatever. That’s just not how it works. And then the third component here is the value per unit. So, bad example could be each conversion is worth a lot. So the question is, what does a lot mean?
And, you know, again, like, in terms of thinking in the range of possible numbers. a lot does not give you anything specific. So, so always think, like, you know, like, on the right, which is a good example, is that… hey, today, our average order value is 67%, or $67 at 75% margin. So the, per conversion, it’s worth $50, something like that, so… Now, if you just kind of, like, multiply these three together, and then each one of these, you have the confidence, just like what we just went through, then probably it’s going to be a lot more convincing whenever you present your case for why we should or should not invest in certain Improvement sets, or feature sets. Does that make sense to people?
Cool. I saw in the chat, someone says, would accept… oh, would not accept, okay. I have a heart attack for a little bit. I thought someone says they… their CFO would have… would accept worth a lot.
Sravya Madipalli: Also, anyone… Please, you know, give your comments or any questions in the chat as well. I can, I can keep answering, or I can, like, whenever Hai pauses, we can take any of the, like, important questions. We do have Q&A at the end, though, but just saying.
Hai Guan: Cool. Yep, awesome. So, you know, coming back to another question here, or I guess another scenario here, let’s call it, like… so, imagine you have, like, an e-commerce company, and we’re gonna get into an example company like that in our Clock Code section. session as well, in that, think like Amazon, and then you have a feature for save or later, so it’s something like, you know, hey, I don’t want to buy now. But I don’t want to pick again, in the future for the stuff that I’m interested in. So, like, you know, I have a safer feature later, and it’s a $5 million opportunity. And then the pitch here is, lots of users browse but don’t buy.
If we add the save for later, more of them will come back and purchase, and this could be huge. Sounds pretty reasonable on the surface, right? Now, because you’ve seen, sort of, like, you know, hey, how can we be concrete in the previous couple slides, that is actually what that actually adds to the credibility of, like, you know, hey, if we can improve it in a numbers-grounded way, then this would become even more effective in terms of the pitch. So, how would you do that? you know, just some, just some, some, some examples here. Again, if you look at those three components, what is the users affected on this, in, in this sentence? It is extremely vague.
The users affected here is lots of users, so, you know, like, not… that’s not very helpful, and probably people aren’t going to be patient about, you know, listening to pitches like this. The fix is be concrete. So, 28,000 monthly users who viewed 3-plus products, but don’t add to cart. Like, that is extremely specific, very concrete, in terms of the addressable market, like, right off the bat. Second one, on the improvement rates, again, like, very vague here, more of them, like, we can get more of them to come back. Well… what happens if you attach a number to it, like a believable number?
So, it could be 8-12% recovery rate, which comes from an industry benchmark, so you’re not making that number up. It is grounded in something, some sources. That would be a lot more effective than the, Than the, than just, like, you know, hey, more of them, more, like, it’ll be… it’ll be great. And then the value per unit, the vague part is, could be huge. Yeah, anything could be huge, or it could be small, could be anything. the way that you add credibility to it is, again, attach a number to it that is grounded in some sort of experience or data that you already have. So, $45 average first purchase from recovered users.
That… that itself would just complete this whole… this whole, whole flow, and make this pitch a lot more concrete for, for whoever the recipient of the pitch is. Hopefully that makes sense. Okay, maybe drop me in chat. A or B? Which one do you think is better? Well, I mean… well, yeah, I’ll let you judge. A or B. Oh, well, I gave it away. Who’s gonna say B? Or who’s gonna say A? B is better, B is better, B is better. So this is a trick question, guys. It’s not B is better, because B doesn’t have a pitch. I’m just kidding. It’s, Yeah, I forgot to… forgot to actually make a pitch out of it, but the idea is there, like, I think you guys got it.
It’s, you know, instead of… Like, no trail, like, instead of a pitch that’s non-traceable. like, think of it that way. Have something very specific that every single assumption could be discussed or could be debated, by different people, and then as long as you present your first version of how you arrive at that, at the beginning, that’s gonna be almost like setting the anchor for what the recipient would ultimately probably, you know, like, latch onto, if that makes sense, from a psychological perspective. So, you know, like, the more that you can Ground your things, or ground your… ground your opportunity size in as many verifiable sources as you can, then the better. Okay. Cool.
Okay, so a quick plug. What you just saw is one lesson out of, I believe we have 100 lessons in AI Analytics for Builders. This is our 5-week Full course around, kind of, like, developing the fundamentals and the foundation of what Good analytical thinking and judgment. should be, and, while delegating, sort of, like, the execution of analysis of the analytical workflow to, to AI. So, this is, yeah, one out of 100, I think, you know, plus or minus 10, let’s call it that. But we have a lot of framework, we have a lot of practices, and we have a lot of, a bunch of stuff In it that starts next week. So, next Monday is our day one of, of, of this course.
you guys, if you are interested in signing up, 30% off for attending the lightning lesson, and what you’re gonna learn in that course is framing questions, using Cloud Code as your AI analyst, defining metrics. debugging funnels and root causes, and then, almost like, you know, like, design experiments, size opportunities, storytelling, how to convince people based on the outputs, how do you drive decisions, things like that. So lots of framework, again, encoded based off of our 50 years of experience. We’re giving our age away, but that’s cool.
Sravya Madipalli: And one thing I want to add to that is you’ll get the repo. We have a free repo right now that you could use, like, you all should definitely check that out. I’ll share the free repo link as well. But you’ll get an advanced version of that repo, guys, and it is… pretty mind-blowing, the amount of things that the repo could help you literally do today. So, to understand that, you should check out our free repo once, I’ll share the link right now, but if you’re part of this course or the… any other course, we’re basically continuing to add what we are learning into that repo. So, currently, it has up to 50 skills and, you know, 50 agents.
They take any workflow and make that AI workflow, so you’d get that repo, you could take that repo, plug into your work, and start using it. So, that’s something that you’d get as part of this course, too.
Hai Guan: Yup. And, yeah, and we open sourced, sort of, like, the AI Analyst repo, as well. That’s, that’s free, just not as powerful as the, kind of, like, the one that we share in the course, but you can also feel free to check that out as well. It’s, totally accessible. Okay, cool. So, let’s see, so you learned a formula, here’s a prompt that can help you check, kind of, like, you know, what’s missing whenever you need to, or whenever you get, kind of, like, a proposal or a pitch from someone else, like, what, what should, what should be included.
So… I can… let me just… I’ll share this with all of you, like, I’ll send it out, later, after the course, along with the recordings, but, like, you know, before we get into Cloud Code, this is kind of, like, a very simple prompt that, all of you, and we’ll… we’ll actually look at it together, to… Kind of like, to, to see how that works. So… let’s see… do you guys see my Clawed web?
Sravya Madipalli: Yes, we can see you.
Hai Guan: Cool, alright. So, I am going to paste. In… So, okay. Before… before I get there, let’s… let’s analyze what this prompt… well, let’s look at what this prompt does. So… this is, like, defining Claude to be like, hey, you’re a senior product analyst reviewing opportunity sizing, And the opportunity is, and we’ll paste something in here, like, whatever pitch you get, that’s sort of like the opportunity, and then evaluate it against each of the components that we just talked about, right? And then for each component, give me a rating of, is it clear, is it vague, or is it missing? And rewrite the opportunity with specific numbers and why you would Flag it the way that you did.
So, roughly, that’s kind of like the setup for what this prompt does. So, you can fire up your favorite chatbot, anything, like, you can fire this in Cloud Cowork, ChatGPT, Gemini, anything. So, I just have Cloud open, so I pasted it in this guy, same thing. Formatting is a little weird, but that’s okay, so… You remember, there’s a role, there’s an opportunity, and then, this, this whole thing. So let’s just see what happens. Let’s see, so a question about why Claude and not ChatGPT, you can paste in any chatbot you want, like, literally anything. Grok, Gemini, anything, so this would just work. Okay, cool. So, let’s see. So, it basically evaluated this opportunity pitch that we gave it.
we should improve our mobile checkout experience, it’s clunky, and we’re losing sales. Like, I didn’t give anything else, just this line. So, it says, okay, in the components of users affected, that’s super vague. So, we’re losing sales means nothing about scope, who are these users, and stuff like that. So it gives you the critique around why it’s… why it’s vague, and then how you can fix it. So, concrete estimate to plug in for a mid-size company e-commerce sites, assume 500K. So a lot of these are assumptions, but, like, it gives you an idea how to sort of, like, plug in the numbers yourself, and why it’s doing the judgment that it’s doing.
Number two, improvement rate is missing, so clunky is a vibe… clunky is a vibe, not a benchmark, according to Claude, so there’s no definition for what clunky means, but here’s how you can anchor it. For example, internal, you can have a conversion rate on desktop versus mobile. experiments, if you have run any prior A-B tests around checkouts, industry benchmark, these are all… these all look very reasonable, and is exactly kind of like what, most people are probably gonna anchor themselves on. And then, again, number 3 here, value per unit, that is also vague. Losing sales implies revenue, but doesn’t quantify it.
No average order value, no margin, no LTV, long-term value consideration, stuff like that. And then it says, okay, for a really good… pitch, here’s… here’s a full… well, that’s a very long one. Here’s a full, sort of, rewrite, that gets extremely specific about each of these components. So, feel free to play around with it. I’ll send it… I’ll send it all… send it out to all of you guys, and you know, plug it into any chatbot, and the next time when you hear something about opportunity sizing, especially Attendee, you can plug it in here, and it’ll tell you… it’ll tell them what they need to come back with.
Sravya Madipalli: Hi, a couple of questions, do we take them quickly now? So, the first one is, why is the role important in prompt? Attendee asked this, and Attendee gave a very good response, but maybe a quick, something quick.
Hai Guan: Oh, okay, cool. Yeah, so LLMs generally are sort of like a world knowledge, think of it that way, and if you ground it at the beginning of what their role is, it almost makes them very specialized right from the beginning. So it goes from, hey, I know everything, I’m very generalist, to like, okay, let me look straight into just this slice of, of role that you’re assigning me as. So, you know, definitely try it out and see what the differences are. That’s generally been sort of, like, the prompting technique from the very beginning of LLMs.
Attendee: The question I have was… the same question could be coming from a finance person, could be using it. It could be a product manager, a senior product manager, everyone does opportunity sizing. So if you say that only the product analyst will give you this answer, a finance person answer will be different, that doesn’t make any sense.
Hai Guan: Yeah, I mean, I could see that. I could see, their answer should be the same. Actually, that’s a really good idea. Like, if we try it out on having it take on a different persona, it’ll be interesting to see what that looks like.
Sravya Madipalli: So, nothing…
Attendee: Can I do it now? Maybe change a…
Hai Guan: I do want to show you guys Cloud Code. So, this one is more like a… almost like a starter, to… so that everyone has access to the, the chatbot.
Sravya Madipalli: So, Attendee, you’ll get access to all these, you know, prompts, you could definitely try it out. Like Hai said, you will see a different persona. So, as soon as… let’s say you’re a junior analyst or senior analyst, you’ll have slightly more nuance and flavor to it. I would probably think, as an LLM, that’s what that would do as well. So, if it’s a deterministic answer, then probably the exact number would slightly be the same, but the flavor in which they share it It’s probably going to be different. And the reason why we give personas is for LLMs to make it… to make the scope down as easy as possible, so that it thinks in that direction of a product analyst, right?
But product analyst does have product and analyst in it, so it could pick up things like a product manager tool. But yeah. You can ask these questions later as well. There’s a Slack community, I’ll share the Slack community, Attendee, so you could join there and ask these questions, and we’ll try to answer them too, okay?
Attendee: Okay, sounds good, because I’m a director level, and I want to understand, is there a difference now coming around here? If I’m gonna ask my team to use the LLMs. So people would have been asking different questions with different levels, and At the end, you have to have the same answer. You can’t have an answer coming from an analyst one number, a product number.
Sravya Madipalli: Absolutely. You hit on a very important topic, that’s literally what Hai and I, we do as part of our work, too. So the validating systems, and how do we ensure that the answers are deterministic? So that’s a big topic that we could discuss, maybe in the Q&A, or it could be in the Slack channel, Attendee. I can share the link, okay?
Attendee: That sounds good.
Sravya Madipalli: Thank you. Yeah.
Hai Guan: Alright, cool. Thank you for that question. So, let’s, let’s get into Claude Code. Let me find where my thing is… share… This one. Alright, cool. Can you guys see my screen here? Am I…
Sravya Madipalli: Yes, hon.
Hai Guan: Do you see a black… yeah, okay, cool. So, I am in… for those who are familiar with, IDEs and text editors and Cloud Code in Terminal, this is my setup. This is VS Code. So, think of VS Code, for those who are not aware, VS Code is, one of the IDEs out there. So, on the left, I have file explorers, folder, file, folder explorers. On the bottom, I have my terminal. And then, on the top, I can… preview any of the files and see what actually is inside. So this is my setup for Cloud Code. We go into it in the courses and bootcamps and things like that, but if you use Cloud Code yourself, find whatever setup that makes… that’s best suited for your use case and for how you feel comfortable doing it.
This is the way that we all are pretty comfortable with. So that’s sort of, like, the intro to what we’re doing here. And because our people… did people follow the news yesterday? Opus… Claude Opus 4.7 dropped, right? because of that, I’m actually gonna do something very different here, which I’m gonna have Opus 4.6 and Opus 4.7 side by side, so we all… we can all take a look at, kind of, like, what, what the differences would be, given the same sort of, setup.
Let’s see… so, and for those who are not aware, when Opus 4.6 came out, that really crossed the chasm around the analytical capability of AI slash Cloud that makes a lot of the AI Analyst repo, you know, skills, agents, and its ability to become, like, a senior product data scientist possible. So, you know, if you sort of extrapolate all of that, then 4.7 is only going to become more capable with the harness of AI Analyst. So, on the repo here, we have AI Analyst Plus. This is the souped-up version version of the AI analyst, that you, that you see in the open source, repo.
So this one has gotten a lot more, you know, like, more agents, so there’s a lot of these, for example, like, deck builders, Google Slide reviewers, and things like that. It’s got more, a lot more skills. And for those who are not aware, skills and agents, agents think of it as, like, you know, it can go and do stuff. Skills would almost be, like, the recipe for how things should be done, for example. So the combination of these things make Sort of like a complete harness for this repo to act like a senior data scientist. Okay. Enough of that. I’m gonna fire up off side by side, so I’m gonna split my terminal. You guys see a left and a right side, right? Okay, I’m gonna fire up Claude.
I’ve never done this before, so we’ll learn together what… what it’s gonna look like. Oh no, what? That can’t be the case. Yeah, okay. Yeah, it’s the same computer, so… Awful. Cool. So, I have side-by-side. Right now, as you can see, I’m in Opus, 4.7 here. So, I’m gonna make the right side… Let’s say, model… Oh, I can’t go back to 4.6. How do I switch to Opus 4.6? So, I don’t know how to switch back to Opus 4.6, since it’s not available, so I’m asking Claude to do it for me. So run model and pick Opus 4.6, or type model focus 4.6 directly, so I’m just gonna do that. gold… Whoa.
Sravya Madipalli: Interesting. I was trying to do this. So this, this is something that we, basically, get for if we try to do things live, and…
Hai Guan: Oh, no. Okay. Huh, okay, well, I guess we don’t have 4.6. We only have Opus 4.7 now.
Sravya Madipalli: They absolutely replaced it. We had it in the morning.
Hai Guan: I guess, quad and fropping move too fast. Okay. Anyway, okay, well, I guess not. Then, let’s just look at 4.7. Let me see… I’m gonna exit this one. Go terminal. Okay, so over here, just a little bit of a setup. We have in the AI Analyst Plus repo, we have a, set of or I guess, a bunch of data… or, like a dataset, that is based off of a fictional company called Nova Mart. This is actually what we teach in the course and the bootcamp as well. Think of it as the thing that I just talked about. probably 10, 20 minutes ago. This is a fictional e-commerce company, think Amazon, so there’s, you know, like, you buy stuff, you check out, and things like that.
So it’s got access to this, this data set here. So, I am going to… let’s see, walk you guys through, step by step, on what we’re doing here. So… What can you tell me about the Novar Mart dataset? So, we’re gonna have, we’re gonna have Claude Code help us sort of, like, do… lead up… lead us up to the opportunity sizing side of the house, but we’re asking it to kind of, like, give us an overview of what this is. So… it’s done thinking, Nova Mart eCommerce is an active dataset, simulates an e-commerce company, it’s got this many users, this many orders, this many, events, that is tracked, over in 2024. And so, it’s got 12 tables and Stuff like that.
So, notable patterns, it’s got a checkout funnel, so if you add stuff to your cart, and then you start to check out, you pay, and you, you make the purchase, and stuff like that, that’s sort of like the… Some of the… some of the data that’s available here. Now, let’s see, so, I’m intra- so I’m gonna give it my next question. I’m interested in… Let’s see, checkout… Conversion by device… Can you… We’ll do an analysis on that. So, right here, literally, I’m asking it to just give me a quick analysis around, what are you seeing on a conversion rate. basis by device, so, like, things like, mobile, desktop, and stuff like that.
And, and it will go do it based off of some of the skills and agents, that we have encoded. And we’ll take a look at exactly what it’s doing in terms of the skills and agents that it invokes. Okay, so it’s done… it says, strong signal, gap is concentrated in the start to payment steps. It’s gonna check for Simpson’s Paradox, and see if, if, if there’s any anomalies in there. So, it says, web beats mobile in every single channel, so you can always follow along as you, as you kind of, like, let the, let Clock Code, do its thing. And then it runs into an error, but it’s cool. It’s, self-healing, so it understands how to fix itself.
It learns… it knows how to, get around and or, resolve, kind of, like, errors that it hits. Okay, cool. So, it’s done doing its thing. So, it says web converts at 40% from start to complete. iOS is 29%, Android is 28%, so mobile user is 30% less likely to complete. Then a web user, and then it gives me kind of, like, the breakdown of this, in terms of where, kind of, like, where, where the drop-offs are. So it lists out the entire funnel, so from start to payment attempt, from payment to payment complete, and then, sort of, like, if you multiply them together, what is the, conversion rate?
So it gives me, sort of, like, you know, like, a narrative on, you know, what is, what it’s looking at, so it doesn’t just dump me to numbers. It does some sort of validation checks around, whether this triangulates with other data points in the dataset itself, and then it gives me the sources of where it’s doing, where it’s getting stuff, and it’s, it’s giving me sort of like, hey, what should we be doing next, to either make the analysis more robust or explore more more, more angles. So, the next thing I’m gonna do is, you know, like, this looks fine, what skills? Did you use to do this analysis?
So I want to… I’m curious, like, what exactly did you do to, what did you use in this repo that helped you kind of, like, do all of these, instead of, you know, like, it could be multiple, many different ways of, accomplishing the same goal? So, it’s, it says, didn’t invoke any of the skill, but I apply principles from several skills, so it’s got a question router, it’s got a visualization pattern skill, we have a triangulation skills. And and a whole bunch of other, stuff. It didn’t invoke any of the, sort of, like, analysis design, because it’s a pretty simple, question, question framing, guardrails, and things like that.
So… So you can always ask Claude to be like, you know, hey, what… what did you do? What, what, what helped you get to, kind of, like, some of the, some of the… the… the final outputs, what was helpful? Okay, cool. So, I’m gonna show you this one here, right here, which is opportunity sizing. So, on the left here, on my agents, I’ve got the… where is it? I have this agent called OpportunityZizer. So this one, basically, if I click into it, it’s like, hey, if I invoke this skill, it’s gonna quantify the business value of an opportunity.
Or… or do a sensitivity analysis and identify which assumptions matter the most, and then it’s got, like, instructions for how Claude should be doing, kind of like an opportunity sizing exercise. And so we’re gonna… we’re gonna use this to see what that looks like. For this, and… For this question. So, you know, we looked at conversion rate by device, and right now, I’m interested in, hey, if we fix, or if we improve upon the mobile checkout conversion rate, what would that look like in terms of, you know, like, bottom line and impact?
So, right here at the beginning, I am… doing the, you know, like, I’m invoking… I’m calling the agent, and then I give… I’m giving it sort of, like, the parameter, like, hey, here’s the question that I’m interested in, getting at. Can you… can you state your assumptions and what’s believable? So, we’ll let it… We’ll let it do its thing, and… let’s see… Do we have any notable questions here?
Sravya Madipalli: So, one question from Attendee was, did you specify the conversion metric somewhere, the final checkout for, from when item placed in the basket, or, you know, when user first hits the site, etc?
Hai Guan: Yeah, so, when you first load the dataset to the repo, or to the AI analyst, it will flag, sort of like, hey, based on what I understand, these are the metrics that matter, based on the column names, based on, kind of, like, the context of the dataset itself, and then for anything that you’re like, hey, no, that’s not how you would do it, then you give it more context. So, in this case, when it was first connected to the dataset itself, it figured out, correctly. Okay, so… let’s see. Okay, so it’s, it’s, it’s gonna do some data polling for the, for the sizing model. You know, like, we talked about the three components in the impact model.
What it’s doing behind the scene is exactly trying to do that. Like, hey, can I get concrete in some of these components in that… in that model, such that when I give you the sensitivity analysis, or the opportunity size, I can explain it myself. myself as in, like, Claude. Not me. Let’s see… Cool. So… I guess, Opus 4.7 is, is smarter, but it also thinks harder, so it takes a little longer than… 4.6. Has anyone tried Opus 4.7 yet?
Sravya Madipalli: There’s a chat going on with… I think Attendee, has been talking about he’s seen a difference in the, the information retrieval across, like, browsers. So, I was just telling him that we’ll have… we, do some… we need to do some, deep research work over the weekend to get our course up, and working for 4.7, so we’ll know soon. But yeah.
Hai Guan: Oh yeah, forgot to mention that, for the 5-week course, that we’re about to start next week. we’re actually gonna re-record our Week 3 to week 5 content, because now Opus 4.7 came out, so we actually just finished recording, and so we have to, like, redo that. So any piece of analysis or demos, or, like, exercises that we have students do in those, in those weeks. When… if you’re interested in joining us, when you see it, it’s gonna all reflect 4.7. It’s a crap ton of work. That’s for sure. But I think, sort of, like, you know, like, we can’t even get back to Opus 4.6, as you saw.
So it would be disingenuous to be like, you know, hey, here’s the, here’s the thing, and then, like, oh, oh, 4.7, we can’t really pinpoint the, the behavior. Okay, so let’s see, so it’s done its opportunity sizing, so basically, actually, there’s a report, so we can actually look at it. The main idea is that, it’s got, like, a base case that it identifies, 15% relative lift on mobile conversion, and it says, let’s see… base case, bottom line, and, like, if you calculate those numbers, what it flows through, it also gives you a range of, hey, if the pessimistic like, assumption is 5%, then this means this much revenue, this much gross profit based on the data set. If it’s base, then it’s this.
If it’s optimistic, it’s this. So, and then what it could look like in terms of what effort, what level of effort we’re talking about, in terms of what’s believable here. So, it’s also got some assumptions around, the cost of doing this, and so on and so forth. So, it’s also outputted a, A report for us to read, so we can actually check that out. Sizing, conversion, so it’s this one. It’s a markdown file, so it’s very friendly to to AI, but then for humans, I can just do a preview, and it will look much better, much easier for us to look at. So… Annual impact, so again, like, that’s the bottom line, that it really calculates, kind of like the one-pitch thing.
Let’s see, so… Top-line revenue impact of almost $300K, 113 gross profit, if we assume some sort of engineering cost. Confidence is medium. It’s data-backed from the start, so the very beginning, how many people it touches, and then the average order value, and then the margins are all data-backed. The lift magnitude is not as confident in. So… gives you the full, kind of, like, math around, what these are in terms of the… in terms of, in terms of actual concrete number based off of the analysis. So it’s not, like, kind of, like, making things up, it’s actually going into the data, doing the analysis, like we saw, and then, putting it Help here.
And… where the lift assumptions come from, so it would tell you, kind of like, hey, here’s how I thought about it, here’s where the assumption would be, things like that. And then it will also give sensitivity analysis, and then, like, you know, it’s, right now, it doesn’t have a lot of context around the business, the data sets, or anything like that, so, this is almost like a best guess based on Off of what it’s, reading from the data. Break-even analysis, it’s got scenario analysis. Base case, believable, what’s believable, what versus not. So, as a human with judgment, you’re supposed to be the one that really be critical about some of these assumptions and, and what it’s outputting.
So, it gives you a base, almost like a basket of things for you to kind of, like, think about, but it really stops at that. Like, you shouldn’t just take it and be like, oh, yeah, that’s all cool, like, I’m just gonna one-shot it and use that as the… And I’ll be all. So, final prompt for this exercise. I know we’re running out of time, which is… show me a sensitivity table with conversion rate lift versus average border value, so if we kind of, like, vary the assumptions, In, in conversion rate lift, like, what we can achieve, but also, like, average order value, because An incremental order may not be worth as much, over time if we sort of, like, intervene like this. Let’s see… Cool.
So, it would have kind of like a… It would have, like, kind of like a 2x2 here, Actually, can you visualize this for me? I’m a visual person, so I can’t really read these numbers myself too much, so I’m just gonna ask it to… to do it for me. And the only thing I would say is 4.7 is pretty slow.
Sravya Madipalli: Probably everyone’s hitting it right now. I’ve seen that happen. Yesterday, Claude was, also down, right? Yesterday morning?
Hai Guan: Yeah, yesterday was down.
Sravya Madipalli: Yesterday, day before, I think. Yeah.
Hai Guan: Okay, cool. We can take questions. I need to leave at 1, so I do need to have a hard stop, so we can answer questions as this thing is training itself. Anything that…
Sravya Madipalli: So, one thing that Attendee wanted to, ask… sorry, Attendee, if I’m pronouncing your name wrong, she had a question, I think they had a question around, are you also connecting Snowflake or Looker to help with giving it more context, or any other recommendations? I told her that the… this was around when you were giving Claude Web UI. I said it’s mostly web UI, but we… when we do Claude Code, then we do connect, is what I gave. So she asked, like, a deeper… they asked, like, a deeper question, like, underst… so this example didn’t have the connector set up. I would love to understand what BI have you seen? Best results for pulling opportunity sizes together?
Yeah, and they wanted to catch up on the recording later.
Hai Guan: Cool. Yeah, no, that’s a great question, and in the bootcamp, or… and in the course, we also touch upon this pretty in-depth. So, context is… context, domain knowledge, is everything, right? Like, Claude doesn’t, out of the box, understand any of your company stuff, or your… what the strategy is, what the, you know, like, what the… What the roadmap should be, or what previous… you know, analyses or numbers were like. So, one of the most important things is to be able to connect it to context. So. You know, like… and, by context. we go beyond just the warehouse. Like, we don’t… we don’t just think, you know, data warehouse. Think Notion pages, Google Docs, emails, Slacks, everything.
Everything with an MCP is not… is… is, is… context that you can actually bring in, and it’s actually pretty straightforward to bring that in, as long as your organization approves, for example, like, you know, AI, security, legal use cases, and things like that. hopefully that, hopefully that makes sense. But, like, the more that you pull in from as many, as diverse sources, even if they’re not, like, numbers or, like, data-related. sources. It’s gonna be a game changer for your analyst to actually understand, and give you much better answers.
Sravya Madipalli: One, question from Attendee before we go to Attendee. Is this course relevant for a product manager who is not a data science major?
Hai Guan: Yes, yes. Actually, we… built it for non… or, I mean, like, we built it with an audience for, like, the title for our course is called For Builders, so anybody who builds would be relevant. The idea is, like, we want to empower people, whether you’re a data scientist, or product manager, or designer, to be analytically independent. So, meaning, like, you have the foundations, you have the judgment from an analytic analytics perspective to make decisions, and then we also teach you to kind of, like, leave, delegate the execution to AI, so that you can ask better questions, have AI do the thing, and then you make better decisions off of it. Does that make sense?
Sravya Madipalli: Yeah, one thing I’d like to add to what Hai said is, anyone who works with data is going to benefit from it. People who work with data would probably use these skills that they generated. So, because all of this is going to be around AI and Cloud Code. So, data scientists could come and understand how they could use Cloud Code to replace their workflows and get an understanding of these frameworks and get the repo that they could take and use in their work.
And non-data scientists, like anyone who’s not a tech data scientist, any data analysts, non-tech, or product managers, they could understand what does a tech data scientist’s workflows look like, and how do you use AI to do those workflows yourself. And understand more. So, yeah. Attendee?
Attendee: Yeah, hey there, thanks for all of the amazing content. I’m gonna try to keep it short. I got a kind of a two-parter. I noticed you have a ton of agents and markdown files. When it comes to actually developing those, is the approach more so hybrid of, like, AI-produced with human input? And then the second piece is, you know, with this example that was being shown. Is it fair to say that this is very bespoke to the actual dataset of, like, Nova Mart itself, which would be more realistic for, like, a business setting?
your AI is continuously learning on your data, producing the outputs, so… hybrid inputs on Markdown files, and is it more so bespoke to the actual dataset where you can’t really funnel it to other datasets?
Hai Guan: Yeah, let me answer the second one, and Shabi, feel free to jump in if you have thoughts around that. So, your AI analyst is gonna be bespoke to your thing. Like, if you’re, you know, like, think of it as you clone it once. And then you start giving it context about your company, and then you develop it like that, almost like grow your data analyst. alongside you, with the context of your company and stuff. And then I will have no clue what… how you’ve grown your… your thing, but we teach you how to grow it, like, basically. Like, how to grow it, how to set it up, how to, you know, add more skills, add more agents that would be relevant for your business. And so on and so forth.
And potentially, this could go beyond just, like, you know, right now it’s, like, product data scientists, but you can imagine the framework is basically transferable to financial analysts, marketing analysts, whatever analysts out in the world, as long as you have the expertise to be able to develop alongside with it and learn, or help it learn. So, and relatedly to your first question, when we developed this. It is a, kind of like a, hey, we know what the best practices are for analytics workflows, because we’ve been doing it for so long. Claude itself, off the bat, also knows some, you know, like, how to do data analysis, for example.
It’s just not in the same way that we, you know, like, that someone has, you know. decades of experience would do. So, we basically give it all the best practices, we check, kind of, like, you know, where it doesn’t make sense, where it needs to be enhanced, and stuff like that to make the version that we have today.
Attendee: Awesome.
Hai Guan: Absolutely. You can absolutely build… a very similar system with your own expertise, so it’s all possible, so I think that’s kind of, like, the main thing.
Attendee: Yeah, I just imagine with new questions, you know, you have the opportunity sizing, you could advance that to, like, Monte Carlo simulators, and just keep going and going and going. And it’s all built around the root of the dataset. So, awesome, thank you so much.
Sravya Madipalli: We could take one last question. Attendee’s asking live in-person lessons for the course? Like, how does that work?
Hai Guan: Yeah, yeah, great question. So, this is a hybrid, so the… we have the lessons recorded, so they’re async, so you can watch them at your own pace. And then every week, we have a kickoff at the beginning of the week, and then two office hours. dotted throughout for people to come in, ask questions, or share learnings, or, or, you know, just powwow, or, like, just talk to us. Like, we’re… we’re pretty friendly, too, so, you know, we can definitely talk. And so, that’s kind of like the setup, but the lessons themselves are pre-recorded, and you can always kind of, like, go over… go through them at your own, you know, like, however many times you want.
And so that’s… that’s the hybrid component of, of the course. And then what is the location? Location is online, so it would be virtual. Cool. I need to drop, because I have a meeting with our CM… Chief Marketing Officer, so, Sravya, you can…
Sravya Madipalli: No, I need to drop off, too. Oh, okay.
Hai Guan: Nevermind.
Sravya Madipalli: meetings as well, so we try to sneak these workshops in, guys, to give you as much value as we can, while we obviously have our day job. So, really appreciate you all joining. We’ll share all these resources with you all, yeah.
Hai Guan: Thank you, everybody, and yeah, we’ll send a follow-up email with the recording and the resources and stuff. Thanks, all.
Sravya Madipalli: Bye, guys.
Attendee: Thank you very much. Bye-bye.