Speaker: Uh, honestly, this is a little bit of a click-baity title, but what we’re going to talk about is exactly what it’s saying. Uh, we’re gonna go into a framework around defining something called True North Metric, and then also the inputs that you can… You can decompose into to help move that metric. So, uh, before we begin, my name is Hai, uh, I’m the head of data at a legal tech company called ENTRE. And, uh, I previously worked at companies like LinkedIn, Pinterest, Nextdoor, Meta. I’m here joined by Maya, uh, we call them co-hosts, because we also host the podcast, uh, Shane Butler and Travia Madapali, um, would you guys like to introduce yourself a little bit? Hey, everyone.
I’m Shane, principal data scientist, the same company I worked at been in data science for about 10 years. Hello, everyone. I’m Sravya. I’m a Senior Manager in Data Science at Superhuman, previously called Grammarly. I worked. Hi, Shane and I knew each other from Nextdoor, and I started my data science career at Microsoft. I was there for 6 years, and yeah, been here for a while, like. More than a decade now for sure. Yeah. Cool, awesome. And, um, yeah, so, uh, we also run a couple courses, and we’ll, we’ll have more information later in the slides.
One on AI analytics for builders, uh, effectively helping everybody, regardless of your profession, to be analytically independent, and also AI to actually execute on analysis, on any analytics tasks to help you make better decisions. So that’s, uh, that’s a course that we run, um, and separately, Shane, uh, himself is also running an AI evals course for product development. So, you know, in the age of AI development, uh, the way to evaluate if your product is working or not is very different, so that’s, uh, that’s sort of like a dedicated deep dive. around that.
Um, we have a lot of free lessons, just like this one, coming up as well, uh, and all the course information you can find on DataNeighbor.com. Uh, alright, cool. So, let’s get into it. So, what are we talking about today? Um, we are talking about, obviously, metrics. Uh, who here have seen this problem here? You’ve got a lot of metrics at your company, and you… don’t know what to do. with them, or you don’t know what they are. Type 1, you need it. Yes. How many metrics do you look at per day in the chat, or you’re supposed to look at per day? You ever ask someone to, like, give you their metric, and they give you like 7 in return. You’re like, no, which like which one?
I guess only Attendee faces this problem. Cool, and that’s cool too. Um, this is a very common problem, um, honestly, where, uh, we have… Um, uh, dashboards, uh, especially if you work in, for example, in tech. There is a lot of dashboards around. There is, even in Google Analytics, has, you know, is a dashboard in itself, and if you go to the engagement tab, there’s a bunch, there’s a billion metrics, right? Attendee, uh, if you… if you remember those, uh, different tabs. Where, at least for me, when I looked at it, I’m like, what am I looking at here? What do they mean, kind of stuff. So, very common problem that a lot of teams measure either the wrong things, or are measuring a lot of things.
And so, you know, like, some would go up, some would go down, But really, none of them focus on hey, this is what you should be looking at, so that… you can measure whether our product, our services is actually delivering value for our customers, or our users, whatever your business may be. Um, this is… Very common, as I said, but very fixable. So, we’re gonna… we’re gonna talk through how to… how to think about it. Alright. Actually, yes, okay, cool. So, um, I’m just gonna highlight a very simple example here, where, um, metrics… can also mislead, which is actually worse than if you have a lot of metrics and you don’t know what to do with it.
Um, so, uh, the scenario here is we measure something called daily active users. Um, are people familiar with what that is? Have you heard of it? Before? So, roughly, something about… how much… how many times, or, you know, like, if a person has shown up to your… to your product or business. Um, and, uh, we count that as kind of like a DAU. So it’s a very, very, um, uh, very common metric for a lot of companies like social media in the social media space, um, and just generally. So, let’s say, um… You know, like, the scenario here is, uh, uh, we saw daily active users went up 30%, and everybody celebrates, right? So, so it’s like, oh, wow, great, up and to the right.
And then down the line, we’re like, oh, wait, hang on, turns out a lot of it is bot traffic. So, like, these are not humans. Um, I guess in today’s world, it’s a little less, uh… Less, uh, less like an anomaly, but previously, when only humans browsed browse, uh, browse the internet, uh, like, bot traffic is real, and that is not something that we care about.
And so, um… you know, like, probably the team could be making decisions off of The DAU trend for the last, uh, however many months, and then, uh, let’s say they saw, okay, yeah, we will launch this feature, and then it drove a lot more visitations, so we keep building and doubling down on it, and then 3 months later, they’re like, oh wow, this is actually not true. Uh, not real traffic, so they wasted 3 months building on a false signal. So that is actually very common. Uh, if the metric defined up front that guides the roadmap or the plan is not actually pointing towards the right thing. So, better metric judgment would have saved them.
Um, again, this is a very, sort of like, a very simple, high-level example, uh, but this happens all the time, and more times than, uh, companies and teams are willing to admit. Cool, okay. So, today’s framework, so what’s our solution? Um, solution is to equipped you with a framework that you can take home to, uh, to follow every single time when you have to define metrics. So… Uh, this is what’s called the True North plus Inputs Framework. This is used by a lot of companies, um, in the Silicon Valley, so Airbnb, Slack, Netflix, um, and many, many others have been modeled off of the exact same framework. So, sometimes you might have come across, um, a term called North Star Metric.
It’s synonymous. with True North Metric, so, um, either way, it’s fine. Um, basically, step one is to define what’s called True North Metric. So, one metric that captures the core customer value of the products or services that you’re offering. So, uh, like, This is basically, think hard to the end state of… what you’re building. What would it mean when… you have delivered value by the end of it, and is there some sort of way to measure that? That is sort of like the true north. So… This… this actually takes a lot of time, a lot of debates, and uh, you know, sometimes it’s obvious, sometimes it’s really not.
So, uh, focusing on this is gonna lead… down to, sort of, like, number 2 and number 3 here. Number two is, once you have a true north metric defined, then you can decompose it into what’s called input metrics. So, actionable levers that teams can actually uh, pull to influence the ultimate true north metric. So, we’ll get into some examples, um, uh, uh… in just a little bit. And then, how do you actually then define those input metrics? You can actually do this decomposition, which is number 3. It’s called the BDEF decomposition. We’ll, uh, we’ll explain what that stands for.
Uh, but almost all inputs you can imagine would be very similar… have similar flavors that falls into one of these four buckets. So, that’s gonna be a checklist that you can, uh, that we’ll go over in just a little bit. So, the key insights from this… from this, uh, from this slide, from this entire framework, honestly, is that… The True North is the outcome that you measure, and inputs are the levers that you can pull. to actually influence that outcome. Does that make sense to folks? Yes, no, one, two, one for yes, two for no. 3-4. I did not pay attention, and that’s okay. Cool, okay. Lots of ones. How north would be the true north metric? Um, as north as possible.
So, meaning as close to value as possible. That’s the way to think about it. And if it’s not possible to measure that, go one step self? If you can’t measure that, go one step south, kind of like doing it think of it like that. I’m just painting it, like, conceptually here. I kind of think about it is… and I got into this, but like, I think about like, what’s the the person as a human trying to do in their actual life, irrespective of the product. Like, imagine your product doesn’t exist. They’re trying to get something. And they’re trying to, like, get entertained, maybe, if it’s social media. They’re trying to get some workflow or some job done, some task, like… define that.
And then, like Hai said, is like, can you translate that to a behavior that’s represented like some like signal for that in your product? And then… A lot of things you can, and that’s when you keep going. If you can’t, you can build it, you can try and build that signal into your product, you can work with your engineering teams, try and build that way to capture that, otherwise keep going south. Yup. Good question, though. Uh, alright, um… True North is the metric that captures core personal value of a specific output of multiple products that a company might have. Uh… yeah. Each product to have its own North Star metric, and then you could have like a company North Star metric, too.
But sometimes it’s harder with the company stuff, because that’s a lot more lagging. So it’s a trade off of, like. You want you want a North Star metric to be something you can like kind of get a vibe on in like weeks, not like months or quarters. So when you start going to like a holistic kind of company value. Sometimes you don’t get that signal till later on. One other thing I’d like to add along with what Hai and Shane just added is generally core customer value will also be linked to what business gets value out of. For example, Meta’s daily active users means more activity that says users are, you know, benefiting from the product, which is amazing, which is the customer value here.
But it also implies. more ad revenue from these people coming in, and more revenue and, you know, Meta’s, uh, like, uh, like actual business value going up as well. So mostly the, uh, there’s a big correlation between true North Star metric, where I’m the, uh, like, you know, business, uh, like, what the business gets out of. getting the not symmetric up as well. Cool. Um, let’s, uh, let’s move to the next one here. So… Okay, bad versus good True North metrics, and here are just some examples of it. Again, I’m using an AI tool, if you guys are aware of it, Gamma. I don’t know how to do animation, or else it would have been a lot more suspense.
Uh, you can imagine, but, like, now it’s, like, all wide open. Um, so, uh, on the left, we have… the bad ones. And on the right, we have the good ones. The bad ones could actually be a lot of these things that most people think are good metrics. So, for example, daily active users. revenue and sign-ups. Like, these sound pretty reasonable, right? Like, most people report on one flavor or another of something like that, but they’re actually not good True Norths, and we’ll get into why that is the case. Uh, the really good TrueNorth metrics are things around, um, for example, for Airbnb, it’s, uh, how many nights are booked, uh, for Slack.
It’s how many messages were sent, and then for Netflix, it’s how many hours were watched. probably, like, I’m gonna spell out the exact difference between these two, but, like, Do you see any difference between these two columns here? Just, like, you know, just feel, just by feel. Like, they’re different, right? One is a lot more specific, the other one is… Much less, uh, yeah, business versus behavior. Mm-hmm. That’s right. One is customer-oriented, the other way is company-oriented, yup. Uh, cool. Uh, share example. B2B… yeah, we’ll get into that. Cool. Uh, so, let’s see. So, you know, why are the good ones? good ones. Um, they typically share some of these traits here.
Uh, so let’s do a counterexample, which we’ve been talking about for a little while here. Daily active users tells you… daily active users is not a good one. Why is that? Because DAUs tells you people showed up. So, like, they visited something. Like, they could visit a website, they might have visited, like, an application, or whatever it is. doesn’t say anything about whether they actually got value or not from that product and services. So it’s just, hey, they showed up, and we count them. So, there really is nothing that says, like, unless you’re… unless your core business is for people to just show up, like, just show up, regardless, don’t do anything, that’s fine.
It’s… it actually doesn’t tell you anything in terms of value. That’s probably the shallowest thing that you can measure. Now, why is nights booked a good one? If you think a little bit more about it, it actually means a traveler found a place to stay, And the host got paid. Like, you have two constituents as a company for Airbnb to sort of, like, provide the services for it. And this one metric captures the value from both sides. Not all metrics is… can be sort of like a… can be like that, but this is almost like a perfect example of, uh… of just something that touches upon, uh, kind of, like, both sides of the equation. If it’s a marketplace sort of product.
And then, just generally, good, true north metrics are leading indicators of revenue. Typically, it could be revenue, could be other sort of, like, company metric that, uh, um, uh, that, uh, Uh, that someone on the chat observed before. And it’s also a direct signal of customer value, so… They got value out of it, and then if you do this really, really well, and you move this metric up and to the right, or, you know, like, down and to the right, if that’s your… if that’s what the metric… a form factor is, then we have really good faith that revenue and business outcomes will materialize. Does that make sense for people? Hey, I had a question in the chat from Attendee.
around examples of true north for large B2B enterprise. I don’t know if you got this one already. Uh, large entity… large… B2B Enterprise? Yeah. Rather than consumer tech. So we work for our B2B, and so, like, I can just give you an example from our work, because I just typed this out to someone else. It’s like our product is a legal tech product that has it has many products within it. One of the products I work on. It’s called Market Builder. It basically helps lawyers, gives them suggestions and automates a bunch of the edits they do during the contract negotiation process. So edits on a document is what you can think of it as.
And we have North Star metrics around basically the time we save them from the start and finish of like a contract comes in. And then a contract goes out, but we also have a bunch of guardrails on, like, the level of quality of those. So it’s like a metric that kind of like combines both of those together, and that gets a signal of like, Hey, if we’re for maintaining quality of edits, and people are like. taking these and actually, like, lawyers reviewing them and not changing much more after it, and it’s reducing their time like those are. pretty good signals that that B2B product is working. So I think B2B SaaS is actually pretty good because it can.
It relates a lot more to like people’s actual like productivity and workflow. Sometimes the consumer tech stuff’s hard, because it’s like. engagement entertainment base, like, I don’t know. Social media North Stars are hard, because it’s like this person’s spending a lot of time on TikTok. Like, that’s… they must be having a good time, but it’s like, I don’t know. That’s really healthy for them, but uh… I don’t know, does that answer your question then? Yeah, and we’ll have an example, actually, for we all to kind of work through in just a little bit. It’s a B2B. Uh, cool. It’s like and then Attendee, it’s it’s like, so it’s. I think product adoption is a good signal.
Like someone’s using the product, adopting it. Like we can infer they’re getting value out of it. But if you can. That’s, like, a little south of the North Star metric. Like, if you can find something where it’s like, hey, what are they doing in the real world? And are we impacting that? Like, for my example, with the lawyers of, like, can we get the timestamp of when we know they like. received this document from a from one of their customers, from one of their clients, and then they sent it back. That’s, like, more of a signal, like. They’re adopting it, and then that adoption is actually causing them to have, like. of more efficient or higher quality or a lower cost workflow.
Yeah, and uh, once we get to the input metrics side, You’re gonna see the difference between these two, and you’ll understand, for example, like, adoption metrics, where does it actually fall? Good question, though. Alright, cool. So, let’s, uh, let’s move. right along. Uh, so, um, this whole True North metric framework, uh, like, you know, it’s an iteration, so it’s not like a, uh, uh, hey, once we define something, we have to stick with it forever. That’s just not how it works.
metrics is always iterative, so the business changes, the information about the business could change, and we would no… we would probably understand more of customer behavior, um, uh, as… as we sort of, like, you know, operate a business for longer, or operate a product for longer. So… even Netflix, I mean, it says, got it wrong first, but it really is, like, an iteration. Um, they started with something like subscribers added, and then later shifted to hours watched, as we saw. In the example slides, why was subscribers added not useful? Because it didn’t really tell them what to do. What about the subscribers?
Did you add… you know, quote-unquote, kind of, like, very, very lowly engaged subscribers? Do they actually watch anything? Are people actually, uh, you know, getting value? Which probably is entertained. Um, by your shows and stuff like that, like, it’s just not clear. It’s what we call a vanity metric. Measure something for the sake of measuring stuff. Uh, still… still useful in the grand scheme of things, but, like, not for, sort of, like, understanding value that you’re delivering as a product or services. So, once shifted to hours watched, reflects values delivered, so, um, you know, like, again, like, did… are the people entertained?
Uh, and when it moves, they know that their content strategy is working, or not working. And they have a way to understand if that’s because of the things that they put out. Uh, okay, cool. So, the way to evaluate your, um… True North, uh, is, uh, it’s called 7 Test Checklists, and we’ll send all these out, uh, afterwards, uh, so, um, don’t… Uh, don’t worry about taking notes and stuff. I know 7 is a lot. But generally, you have two criterias. One is practical, one is strategic. So, on the strategic side, we’ve talked quite a bit about some of these angles.
reflects customer value, so not just activity, like, not just did they do something, unless that doing something actually leads to value, or is actually the value in itself. Uh, it leads to, hopefully, business outcome. I wouldn’t say, like, revenue, but, like, most likely, a business is… exists to you know, like, to make revenue, to make money. So, eventually, it leads to that business outcome. Uh, it is hard to game, so… You know, like, you can imagine if I define my true morph, be how many clicks on a button in, uh, on a website, uh, and that my true north, then that’s very easy game, right?
Like, I can make it neon, I can make it flash, I can make it bigger, like, you know, then people would click more. Um, so, uh, something that is hard to game. Uh, uh, one number, you know, like, this is, this is kind of, you know, more creative than, uh, than it is, uh, uh, technical. Uh, a number that the… whole company, or the organization, or the team, can rally around. So, like, people, you know, like, people feel like, uh, they’re invested in it. Uh, practical criteria is that it’s actionable, so you can actually… it can actually inform what you… what you can do. Later down the line, uh, it is understandable, so… Time and again.
I come across metrics that are defined, that are very convoluted, so by the end of explaining what it means, Uh, you know, like most people have already checked out. So that’s also not good. Uh, measurable, so you can actually measure it. So, if the data doesn’t exist, or if it’s just no… there’s no way to put it in a numbers format, that’s probably not practical. Does that make sense here? Cool. Right. Moving right along. Okay, so we have True North Metric, and everyone here is an expert at defining that now. Uh, or at least knowing the high-level framework of it. Let’s talk inputs. So, um, what’s input metrics?
Uh, so… Teams don’t really… I mean, teams do own True North Metrics, but then you would have, sort of, like, actual sub-teams. that own the different inputs, so the levers for moving that True North metric. A very simple example here, uh, and… sorry, used another consumer example here, but uh, we can get into… we’ll get into that, the B2B example in an exercise in just a little bit. So, imagine you are, um… Actually, I don’t know if, Sravya, that’s what your company does. Let’s call it, like, Sravya Company has some… has a… has a product called Grammarly, right? Like, helping people to write better, be grammatically correct, use AI to refine writing, uh, I’m just gonna make it out.
Perhaps her company wants to get as many users to try out the product as possible, so that… the, uh, there’s a growth team that Shravia is on. That basically is responsible for a true north metric called retained signups. So people who don’t just sign up give you an address, email address, and, you know, never use the product, but, like, they actually stuck around for some time after they’ve signed up for, uh, the… you know, like, the Grammarly widget. Uh, and so that… that’s a reasonable true north. Now, if you imagine, sort of, like, if we just express that in a very simple equation, what… what does… what does that actually mean, right?
Like, you can actually decompose it into the different, uh, percentages and rates and stuff to arrive at the exact number here, which is how many traff- how much traffic can you attract? to, for example, Grammarly. Uh, and then what’s the conversion rate? Because you have to, like, go through assign a flow to… to do that, right? So you have to first land on Grammarly’s website somehow. Uh, and then, uh, what is the conversion rate on signups from those traffic? And then of the ones that signed up, what is their retention rate? However we define that. Um, first 7 days, first 30 days. Doesn’t matter.
This is basically what it is to… that… that helps you get to a retained sign-ups, sort of, um… sort of final outcome here. Now, if I were tasked, if I were to head of growth, the growth team, and I’m like, okay, move, retain signups. I would have no clue how to do that. Like, that’s, like, what levers can you actually pull? But when you are able to decompose it a little bit more into… You know, one team responsible, maybe the growth marketing team, responsible for driving traffic to the website. And then you have another team that’s responsible for making the sign-up flow really, really efficient, or really, really pretty.
then… then you’re optimizing for the middle, and then you have maybe an onboarding team that makes sure for every single new person, they have a really good experience when they onboard to this product. then they, in fact, impact the retention rates. You see how these different teams can work together to actually just move? their respective inputs. And then the True North would actually move along the way. that’s kind of, like, roughly the structure that we’re talking about here, where the true north is the outcome. Which, sometimes, is a lagging indicator. the inputs are the levers that you can actually dial up or down by the effort that you put in.
And then each team knows exactly what they’re influencing, so, like, the growth marketing team knows that they need to drive traffic, the onboarding team knows that When somebody signs up, I need to give them a really great experience. And that’s all they focus on. So, when all these things sort of like, you know, or even… not all of them, like, even one team is killing it, and the other teams do nothing, Your True North still moves. And the thing with input metrics, why we don’t just have input metrics, why we need because like, right, like you could think like, oh, traffic signup conversion retention rate. Those are my. My North Stars input metrics.
These ones are pretty straightforward, but, like, say you’re, like, building a product, sometimes there can be like highly gameable. So I mean, even something like, yeah, traffic. I don’t know if you’re just like getting traffic, like by bots like high said or by like. Paid traffic or by like email blasts that it’s like people are coming in once, but then they’re not retaining, then it’s working against all of those other. Input metrics, and so your North Star metric won’t actually go up. So the North Star metric is like a really nice way to validate of like, hey, is this actually a strong input metric, or is the input metric. too gameable. Yep, good point. Cool.
Um, again, not… I mean, like, most… metrics. Don’t decompose so nicely, uh, like, like, I’ll be honest here. It’s very rare that you get something that decomposes into, like, an exact equation where you just dial up, dial down, and, like, you know exactly what the true north is gonna do. A lot of it is, uh, sometimes you have to have faith, sometimes you have to have proxies, and that’s all okay. The idea really is, like, we understand, intentional, from a… like, intentionally, that, uh, that the metrics that we choose we have faith that it actually could move true north. It passes the smell test, it passes the, uh, sort of, like, the gut check. Hey, Haya, a couple good questions in the chat.
One, maybe get through this. This one first, and then. from Attendee again. So all these metrics have like cost to maintain and also cost of resources to like argue about them and have meetings around. Do you have any rule of thumb for, like. the max number of metrics a team should be. The MacBook. I guess it probably depends on the size and stuff like that. But it’s pretty. It’s pretty good valid question. Yeah, it is. Uh, no exact rule of thumb, but you can imagine 100 is too much… too many. 1 is probably not enough. So, um, I’m not gonna… I’m not gonna say, like, it’s somewhere in the middle. It is somewhere in the middle. Uh, the… the point that you’ve… 42.
that you’ve raised is actually a really valid one. How much attention can you hold while you can still not go crazy with these numbers? That’s… that’s… Next up, cool. What did the kids do? 6, 7. Is that a periphery? That’s kind of appropriate use of this. 6 or 7, 6 or 7, that seems… that seems fine. One thing… sorry, go ahead. I… I was gonna say, like, I think something that that I do like kind of like what I said before, like, take your input metrics, see which ones have the highest impact on your North Star metric. And also have like like you get the highest leverage to move. So basically most ROI. And then you can kind of like make a list. And some of that you can use data to do.
Sometimes you can talk to experts. Sometimes it’s just intuition. And then you can like prioritize them. And then maybe instead of capping the metrics, cap your meetings, or whatever, where it’s like, Hey, we’re going to go through this stuff like we’re going to get through these really important ones first. Or we’re gonna have our team assigned to moving these really important ones first. It’s more of like a resource problem and just like prioritization at that level. Another thing that I’d like to add to Shane is that once you get into war rooms of why your North Star method dropped, you’ll know what are the main reasons why you’re not staff metric dropped.
You go to the metric tree and see what affects your North Storm metric most. The more and more we get into those. modes to understand why you’re not summertime moved, you’d understand what are the top methods that influence it. One other thing I wanted to add on the questions, I think Attendee, I don’t see their name. Their question was about having weekly, like seven days, like, how do we have a time bounded, the metric seven days, 21 days, like, you know. And they were also asking questions about Q1, like in different quarters, like, how do you do that? So I had a short answer there. Not sure if it was helpful, but wanted to give like a live understanding of what I meant.
Based on the type of product, if you are a daily product, let’s say. you expect the user to come to see and, you know, look at the feed on a daily basis, like Instagram, for example, I would say that’s a daily metric. If you have a weekly product where you see, like, you know, work week, almost like you have users, you think, like a project management platform or, you know, something like that, that would be a seven days. And if you think it’s a. monthly, like, tax tooling something I think Pinterest used to do monthlies as well, where you expect like it’s event based or, you know, something that’s related where people come in once a month and that’s good enough, then that’s.
a monthly product for you. Uh, and there’s also, like, a question about A-B test, like, it’s a very good question. It has tons of things, uh, in line, so we’ll probably do a bigger deep dive in our course later. But, uh, to answer in short, A-B tests are time-bounded by the test itself. So you would have metrics for the test. Basically, based on how long the test is, you have metrics to measure the metrics within that A-B test. Yeah. Cool. And then for, um, like a dental practice, it might be a 6-month thing, because, you know, like, that’s, uh… Or quarterly, probably. So that’s very different. All depends on the context of your business and product that you offer. Okay, cool.
So, moving right along. Love all the questions, by the way. That’s, uh… you guys are awesome. Alright, cool. Uh, did we go through this slide? No. Um, okay, so how do you actually, uh… choose inputs? Like, how do you… where do you… how do you find your input metrics? Um, a… This is sort of like the… what I was alluding to a little earlier. The BDEF framework, um… Uh, probably a better way, better than this, but uh… Generally, you can… Uh, you can… you can imagine, most of the input metrics that you can think of would probably feed into one of these categories. So, the first one is breath. So, how many people, like, how broad are we talking about?
Um, and… Again, using Airbnb as an example here, it could be active listings, or it could not be number of guests. So, like, how broad, generally, are we measuring something? So, that could be a flavor of an input. Depth is the second one, which is… how much per user, or per customer kind of, uh, kind of metric. So, in the case of Airbnb, it’s average booking value, or how many… average number of nights per booking, like, things like that. Um, so it gets into, like, uh, uh, just, like, you know, how deep. you go, in terms of providing the inputs. And then the third is an efficiency metric. So, um, you know, like, how, I guess, how smoothly or how much time is saved. Things like that.
Or, you know, like, what is conversion rate? So, this would be, you know, Airbnb’s case, When you search for, for, uh, for, uh, for a place to stay, what… how likely are you to actually book it once it’s, uh, once it’s surfaced to you? So, efficiency is, uh… is on that. Uh, and then number 4 is just frequency. So, uh, how many times, um, Has this been done? Uh, and uh… And just how broad is the sort of, like, repeat behavior is. Uh, so these are sort of, like, these almost encompass kind of, like, all the inputs that you can imagine. And so, it’s one… it’s one way to sort of, like, you know, put it in a framework around, uh, Uh, okay, my input belongs to this bucket here. Okay. Cool.
Uh, let’s see… okay, so we actually get to this, uh, B2B exercise here. So, diagnose this metric. So, a product manager at a team collaboration tool, so think Asana, Monday.com, or Trello, Uh, so hopefully you’re all at least familiar with some of these, uh, some of these products, or maybe, like, Does anyone have no idea what these are? type of 2, if that’s… if that’s the case. Okay, so everyone knows. Great. So, product management… oh, sorry, project management, uh, software, or tool. Uh, and, uh, this, uh, product manager, uh, comes and says, Hey, our metric is monthly active users, and we want to grow it. What is wrong with this, uh, with this metric? Real quick type of chat.
Frequency is off, vanity… Nice. no business value. Attendee, showing up is not useful for… a collaboration tool? Doesn’t reflect value, yup. Cool, great, awesome. So, yes, exactly. Uh, so, problem with this… why it fails? Everybody got it. doesn’t capture value. People can show up and do nothing, and as a collaboration tool, what does that actually mean? Like, it means nothing. They didn’t get any value out of it. Uh, it’s vanity. So, uh, yeah, like, I think Attendee said it, uh, oh no, it’s Attendee. Uh, no one knows what to do with this metric, it just… it just is. Um, you can’t really move the needle on MAU here, uh, so, you know, like, not in face value, at least. So, yes. You all got it.
This is a bad metric. All right, so once an improved version of it? Um… So, an example improved version of this collaboration tool could be… that the True North could be team projects completed per week. Right? Like, without even going into the detail of what that even means, like, how do you measure a team project, what is a completion, and, you know, like, per week, is it 7 days, is… Is it, like, the Monday to Saturday, or whatever it is? Uh, you can feel that this is different, right? This… this… this gets more specific. And uh… existed… if you’re… if your idea is to be a tool, you actually help people complete something. Right.
Um, and then the inputs to this metric could be any of these following, and or it could be more. So, the breadth… on the breadth side could be active teams with a shared project, On the dub side could be tasks completed per project, so if you imagine collaboration tools, you probably have a lot of checkboxes on, uh, things are signed out to different people, and uh, you know, like, checking… Checking them done is actually a good thing. Uh, could be efficiency, so measuring the time from project creation to first task completion, that itself is pretty valid, like, how many people are actually taking the project seriously, or taking the tools seriously, using it to track.
Uh, it could be frequency, so weekly active collaborators. On, uh, uh, on the project itself. Like any of these could be really, really good inputs to this much more refined TrueNoife. Uh, versus sort of, like, just monthly active users. Cool. And, uh, you know, you can imagine, uh, if these were sort of, like, defined, like, teams are going to be able to know, um, or people, teams, or… are going to be able to know, like, you know, what their row is in moving this TrueNorth. Okay, cool, okay. Quick… Uh, best True North for a food delivery app. Think DoorDash. or grab, or… who eats, something like that. 1, 2, 3, or 4. Give everyone… 15 seconds here. think, what’s the value?
Got a lot of ones, twos… Yeah, this is a hard one. 3 average order value to… what does it mean by a success and failure in one? Great question. I don’t know. Uh… So, I mean, we can… we can define it however we want, so assume it’s, uh… order actually delivered to the client, or the customer, and then how many orders were ordered. or replaced. Okay, cool. Great. Uh, so a lot of… mostly centered on 1 and 2s. drumroll, let me find a button here. Okay, Banzer is number 2. orders delivered per month. Uh, why is that? So, if you remember… on True North metric. a few slides ago. what we’re after is driving the outcome, or deliver the customer value that we want.
So, for a food delivery app, You can imagine, uh, if we actually deliver the order, then the, sort of, like, the transaction ends, right? Like, the value is delivered. Um, like, you know, people are placing order, and they get it, and they can eat. So, uh, why is number one? Not a, um… not a good, uh, or I guess, uh, number one is not actually not a good True North, depending on your team. Uh, but it can be a very good input. So, delivery success rate is a ratio, so, um, you can double your order, and the rate stays flat, and it would look like nothing changes.
Unless your team or your organization’s True North is to actually drive efficiency on, you know, like, making sure that the orders are not dropped in any way or form, shape or form. Uh, YS3 not a… not the best? Um… metric here, average order value. I think some of you picked that. Uh, it’s depth. Uh, so going back to the BDEF framework thing, this is one of the more, like, you know, how many, um… you know, like, yeah, like, like, something per something sort of measurements. And then 4 weekly active customers. It’s, uh, again, it’s not value delivered.
Like, customers can show up, but if you know, we failed, they can place an order, but if we fail to deliver it to them, they didn’t actually get the value. They could be placing a lot of orders. Uh, but it would actually lead to really pissed-off customers. Uh, cool. Yes, one is an input. Alright, cool. This is… this is really a hard one. Uh, I made sure to make this not as straightforward. The other ones are all, like, MAUs and stuff. But and like, when you work with your team to do this, like. Um, there’s kind of, like, brainstorms and stuff you can do, but it is hard to land on these. Like, someone asked earlier in the chat, how often do you revisit it?
And I feel like when a team is first wrapping their heads around, like, what’s our north starts, like, I feel like the first month. to 3 months, it’s like, you… every other week, like, oh, like, actually, we’re gonna do a different metric. And, like, you know, you’re reporting these to people high up, and so it’s kind of awkward, like, every two weeks, like, oh, we changed our metric again. But I think that’s just part of the process, like, take that first quarter with. Like the understanding of, like, it’s not going to be perfect. We’re going to try some things out and like learn what makes sense, and then, like, you’ll revisit more and more. Yep.
Yeah, it’s really hard knitting with with new products. that don’t have historical data. A lot of that’s talking to. Um, hopefully you have, like, some, like, codevs or beta users or a small sample, like, a lot of that’s talking to users for prospective users. Yeah, and if you have no baseline, it’s cool. Yeah. Trust your gut, that’s fine. Um, go from there, gotta start somewhere and iterate. Probably the iteration window is gonna be shorter, um, but, like, you have to start somewhere. That’s… I shouldn’t say this as a data scientist, because it like is not good for getting data scientist jobs. But it’s like, if you have a brand new product like probably don’t need data scientists.
Also, there’s no data like you probably don’t don’t need them yet. That is true, and I saw a question around, what about total order value delivered? Uh, that’s a really good one. Um, it’s actually in the very similar vein as something like total signups. So the number can only go up and to the right, so… Um, like, you know, like, only one direction sort of metrics, um, is typically not suitable for a true north, because everything you do, it’s only going to go up, or at worst, stay flat. So it doesn’t actually give you a measurement of the state of, uh, of value that you’re providing, whether it’s going up over time or going down over time. Good question, though. Okay, cool.
Let’s move right along. Uh, okay, transition to AI demo. So… At the end of this, uh, as a follow-up to this session, um, I will be sending you all this deck, And also a couple prompts for, uh, any of your favorite chatbots. Claude, ChatGPT, Gemini, Grok, anything. Uh, that you can actually practice with, um, with, uh, uh, based on what we talked about today. Alright, cool. Let’s, uh… to the next one. Uh, yes. So, we have a couple free workshops coming up, um… These are the most immediate two. Uh, that’s, uh, that’s happening, so if you’re interested, uh, certainly, you know, like, scan the code or, uh, uh, or, uh, go into the, go to the links.
Uh, we have one tomorrow that Shane is hosting to… on designing experiments for AI features. And then, uh, next week, we’re really excited about this one, which is analyze data with Cloud Code, Opus 4.6. Uh, we have a lot of really exciting, um, and I would say almost pioneering, uh, materials around actually how to effectively use AI to do all the analysis work. as much as possible, meaning, like, uh, we… And I’ll actually show some of that later. Uh, on, you know, like, just delegating the execution work. to AI, and have faith that it actually knows what it’s doing. Uh, so we’re super excited about, uh, these, uh, these sessions, and, uh, certainly sign up.
Um, they’re free, and uh, they’re happening in the next, uh, week or two. One other plug. So like I’ll probably show you lots of like links and stuff. If you just go to AI analyst lab.ai. I dropped the link in chat. Pretty much everything’s on there. It’s like, that’s like the directory like all our courses are on there. All our free workshops are on there. They’re like free email courses. We have a podcast. We have blogs. We’re going to post like analyses up there too, so. There you go. Um, that was, uh, coded up two days ago, and this deck did not incorporate that yet. A bot? And a bunch of things to rewind regarding all our, you know, like, everything, everywhere you could touch us on.
We are on LinkedIn, all of us, Ravia, Hai, and Shane. We post a bunch about all our research in the Claude Gold, and, you know, AI analyst area. We have a bunch of free workshops coming up. We have two courses. The first one is about the AI analytics for builders. And you could find about that as well on Maven. You could also find about that in like AI analystlab.ai that Shane shared. And we also have another course coming up, which is going into the depths of how do you, like, you know, incorporate Claude Code 4.6 into your daily data science and analytics, or even as a PM or as, like, anyone working with data, how do you incorporate, uh. these workflows, the AI workflows into your day-to-day.
So, if you have any questions about those courses and, you know, want to understand more, if that’s the right fit, you could set up some time with me, uh, and I could just help you understand, like, if, uh, are those courses a good fit for you or not, and you know. Cool. Nice. We can go from there, yeah. Alright, cool.
Uh, we’re not gonna have time to do the actual demo, but I’ll send you all the prompts and stuff, but, um… Uh, we have a couple prompts where it helps you… it kind of, you know, acts as a coach to, um, to help you… refine and iterate on metrics, and it will ask you very similar questions that we went through on the framework, and so you can always run through your specific business, your specific service, specific product, and what you’re thinking about against it. So, what we’re doing is, uh, so this is, like, the demo output if we were to do it, um, in ChatGPT, I actually have it here, but uh, we just don’t have time for it.
Uh, it’ll show you, sort of, like, hey, is this failing at this step? Is this failing at here? Is this, uh, you know, actionable and things like that, and it will have help you give, uh, it’ll give you suggestions for how to really think about it. Uh, beyond, you know, what you proposed. Oh, um, I do want to leave some question, or some time for Q&A, so, uh, I’m gonna end in the next couple of slides. Uh, so remember, your true north metric is the outcome that you’re measuring, and all the inputs that we went through are the levers that you can pull to actually measure it, or to actually influence it. So, this chain has to be very clear to you in terms of The cost and the effect.
So, learn the framework, and then let AI help you move. Faster. That’s the tagline. Hey, and I know we’re over… we’re only about 5 minutes. I can stay over, like, 30 minutes. If I instructed if you got bounce, no worries, but I can also stay over folks. Yeah, so, nice. So, Shane will stay for Q&A, and certainly feel free to connect with any of us, um, and we can chat more. Uh, we have a free Slack community on just AI builders generally who are interested in analytics, AI, and just sharing knowledge, sharing discussions. All free. Feel free to join us. Uh, we have… we share a lot of free resources and knowledge on that as well. bit.ly slash AI Connect. It’s a Slack, uh, Slack community.
Okay, and uh, Shravia… gone through this, uh, quite a bit. Um, so we have this AI analytics build for Builders course, Uh, where we go in a lot more depth around some of the, you know, this is one of probably the 60, 70, and 80, 90 lessons that we have planned. Uh, so, um, we’re gonna help… Our goal is to help people to become analytically independent. What that means is knowing how to ask the right questions, think like a seasoned data scientist or data analyst would do, um, and then be able to delegate execution of those work to AI, and actually get very, very good results. That’s our promise, and that’s our goal.
five-week training starting in April for this group in the Lightning Lesson Attendees, we have a 25% off. Uh, promo code for you. Uh, you can scan the QR code, or go to bit.ly slash AIanalytics Builders, and that’ll be, uh, that’ll get you all started. And feel free to connect with, again, with, uh, with any of us. Okay, cool. So that’s the… that’s my last slide. Uh, we’re gonna go into Q&A, but before that, I want to show you a few of the things that we’ve been doing, um… And uh… and then… Hey, before you get into that, or you can open that up. There’s another thing that we just created this week that’s brand new. It’s a boot camp. It’s a weekend boot camp.
It doesn’t go into the whole like analytical thinking framework like this one does. If you are, like, familiar with kind of like analytical framework and analytical thinking already, but you just want to like get set up in Claude code for data analysis like effectively. building, like, your own AI analyst in cloud code. We’re going to do a 2 day workshop. It’s like a weekend workshop. It’s gonna be 4 hours each day. We… this is new, because basically we tested this out. We weren’t really able to do this very well, like, a month ago, but with Opus 4.6 coming out, it’s, like… really remarkable what it can do now. So that’s why we’re kicking off this new boot camp.
If you take the boot camp and then you want to take the full course later. We’ll just subtract whatever you paid on the boot camp out of the full course tuition. But yeah, I don’t know. That’s an interesting way to get your feet wet if you just want to do, like, the technical aspect of it. And by technical, it’s like set up Claude code and speak English to it. You don’t even have to type. It’s not like you’re you’re coding, so… It’s pretty interesting, happy to chat more with that. If you go on LinkedIn and, like, look at some of the stuff me and Hai and Shravi have been posting about this week, I’ve been sharing a lot of the learnings on that. Yeah, totally.
And that’s a good segue into something that I wanted to show you guys. These are… from the AI analyst, that we built. And that we will help you figure out how to actually do it yourself. Um, this is a full deck. The presentation included, like, the actual deck and the imagery and the charts and stuff. Uh, included. This is, like, the very early iteration of this AI analyst. Uh, it would basically incorporate all the, um, all the best practices and, like, figure out all the logic, all the reasoning, uh, things like that. So, you know, this one is done on my own. product, uh, sorry, my own, uh… Uh, real estate, uh, uh, business, and so… this is… this… this data has never been seen.
And then here’s one that Shane did test it out on, sort of, like, public data from Hawaii tourism, and then it gave us, a, uh, basically, what’s wrong with tourism in Hawaii. Almost one shot. Uh, it does its own checks. understands what to… what insights to pull from, and then what you can do about it. we actually tagged a Hawaiian tourism board to share this with them on LinkedIn as well. And then finally, this is the… latest iteration of our AI Analyst, uh, just hot off the shelf this morning. Uh, this is an interactive… HTML presentation, so the other ones were PDFs.
This one, you can actually navigate, um, on… campsite data, again, public data, pretty niche, um, but you can imagine this deck helps you understand If you were to book a campsite, because I had to do it, How do I do it strategically? And, uh, what’s the timing, and where should I go? So, uh, and I can’t really… demo it here, but, like, this button, when I click it, it also has presenter notes, like, exactly what I should be saying at each slide. Uh, that’s… that’s… that’s included in this, uh, in this output from our AI analyst.
We do not write in code, we did not… We did not write any code, we did not write any, um… any of these slides, we didn’t do any of the formattings, we didn’t do any of the recommendations or takeaways at the end. Um, all of this is AI, so we really think it’s a game changer with Opus 4.6. From Claude, and we want to make it accessible to everybody. One thing… one thing, like, just to emphasize like like we’re not selling a product or anything like we’re just going to give you the repo like that we have, but… Like, you can build this yourself. That’s what the bootcamp’s about.
It’s about building an AI analyst and Claude code that is extremely specific to your use case and your data and what you want to accomplish. And, like, I’m gonna post about this later, but it’s kind of insane. You can actually… seed this in such a way where you have, like, a… right now we have, like, a 60,000 line code repo that does all this. When I tested last night was effectively creating like a single markdown file that’s like a genome or a seed, and give it to Claude code, where it just… and feed it like what I want to do and give it my data. And it’s like, go build the data team for me. And over like several hours.
It just spun up a bunch of agents and started expanding itself to like rebuild from scratch its own, but now specific to a different use case I’m working with. So that’s kind of what we’re going to teach here. I mean, if you… we’re doing a free workshop on this next Friday, and I’ll just give you the link to the repo then, too, for the free one, and you can fork it. But, uh… That’s what the boot camp’s about, not trying to, like… I don’t know. I’m not… I’m not trying to go, like, build Tableau or something, but I’m helping to… we’re hoping to, like, help people enable themselves to build things like that themselves. Yep, totally.
And I haven’t written code in a long time, um, so my day job is to manage. That’s basically all I do, um, but I feel like I’m now able to replicate something that I was able to do, and do it faster, better, and uh… and more efficiently. So, that’s… And another example that’s not here is so 13 years ago, I did Microsoft internship and I got a case study that was sent to me to, you know, and it took three days back then that got me the actual on-site interview into Microsoft. I could replicate that in 45 minutes with Cloud Code. And that too with the previous version. With the newer version, Shane, I need to try the one that you changed.
Probably it’s even, uh, you know, it’s going to take even way less time, but it’s incredible how much we could get done with this. Opus 4.6. And we are glad that we are able to test this ourselves and we could teach you guys too. Cool. Awesome. Um… So, we didn’t get into dedicated Q&A time, um, but Shane’s gonna be here to take… questions from any of you who wants to stay over. I need to hop. Unfortunately for a meeting, but thank you all for attending this. This has been great, and thank you for all the engagement. I look forward to speaking with you hopefully soon. Same here, everyone. CIA. Sorry. Any questions? All right.
Let me let me read your thing here, Attendee course is mainly teaching senior level DS workflows. How to recommend students hunting entry level positions is on their resume or in the interviews. Yeah, so I think for okay. So I think for your question, Attendee, I think the 5 week course gets into a lot of this stuff. So the 5 week course, just to kind of break it down for you. It’s like the first. week is more like analytical thinking, kind of like the thing that Hai went through here where he’s teaching you how to frame questions and build questions and build metrics and a lot of other stuff. Like these are just like core skills around analytical thinking.
The second week is getting set up with Claude code and potentially some other. Some other analytical analytic AI analytics platforms like hex or something. weeks 3 and 4 are then getting into like drivers analysis and causal inference like we’re going to teach those actual concepts. But then we’ll show you how to implement them. in, like, with the help of AI, but we’re going to teach the concepts themselves in that one. And then week five is around, like, okay, how do I, like, storytell and provide a narrative and bring this to, like. Um, like, you know, my manager or my exec can, like, get done, basically. So that’s still, like, that course isn’t, like, hand it all off to AI.
It’s more like, how do I do the end-to-end data science workflow? But hand off a lot of some of the technical components to make it faster. The boot camp in itself is a lot more of that, like, okay, let’s work with you to like build our own like. AI analyst or a data team within Claude code. We’re not going to go into the kind of, like, analytical frameworks there, but we’re going to go a lot deeper into Claude code itself. I don’t know if that answers your question. Cool. What’s the repo you were talking about? We’re going to share it in our free workshop. Let me see if I can pull up the link to the free workshop. I need to clean it up a little bit before I before I share it out.
But, uh, yeah, it’s just the repo for our… that we have like built like the AI analyst in. So it has basically like 20 like different agents, like, like agents that are like… UX experts for like storytelling experts for like versus like driver analysis versus like causal inference experts. And then it has, like, a bunch of Claude Coat skills that are already created, and then it has a bunch of, like, helper functions in Python for like making all of, like, the… beautiful charts that high showed, as well as like how to like knit together into a Pdf. Um, so we’re gonna share that repo. Um, if people want to just like fork it and use it.
But I think what’s more interesting in the boot camp is like building it from scratch yourself because then it’s like hyper… tailored to your use case. And like the brand of your company and what you’re trying to work on. But let me try and… pull up the free lightning lesson. For you, and I’ll drop the link. We’ll share the repo once we kind of walk through it in this and do a big demo. And that lightning lesson is going to be like a 2 h thing next Friday. And we didn’t, you saw we didn’t really like reserve as much time for Q&A at the end here. We try to get to them during it, but that one we’ll really try and do one hour. walk through one hour full Q&A.
Okay, that should be the link to the free workshop. No problem. Any other questions? Let me. Oh, the other thing, you know, I read a blog post on this this weekend of what we were talking about. So you can also check this out. The title is somewhat clickbaity replacing your data team. But you gotta do what you got to do for an SEO. Oh, it was nice. You saw it already. Okay, cool. Other questions? No worries if there’s no other questions. I’m going to hang around for I can hang around for 22 min. Okay, here we go. 0 to one stage. When’s the optimal time to define your North Star metric? I think… so, I mean. And there’s 01 in stage.
You’re kind of probably already defining North Star metric without actually defining the metric itself, because you’re probably talking a lot about, like. Why are we even building this thing? Like, what’s the problem we’re trying to solve for a user? So, I think you’re probably already having a lot of that conversation in terms of, like, what value you’re bringing to the user. Now, you might not have data to actually create a formal metric that you’re going to put in a dashboard or a SQL query or a Google sheet or whatever.
But you can still have that kind of user value statement and then be interviewing users or even it’s like with the team like having sessions where you’re working through your product and seeing if it’s actually hitting that. I think in terms of like so so you already have that. I think it’s really good to think about that early on. You’re probably already doing that. And then eventually, when you start collecting data. Or when you’re, like, ready to collect data, like, if you, like, have the product out in front of someone. Have your team think as much as possible. It’s like, what do we need to create within the product where we can like get some reflection of that user value?
Like you might have to build something. You might have to, like, make a workflow or a timestamp or something that doesn’t exist yet that can actually kind of like. give you signal around that, and then instrument that data like as early as possible. So then, once you have it like collecting and flowing through, you can naturally translate that into a metric. Did that kind of answer your question, Attendee? Uh, yes, absolutely, thank you very much, John. Sweet. What else? What other questions? What do you think about aha habit moments? Are they input metrics? for North Star metric, should we validate input metrics? that they actually impact North Star forever. Aha!
Do you mean like the thing that’s, like… I don’t know, uh, Facebook had, like, they’re like once someone has like seven friends or something, there’s like some like trigger. Yeah, yeah. Okay. I think that’s like goals. I think that’s like… You have your North Star metric, or maybe it’s your input metric input metric. I think it’s probably, like, a goal of, like. that you’re trying to reach, rather than but I think it does relate to the North Star metric could relate to an input metric, too. So actually, what I’ve done in my work. with input metrics is I have looked at historical.
data and looked at the relationship between one of my input metrics like around like say like the quality of our AI outputs and then one of our North Star metrics, like the time saved. Um, from someone using this feature, and I plot those together after I do like some kind of like regression analysis to, like. Basically, uh… control for everything else going on, so I know the real relationship between those two. Then I see, like, okay, what’s the point at which like increasing quality? which is my input metric. Suddenly gets a spike. like. Like more increment, for every incremental. increase in quality. I get even more and even more incremental increases in my like North Star metric.
That’s like, I think, a reference point around like, oh, someone’s having the aha moment right then. Um, and then you can follow that all that kind of relationship all the way back, even to see, like, some sort of like. deceleration when people no longer get like incremental gain. If you put more into the input metric, and that’s like that might be a signal to like, Oh, let’s look at a different input metric to move to more star. But yeah, I think. So I guess. To go back to your question, what do you think about the aha habits? I think they may be the key milestones along an NPET metric. that have significant increases to the North Star metric.
So that’s like what teams might go around like, okay, this. quarter, we want to get this input metric to X, because we know that means we’ll hit that aha moment, or that means we’ll hit some deceleration of like the ahas or whatever, and we should focus on something else. Does that make sense, Attendee? Cool. What else? Got 17 minutes. I’ll let it be awkwardly silent when in here for 2 min as people are, if people think of stuff, none can drop. What else is going on? Is anyone here in California? So I live in Tahoe, and it was straight up dirt on the ground 2 days ago, and now I have 4 feet of snow that I’m looking out my window. Sorry, I’m just trying to kill a silence.
But man, that’s crazy storm. Where did I learn to apply these analytics frameworks? Oh, before your first data-related role. Oh, yeah. I did not. Yeah, I didn’t learn these. And in school at all, I learned through a lot of failure early on in probably the first five years of my career where I was like. doing analysis and like analysis that like Vps or directors or someone asked for, and. work on some for, like, 2 months and deliver it to them, and they’re just like, oh, that’s interesting. And then nothing would happen.
And it’s just, like, I think I got sick and tired of it, because it was like, okay, uh… my stuff’s not getting implemented, but then that also impacts stuff like your promotions and raises and all that stuff, because it’s like, even if you’re working hard, even if you’re smart, like, if you’re not having… impact, which I think a lot of these analytical frameworks basically like force you to. do analyses that you know action will come out of and impact will come out of and like it’s really hard to progress in your career, so… I don’t know, like I think part of it was me getting like sick of it and like switching to a different mindset. I think it’s really helpful to have mentors.
So like I’ve worked with high who earlier, obviously, who did this presentation for about six years now. She’s like really good mentor to me. I had some other. managers and mentors, and also like colleagues. I think a lot of it’s just like learning on the job and with other people, but. If… if, like your whole goal is just like I got to set myself up for making it actionable. Like, what are the frameworks that can get me to do that? Um… That’s the best way of learning. But it’s it’s really, I think it’s pretty hard. to learn beforehand. But if you know this stuff, a lot of it’s going to show up really well in your interviews, like, all these frameworks, it’s just, like, ways of thinking.
would resonate very well in the in the interviews. Um, you know, Sravya just did something the other day that actually. So she’s doing a session next week or the week after that is like leveraging AI to help with product sense interviews or something. All right, let me find the link for this one, because this one might be good for you. This is another free one. uh… crack product sense. And Alex interviews with AI. Okay. Interview. Right? This is one. So strawbi’s been working on this. Astrabi is a senior manager in data science like she’s interviewed hundreds and hundreds of people over her career.
And she was just, like, playing around with Claude the other day to like help create like, um… like an interview kind of, like, simulation or interview, like. training partner. I think it actually, I don’t know. I mean, I I I think… it’s kind of a nice way to, like, you can, like, you can do, like, stimulate mode and see how it would work through a problem, or you can like test it against yourself. This is just, like, a little fun side thing she was doing, but maybe go to that. She’s going to have some… she’d be able to answer some good questions for you, too. saying hi from Berkeley. Nice. How do you see the future for the dating us? What should we do? Learn?
Should we switch to data engineering or product management? Dude, man, the plumber thing like I’m seriously thinking about it. My my brother was looking into it a few months ago like like being a plumber is actually like. You have to do this whole like apprenticeship thing. It’s actually kind of hard to get into. I don’t know, though. Seriously, like things that we do with our hands like could be very important. How do I do actually envision it, though? I mean, I think product management is a pretty good route. I think we’ll see a lot of people kind of taking on a lot of the skills of the product manager. I think.
I think thinking like a product manager does you a lot of benefit if you’re a data analyst or data scientist working in product like you basically want to like for me. It’s like, I want to basically do a product manager’s job. But I want to be… slightly more technical, and I don’t want to deal with as many meetings, or, like, talking to as many people. Like, I don’t know, that’s what… that’s what I like about being product data scientist. Anyways, I think AI. I think product managers will actually come out pretty good on top there, like, especially technical PMs. I think a lot of product data scientists will kind of, like, they’ll kind of mesh together into, like, technical PMs.
The other thing I was talking to Hai about as we were building this data analyst thing with Claude Code is I think it’s going to be like people, data scientists and analysts who build systems. rather than fully doing the analysis themselves. So if you can build like systems of agents and that can like. solve the right questions for you. I think that kind of evolves into the jobs, like, managing data agents, but it’s pretty hard to tell. If you ask me, like. kind of like four or three months ago even like even like a few weeks ago before this 4.6 thing, I was… I would have been like, oh, there’s a lot of stuff AI can’t do yet, but… I don’t know if that answers your question.
I’m still trying to figure that out, too, to be honest. I think a lot of people are actually. Yeah, nobody knows. But I mean, I don’t know, like… Plummer could be a good route. I was like firefighter right like in Tah like fires aren’t going out of style like more fires every year. So. They’re gonna need more people in that industry. Um, what else? What other questions? Got 10 min left. Awkward silence. trying to think of anything. talk about. You guys watched that new Game of Thrones show? that’s coming out on HBO. Been watching that. Pretty good. last episode was pretty crazy. I chose anti-gravity because it was free. But I mean, honestly, so yeah, I guess here was my workflow.
I was using cursor. at 1st and with Claude code also. And then. I kind of liked having them both because… I could talk to… cursor… any modeling cursor, and it’s not eating up tokens like Claude code does. And then I was like, I’ll switch over to anti-gravity, because it’s free. And I have to pay for the cursor subscription. And then over time, though, I ended up just, like, not really caring that Claude code was eating up my tokens and don’t even really use the… Agent anti-gravity anyways, so I don’t know if you necessarily do that, you could just probably do VS Code, but I don’t know. Integrity is pretty good. They’re probably just going to make it better, too. But any any.
any of these products could work like Windsurf or. cursor or anti-gravity, or whatever other ones there are. Other questions? Oh, man, yeah, the Opa Claw thing is pretty interesting. I was just starting to mess around with setting up their night, but I actually… I don’t want to put it on my… own computer. So I’m gonna buy, like, a MacBook Mini or something to put it on. Oh yeah, Docker containers. Good idea. Um, I haven’t set it up yet. I’ve watched some YouTube videos on it. It does, I don’t know, it does get me thinking one of my limitations right now is like I’ll be like trying to work with these agents is like, okay, I got to go do something right now.
that I can’t really be in front of a computer anymore with. It would be nice to, like, have them keep… talking to them and having the AI analyst, like, keep going. Like, if I’m on the treadmill or something would be pretty cool. I was like talking to my phone to telegram and then like. get it going still. I haven’t tried it out yet, though. I did my meeting calendar is, like, getting insane, though, so I at least want to set it up. for that kind of organization around, like. meetings and Gmail kind of stuff, I think as, like, kind of like a personal like. assistant, I think, could help. But I haven’t thinking about it. I don’t know if you’ve messed around with it.
Product-driven companies in Bay Area hire DS because the execs themselves do see clear tangible impact from having a team of data scientists in the company. Or is it sometimes more about having enough budget to take the risk? Hmm. I think it totally depends on the company. I mean, I think definitely, like. People realize that like. to make better and better decisions, we need to have. someone like work with the data to like show us some proof. But we also need someone who is like understands the product and is a business has like some business mindset, so they can like, you know, basically. create all, like, the correct metrics, like I talked about today.
I do know companies that straight up have just been, like. We’re at the stage where we have enough money to buy a data team. Now let’s go get a data team, and they have no idea how they’re going to use data, but they just know, like. People say data-driven, we should do it. And then they like. Hire the wrong people, get a kind of shitty team. So, like, there’s definitely that vibe going on, too. Um, I mean, I think at the end of the day, though, people are like. I want to make decisions and I want to reduce the uncertainty as much as possible so I can make. They’re the best decision, and I can prioritize my decision-making. And that’s a lot of the function of the data science team.
We’ll be interested to see, like… I don’t necessarily think, like, just like a data scientist has to do that. I think like. if people like have access to data that they can trust, and they know how to, like. Make the right metrics and how to, like, apply the framework, then, um… Other folks could do that, too. I feel like that’s kind of like the… the core benefit of having the team right now. Definitely, there’s people that just do it because they have a budget, for sure. What else? Got 5 more minutes. I think. We may have exhausted the questions. Okay, if you have other questions. Feel free to ask them in that Slack community. I’ll post the link one more time. Thanks for joining the session.
This is awesome. You guys are… really interactive and asked really good questions. Um, I had a pretty good time. Hanging out with you all. Okay, so. Here’s the Slack link again. You can DM me there or like drop in the channel. So this is a pretty new slack group. We’re like trying to wake it up right now. We don’t have like a community manager or anything. So we’re trying to figure it out. There’s kind of two main channels. One’s like AI analytics, the one’s AI evals. So if you have like questions around either topics, like just ask them in there. I think broadly, then we can start and see if we can get more people talking about it. But we’ll answer for sure.
And then, yeah, I mean, you can just. DM me on Linkedin. as well, if you don’t want to, like, be messing around with slack, and you just want to talk there. That’s cool, too. Sweet. Well, thanks, everyone, again, for thank you all for staying over 30 min. Really appreciate it. And thanks for all the great questions and being engaged. Hope you all have a great rest of your day, and hope that we talk again soon, and maybe I’ll see some of you next week in that workshop that Shavi is doing around like interviews or the cloud code workshop. See