Hai Guan: Cool. All right, welcome again. So today, we’re talking about defining North Star metrics with AI, specifically. The theme of this, if you’ve been following us, previously or have attended previous Lightning lessons, we had similar topics on North Star Metrics and how to define really useful ones, prior, probably, like, a few months ago, when we did our last one. This one is a little bit different in that we’re gonna talk about How to choose a North Star metric on purpose. So, specifically, starting with the premise of what is your promise to your customers, and we start from there. I think there’s obviously for folks here, probably a common pain point that you’ve seen, even if you’ve not experienced it in your daily work. daily work, is that it’s really challenging to define a North Star metric, and even with AI helping you with that, defining something with clear intention may not be as straightforward, and certainly not as, not as simple as
just picking something and go run with it, because the consequences is typically pretty big in terms of what guides your business operations, strategy, and roadmap. Let’s see. So probing question before we get into the content. Where did your team’s North Star come from? If you want to drop in the chat, just number-wise, one for… we chose it on purpose. Two, if we inherit it from whoever was there before. Three, if we copied it from another company that’s doing similar things. Or four, we just don’t have one. Nice, lots of ones and fours, wow, the extremes. Okay. Nice. So, a lot of intentional choices. But also, a lot of… just haven’t… really looked into one at all. And that’s okay. You know, for folks who are number one, congratulations, you’re good to go. But hopefully the content is still going to be helpful in helping you evaluate whether the metrics that you’ve chosen actually reflect whatever you guys are,
working on and where your company or team is headed. And for folks who don’t have one, hopefully this session would inspire you to take on the exercise of coming up with one, because typically, when there’s a KPI, when there’s a goal, it’s a good practice to have. Oh, and I’m gonna talk a little bit about the format of today. We’re gonna go through some slides. We might not have enough time to finish all of it, but that’s okay. You guys will get the follow-up materials and stuff like that. We’ll do a little quick demo in between, where We’re not going to have anything fancy set up or anything. It’s just a Claude chat interface. But I wanted to show you some of the ideas that you can work with LLMs to refine your thinking and use it in a way that actually helps you arrive at the most important point, which is the human judgment at the end, where decisions actually do matter from humans, not from AI.
So we’ll… we’ll do… we’ll do all that. Cool. So… You know, like, I think… I think this is probably not a surprise to a lot of folks that many North Stars are actually defaults and they’re less so on decisions. And even if there are decisions, you know, based on a poll that I just flashed in the previous slide, probably a lot of them were you know, like, oh, some other company is doing it this way. And so, you know, that’s… that’s totally… that’s totally normal, and something that we’ve observed across companies in the industry. So, I think most… most people underestimate just how few metrics were really chosen from that perspective. Someone in our cohort actually brought this up last week, that, They work with teams that have never defined a metric at all, and usually when nobody, you know, like chooses, or nobody does the intentional choice of
coming up with a thing that really represents what they do. You know, like, still, something is gonna show up anyway. Like, hey, we need metric, probably some sort of top-down mandate or something, and Someone’s gonna someone’s gonna pick something. Perhaps it could… it would become… it came with the dashboarding tool itself, and stuff like that. Or maybe, like, some last… some other teams use it, and so, you know, they would just use that. So that’s kind of like the… the… a lot of the defaults for how metrics, you know, like, exist, in a company or in an organization. A chosen one. Would really look like, you know, someone has gone through the process of saying, what is the customer value, like, starting from first principle? whatever the product, the services, the goal of a given company, team, or organization, like, what is the customer value that it stands for? And, you know, what action does it actually make sense to be counted for
coming up with, like, for a representation of that value, and then what unit is it, and, you know, what time frame, and stuff like that. So there’s a lot of, like, actual discussion to Going back to the drawing board to be like. Hey, these are some of the most important aspects that we need to consider for a metric to be truly representative of what our company is driving for our customers. I think the tagline at the bottom of this slide is that you can always ask a chat bot, for example, like AI, to give you, hey, can you give me a North Star metric for my company? And then, you know, tell it, like, your company. It would, you know, it would pick something for you, but it really doesn’t give you, sort of, like, the depth of the considerations, the decisions you need to make, the… the guardrails that you have to think about in terms of, in terms of what… what ultimately should be picked. And those are really important, because this is going to be one number for
your company or your team to actually operate against and so the cost of getting it wrong for example is going to be pretty high. Now, there’s not going to be objectively right or wrong metric, or anything like that, but If… but for something that consequential, the recommendation that we always give is that the human should always be in the loop, and be in the driver’s seat, and really drive the whole process. Whoops. Okay. And, we’re gonna talk about, you know, like, this very… common, metric, or, you know, like, hopefully everyone has heard of some variations of this, weekly active users, right? Like, that’s, that’s very… Pretty common metric across many, many companies use it, certainly in the social media space or the technology space. A lot of companies measure their… the performance of their applications or their services based on this metric.
Now, if we break it down a little bit, this is actually, you know, obviously 3 words. There’s actually 3 decisions that’s baked into this metric itself. So, for example, weekly is saying. Like, the value of that… you know, assume this company adopts this as their North Star, weekly actually says the value that they provide comes around once a week. So that’s the kind of, like, timeframe that they really shoot for, and that’s appropriate for what they’re doing. Active is also pretty deliberate in that certain actions would count as getting that value. Like, there’s some representation of value in whatever the user does in that application or in that service that would let this company count it as active. And then users is the unit of obviously like, you know, like what the who the value is provided to, right? Like, you know, it could be an account, it could be a household, it could be a team, it could be any way that you shape or form that you count it as.
The idea, when we zoom out a little bit, is that every single word in a metric actually do have a… do have and should have a decision behind it, and not just be like, oh, you know, like, Pinterest counts weekly active users, so we should do that too. And we’ll get into a bit about, like, where do we start and things like that. the worst that can happen is that, you know, some other people count it this way, so we should get a variance of that, and then that actually becomes your metric. But then, when you try to move it, or when you do the thing that you know is best for your customers. that this metric either moves or it doesn’t move, or it moves regardless of what you do, then that’s probably not a good one to continue with. And so that’s what’s called vanity metric. And, you know, like, it would look really obvious when we see some examples, obviously, but during the
during the, during the, during the process of actually, picking, choosing, or when there’s no process for doing that, it might be the path of least resistance. So, something that we caution against. All right, cool. So where do we start if not from somebody else’s metric or maybe like from, you know, industry common common form factor of a given metric? The really helpful way to start is, again, zooming back to first principle of why does your product and services exist? So, from the customer perspective, what did we promise them? That would be a really good way to start. And then you follow this lead down the path towards something pretty specific. So, for example. The first step here is the promise that we just talked about. Like, what is it that we’re promising the customer that we’ll offer them value for?
The second one is the moment. So when they received it. what should happen like, what? What? What? What says that this promise was there, for example, like the the moment that can that can best proxy that prom that promise. And then what action? Like, what is the observable event that marks that moment? The fourth is the units, like, who… who is this person? Or, like, who do we count as value delivery, right? In the case of weekly active users, the users is where this is. That’s… that’s kind of, like, the… the unit there. And then fifth is at what cadence, right? Like. How often is the value coming around? Like, how often are you delivering that value to customers in order for you to be, like. hey, that’s, that, that, that fits what we’re here to do, that kind of stuff. So these are all, like, you know, none of these are technical things that we’re working out, right? Like, they’re all just, they’re all discussions, they’re all brainstormings, they’re all just…
Debates to, to really be very clear about. this is where we’re headed, and we are building out our roadmap strategy to go after this. And the whole entire exercise is really to, you know, like, distill all those down into something that hopefully, at the end, we can measure it as closely as possible. Alright, the next few slides, I’m going to walk through a few examples, and then we’ll do a demo around one of these examples. So… Imagine you have a language learning app. And the promise is a daily habit, like a daily habit of perhaps every day, if you spend some time learning about a different language, the idea is that. you would be able to learn that language fluently at some point in time. And so the promise here, for example, is the daily habit, like build a daily habit for learning a language.
what is the moment? The moment is when a lesson is finished. So perhaps there’s some curriculum in that app on a daily basis where you get to finish a lesson, and that kind of stuff. And so the best proxy for building a daily habit is that you completed a lesson, on… on… Like, that’s the action too. Like, the moment is when you’ve finished a lesson, and then the action itself is when the lesson is completed in the application. The unit could be the learner, right? The person who’s looking to learn using the app. And then the cadence, obviously, if we… if, if… if we at this company is promising people to build a daily habit, then we would want to make sure that they actually do have the cadence of a daily, you know, lesson completed, completion,
to reflect the the thing that we’re striving to to offer. And so in this case the daily active learners, if you kind of, like, piece them together, is really the… could be the… a good top contender for the North Star mentoring for this application, because you’re pretty clear around all of these promise, moment. Action, units, and cadence. And so this is very much a fit for the for the broader concept of a weekly active user. So that’s totally cool in in in this this use case. Now, example two. Imagine you have a hiring platform for small businesses. So the idea is, you know, like, the promise for this company, for example, or for this platform, is to fill the role. So, helping small businesses to hire and So, the promise is, obviously, if you have a role to fill, open position, the value of the platform is to fill that role. And then the moment could be a hire is made, for example.
The unit is the employer. not necessarily the recruiter. So, like, you know, for any given company, maybe that’s what you care about, instead of, you know, were the recruiters hired or not? Like, that’s not the point. And… Perhaps the cadence is you know, some sort of a few times a year, right? Just making things up here. Now, if we try to… Fit weekly active user, for example, into this, into this company. it probably won’t really make sense, right? If you think about it a little bit, because what happens if a success happens here? Like, what happens when a promise is filled? Which is, a customer makes a great hire, and so if they made a great hire, then they have no use for this platform anymore. Like, they would stop logging in, so your weekly active users would, by definition, drop, because
they’ve been successful at what they’re looking to do, and you’re successful at providing that promise and fulfilling that promise. And so, a metric like weekly active user here would not really make sense for that… for that… for this intention. And so, if you optimize for this, you’ll be… effectively optimizing for the wrong thing. And so what’s you know, like, what? What’s a better alternative? Well, there could be… there could be a few, right? Like, it could be something like how many rows were filled on some sort of cadence. That makes sense for a hiring timeline. Now, that’s a very slow and, you know, like, like an infrequent sort of metric. But you can always find earlier inputs that also… that would predict a higher, and that would be totally okay if you move upstream from that. And so, the idea is not to have the perfect metric.
The idea is really to think about all these dimensions where as much as you can lay them out, then hopefully there is a kind of like a clearer sense of what may or may not be appropriate for the for this company or for this business that you’re looking to have a KPI for. Cool. All right. So the third one here is let’s imagine a tax preparation application and they report weekly active users. Do you think this is a good North Star metric? 0, not good. 1 is good. Okay, everybody’s 0. Why? Why is that not a good metric? Attendee says seasonality. Attendee says, taxes aren’t done weekly. Attendee usage during part of the year. Right, yeah, these are great points. Yeah. Exactly. You know, in the US, you file taxes once a year, and so in whatever, like, like, when you’re out of that season, then if we measure this based on… if we measure this business.
Based on, based on weekly active users, then it would look like this business is, just doing terrible, almost like, you know, like, majority of the year, which is not exactly what a tax prep application is intended to to be to be used for. The question is, then, what should it be, right? Like, if it’s not weekly active users, for example, what is that thought process? What could it look like, and how do we actually go through one? That is, that is actually less obvious than it sounds. I don’t work in tax, nor do I work in fintech company, so maybe this is obvious to folks who are in that space. To me, it’s certainly not. And so, what can it be? Well, it really depends on what the app promises, right? If we go back to
The earlier slide around the five steps of consideration. So what is exactly… what exactly is the promise of the tax? It’s a mouthful. It’s kind of hard to pronounce. The tax app, let’s call it. You can have a promise of done right, no stress, for example. Like, that’s the promise that the app is making to customers. it could be get every dollar you’re owed right like that’s perfectly great value prop for for a business it could be done in one sitting so like you know like you don’t have to come back multiple times to correct things and and and stuff like that so it’s like very smooth efficient facilitation of your tax preparation. Or it could be your money handled all year, so it’s beyond just filing taxes in the way that we think about a tax app. It could also be helping you prep and strategize, and also optimize throughout the year.
And Either… or, like, all of these would themselves have very different candidates in terms of metrics. And zooming up a little bit, the moment for when these things would be realized would also be different. So if we, you know, like, the very bottom one here is probably the most intuitive, where if we want People to actually get their money handled all year, then. you probably want them to actually use your app. In that case, maybe not a weekly active user, but probably some sort of monthly metric. In this case, the example would be… could be households engaged each month with the application. Because you do want them to be like, you know, hey, checking with your finances, making sure that you’re on top of things, and stuff like that. So, you know, none of these are wrong.
And none of these are objectively right. It all really depends on the promise. And so, hopefully, this helps illustrate why having that conversation about what is the promise at the very beginning is going to flow through down the stream of making Making it obvious what the moment could be, and also then, what is the North Star that can best proxy that. And, you know, like, all of this is, again, not a technical problem. All of these are human and process problems. Like, you would have to… you would have to have that conversation, you would have to have the, the debate, and then, you know, it also shouldn’t be fast, like, it shouldn’t be a, let’s do it for 5 minutes, and we’ll take one and go for it. Like, there’s probably some sort of alignment, some sort of, deliberations, some sort of
I don’t know, like, artifacts that you would create around, you know, hey, we are confident that we’ve made the best decisions slash choices, given what we know, and, and, you know, over time, you would probably want to revisit if this is still the right guiding principle, or the right guiding North Star for the business. Alright, how are we doing on time? We’re at the half-hour mark, so I think we should be good. What we’re gonna do is… I’m gonna do a little bit of a demo here. It is all just prompts, so no fancy setup, as I talked about. No clock code, no, no… no specialized skills or agents or… or anything like that. So we’re just gonna go through, you know, like a process with AI to help us come up with candidates for
for, North Star metric. So imagine we have, you know, continuing with the tax preparation app, let’s call it Tidy Tax, and this is obviously made up. And, And then we’ll go into, sort of, like, step-by-step of this workflow here, and I’ll explain a little bit once we get the prompt fired up, and we’ll have some interactions with it. I will send out the entire prompt and everything to all of you. After the session today. And, Also, the recording as well, in a couple days, once Maven has a chance to actually process it. So, I am going to… move my clot over. Can you guys see this? Hopefully you’re familiar with something like this, where it’s just a chatbot, like, nothing, nothing fancy. So, ChatGPT works the same, Gemini works the same, if you use those. This is just Claude on the desktop app. You can always go to the web chatbot as well.
Am I sharing my screen? Yes. Do you guys see… Do you guys see it? Okay. Then, Attendee, maybe you clicked on speaker in your Zoom. On the top, there should be some tabs around what you can see and what you can focus on, so see if you can do a full screen or something. Alright, so I am going to copy and paste a prompt in, and then we’ll take a look at what it actually is doing for us. So, just a Claude… chatbots. Again, we’re not in cloud code or anything, just generic, can ask you questions, kind of this kind of stuff. So I’ve just fired off a prompt, and we can… Just go through it together. The prompt is, I need to choose a North Star metric for my business. And we want… Claude to go through a series of four steps. The first one is to actually interview me. So one thing that’s been really helpful with AI interaction is not to just
have, you know, like, get AI to provide an answer on the spot, but more, like, help it get more information off of me that I may or may not have enough sufficient enough of during the prompt in in the prompt itself. So in this case I’m asking Claude for step one to actually interview me one question at a time. So what does the business do? Who is the customer? What do you promise that customer, or what do I promise that customer? What is the moment that they would actually received it, that promise? Who receives it? How often does this value come? And… What is the company trying to prove at this stage? What can it not trade away, and what can’t it measure? And then ask it to, if anything is vague, just push back before moving forward.
And then step two is do the research. So, search for companies of similar flavor, let’s call it. with similar business models, similar value cycle, for each of them… for each one of them, what are they… what metric are they, are they… are they leveraging, and what kind of promise, are they fulfilling off of those? And so this gives us some sort of competitive intel around, hey, what is everyone else doing? So we get a little bit more targeted in terms of the candidate sets, or at least, you know, a research set. And then step 3 is giving the candidates, so 4 to 6 candidates, North Star in a table, and then. things around the promise, what would it actually incentivize my team to do based on that metric, how it could be gained, and when might we be… when might we outgrow that metric itself? One concept here is metrics are not static. Like, over time, perhaps a company is
perfectly reasonable to have outgrown some sort of metric, and that no longer reflects the value that you… that you want to deliver, so be extremely mindful of that. When you set something, like a North Star, it doesn’t have to be forever, and it shouldn’t be, either. Alright, and then step four is, what is your opinion? You know, like, pick one and tell me what… what is it, and what would change your mind? So, now it’s going through the interview me stage, which is question one, what does the business do? So I have. Obviously, I have some stuff ready. Tidy Tax is a consumer tax prep app. Customer is a household with a fairly simple return, mostly W-2 income. For folks who are not in the U.S, W-2 is just your paycheck, basically. Filing their own federal and tax return, blah blah blah.
And question two is, what do you promise? My promise here, like what we saw in the first candidate. is… Assume tax is done right, no stress, and if we could only keep one of these, it would be done right. And then question three is, what is the moment that I actually received done right? So, next response is, when the federal tax return is accepted on the first e-file submission, no rejection, no fixing, and resubmitting. So, it’s, like, accepting on first… first pass. Question 4, if you remember who receives it, so this is the, like, who the user is. And in this case I’ll do the household. Couple filing jointly counts as once. Counts as one. And then how often does the value naturally come around for a household? Let’s call it… once a year. almost all of it between January and April. That is the US tax filing season.
And then the last question, what is the company trying to prove at this stage? What can’t it fade away? And what can it not measure? So we need to prove people trust us enough to come back next season. We can’t trade accuracy for speed. We can see accepted and rejected status from e-files. But we cannot see what happens later, like a letter from the IRS asking to… asking to, to… asking the users to resubmit, for example. So these are nuances. That is. that is within the context of the business, obviously. And so for your company or for your team, it will be obviously very different, and you would have your own nuances and stuff. So this is This is basically trying to cover the, or trying to extract as much information as possible, not just what is possible, but also what is the limitation.
So… Alright. So this is what it comes back with. If you recall, step two is what similar businesses measure. So went out and grabbed some of the, let’s call it competitors, or companies that are doing similar things in this space, and what they actually report in terms of the metric that they really care about. And then what promise was it? Does it? Does it imply based on this? And then some sort of reasoning for whether those metrics could transfer to TidyTax based on our… our goals and objective, and our promise. And so I’m not gonna read all of these, but, like, it looks like it went and grabbed Intuit, H&R Block, these are the two biggest leading tax prep, consumer tax prep companies in the U.S, IRS directly, so the tax, the Yeah, internal revenue service, the tax, the tax people.
and also lemonade. And then it will have, you know, all the… all the different, the different information and reasoning off of those. And then step 3. is the candidates. So, for this company. Tiny tags. These are the… 4, 5? Oh, 6. So we came up with 6 candidates for what a North Star metric could be, and then what is the promise that it delivers? And then also what it would push the team to do, and how can it be gained? And then… how could it be outgrown by the business? And that’s almost like, you know, be prepared ahead of time, like, when we hit that stage. Like, right now, we’re defining this metric, or maybe adopting this metric, but by this point, we should really come back and revisit it. And so the default, likely default, based on our prompt, that Claude thinks is probably accepted returns, because households with the federal return accepted through TidyTax is, is the… kind of like the action slash the moment.
That would be like, hey, we help people get taxes done. And so on and so forth. And then the next one would be something around, like, households whose federal return is accepted on the first e-file submission, per tax season. Right? Like, the first… the first metric here, the promise is only done But then the first try qualifier in the second metric, for example, also tells… also is a proxy for something like a no stress, because, you know, you don’t have to… you don’t have to revise it, or, you know, you don’t have a rejection. that you have to… that you have to make, and stuff like that. And so on and so forth. This one is, like, a percentage metric around what is the percentage of the rate in which the return gets accepted on the first… on the first try. So that’s the, kind of, like, still the… on the done right, promise.
And so on and so forth. Again, I’m not gonna read through all of these, but the idea is that this starts to list out options, and hopefully, they’re pretty tailored towards the actual goal and the promise that your company is having for customers. And… It will say why number one is the default, because the largest company in the category reports that as a major metric in their filings. The quant opinion is. actually first try households. So, metric number two around how many households in their first try, actually got their tax returns, accepted. And then this is what would change its mind around, you know, this may not be a good metric. Things around if the first try acceptance does not actually predict return. And if rejections are sort of like out of control, out of the control of the business.
And also if this behavior, first try acceptance, is already the default, for example, like if 100% of people’s tax submissions are accepted on the first try anyway, then we really don’t need to measure it specifically. So these are all reasonable, right? Things, and these almost give, like, the next step around, hey, we think this could be a good candidate, but we also want to make sure that. We can address these to the extent that we can, so that we can feel a little bit more prepared around whether this is good, or whether it actually truly reflects things in our control. All right. 1041. I’m going to fire off one more prompt here. Which is… Let’s say our CEO wants the North Star to be households that come back the next season. So instead of the number of households with first try acceptance.
The CEO says, hey, it should be… it should be this. because we’re trying to. That’s what we’re trying to aim for like we want repeat customers. We want repeat businesses where people, you know, like not just come not just do do it once with us, but like next season as well. And so this is like a like, let’s do some sort of adversarial review. on, what are the, you know, like, the holes that we can poke with, with this metric. And so this is, you know, we are asking Claude for some sort of, like, you know, kind of, like, pros and cons, and for and against this, retention metric, this practical retention metric. And so, even when we tell Claude that, hey, our CEO wants this, you know, Claude’s, obviously Claude is not going to talk to a CEO, you’re going to have to, but the idea is, you know, like, it would be… it would be giving you its opinion that this is not
This is not the PIC that it would choose, and then there’s a case for it. for why that’s a good… that could be a good candidate, and then the case against it. And so, you know, like, we can… we can read it afterwards, but the idea is, like, you know, hey, if we… if we push it a little bit more on, you know, not just the, hey, I picked this, humans still need the judgment around whether that makes sense, or, you know, hey, if I… if I have another opinion somewhere, if I put it in, can you survive under Let’s call it, like, like, scrutiny. All right. So I’m going to go back to the slides here. You’ve got 15 min. That should be plenty enough. So this is the prompt that we went through.
The interviewing, the similar companies, candidates, and then the opinion. Oh, we did the demo. Cool, okay. So… Yeah, so just a few things to… Just a few things to recap. The default was the returns file, and we saw why that wasn’t a A, a… That would have been an obvious one, but then that’s not really a great one. And then the second one is kind of like the thing that Claude picked. I do agree with that on surface. Like, again, like not knowing, not understanding this business. much in my past experience. That sounds pretty… that’s… that sounds pretty reasonable. So it’s, like, the second one, which is the households accepted on the first submission. And then the CEO had a pushback around it. The next few slides, I’m going to talk a little bit about how to judge some of these metrics candidates to further evaluate if they’re the right fit or not.
Oh, and before we get into that, the thing was pretty helpful that we just… that we just looked at, which is what it couldn’t know, right? Like, when Claude gave us the three things that. that it’s like, hey, I would change my mind if these things are, like, these are assumptions that it’s making. But if these turn out to be true or untrue, then it would change its mind. These are actually really good inspirations for what, for example, like further analysis would be helpful if you have a data team or if you’re in the data team, for example. So, in the… In the example that we had, which is there’s a response from Claude that basically says, hey, I would change my position if first time acceptance does not predict returns. Like if a customer does not come back again.
Then… then I would, you know, like, then this metric probably has no, you know, like, even if you drive this metric, if you move this, if you optimize for it, it probably doesn’t matter. Like, that’s not what people care about. That is actually really a really valid and important question, where you can probably see in the data, retroactively, to see if that’s true or not. And if there’s no correlation, then you can be pretty sure that there’s absolutely probably not going to be any causation at all. And so that may not be the best metric if the promise and the intended outcome for the business is to get people to come back to the app, for example. And then the second one is, is there still room left to improve? So again, if you already hit the ceiling in your metric, then that’s also pointless, right? Like there’s nothing else to improve. And so you’re just, you know, like every week it’s just going to be flat week over week. That’s pretty much pointless.
And then, I think. another dimension that it flagged was, do we actually have control over the rejections? So, if it’s something that the company cannot, cannot control at all, then… Again, like, there’s nothing to optimize for, because you’re not in control of improving the outcome, or improving the value delivery for the promise of your business. So these are all really, really good follow-ups. And the idea is not to, you know, like, hey, this sounds good, let’s do it, but also considering the other sides of the house to make sure you’ve done your due diligence around this as much as possible. Like, you know, smaller teams, smaller companies, less data or less sophisticated sort of like data systems would limit your ability to do something like this. But if you do have the ability, I think these are worthwhile.
All right, cool. I’m gonna quickly fly through these, these couple slides here, because we already did a lightning lesson previously on on some of these. For example, the 7 question checklist of what makes a good metric. That is one one one framework you can use to check. If you know when you have your promise, when you are clear about your intention, whether the metric itself passes the check around, around some of these dimensions that you can feel confident in optimizing for it. So I’m gonna drop the link here. If you wanted to re-watch this, this was back a couple weeks ago, I think, or sorry, a couple months ago, so I’m not gonna fly through that. I’m just gonna skip that here. And then… The other consideration is sort of like, you know, considering what’s a lagging metric, what’s a leading metric.
the CEO’s, CEO’s I guess preference was a lagging indicator, right? Like, retention doesn’t… like, whether a household comes back to use the product again, you have to wait quite a long time to see if whatever you did this season, for example, actually moves that. So you want to see if there’s any leading indicators for, for that. And so your metric should also reflect something that is Timely, that is frequent enough that you can… that you can see the effect of it as you As you make improvements, as you, as you, inch towards fulfilling that promise, better and better. And then another lever, once you have the North Star, is to break them into… into drivers. So… In our case, like, households with a first-time submission, accepted. For example, it could be… you can start to break down into drivers of where you can improve upon the metric itself.
And AI is really good at that in doing this, which is, hey, what is the breadth? So like how many households actually start a return? the tax return, how many get to the moment, if you will. So, start to accept it, and then, and also, like, accept it. The first time. And then, you know, and then you could have a retention metric around what the CEO was talking about. Like, you know, you can have new users. starting to file for taxes on your service, but, like, what percentage of that is returning, and, and that would give you an indication of, you know, if you’re, if the intended business outcome is also, is also fulfilled by this, this leading indicator in the North Star metric itself. I talked a little bit about this, which is the North Star should just evolve as your business evolves, and so the promise
Could be that, you know, like, tidy tax instead of done right, low stress for its promise for customers. at some point it might want to pivot into. Hey? We want to expand to handling your money as well, and And then… You know, like, and then the original metric that we sort of walked through as a candidate, that the monthly number was wrong for the previous promise, could become the right one today, when we decide to… when we decide to pivot to this. So no North Star is set in stone. No North Star should be permanent, and certainly a revisit is always a good exercise. Some common triggers for that would be, you know, if the promise changed. So you wanna revisit this entire exercise, for example. the goal it served has been met. For example, like, if you have a… if you have a metric, originally you had a lot of room to grow, and that really is going to drive your outcome, then perhaps you’ve done a really great job over the course of, let’s say, a year or two, or, you know.
whatever, and you’ve maximized the ceiling there, then it’s totally cool to move on to sort of, like, the next thing that really guides the next phase of the strategy. The other two is, if the team has learned to game it, for example, like, if a metric is gameable, then it’s probably not a good one. So, you want to have a honest conversation about whether this is actually resulting in business outcome or you’re just trying to optimize for something that looks up into the right. And finally, like if it stops predicting, it says retention and revenue, but like really the business outcome that you’re intending it to have, you know, like beyond the just the promise itself, like promise is the value for the customers. As a business, you also want business outcomes for yourself. And so what is it that is being, if this is not predicting that, then maybe even the promise is not, you know, like good enough.
Business, business… I guess, business intention for for for this for this product or service. Cool, alright. So, that is all I have in terms of content. Again, I will send out all the resources and stuff, to all of you. The… what is next is we run these free workshops every single week, and so the next one is going to be around, now that you know how to define a metric, how do you actually codify it so that AI, can, can define it for you, and it can also put it in a format that can be reusable, scalable, that kind of stuff. So it’s… It should be a really fun one. It’s around build a semantic layer, so AI defines your metrics. This is probably a hot topic these days, because many people are asking AI for answers, for data questions, and data requests, and the best
context fixes that we’ve seen is really to be clear about how certain metrics are defined and how you define them, how you store them, how do you make your agent discoverable. How do you… how do you make them discoverable for your agents? So, that’s gonna be free next Wednesday. Attendee already pasted the thing in there, so that’s great. And then in November, we’re partnering the AI Analyst Lab. We are partnering with Hex, the leading AI analytics platform. on enterprise context management for analytics. So that will be a 4-hour workshop for 25 bucks. We’re charging a nominal amount to make sure people are serious when they sign up for it, but we’re partnering together to really demo and showcase how to do context management, like defining metrics, finding metrics for agents, storing context, how to store context, what context is enough.
And how to discover new context around your company, so that it is not a, you know, one-and-done kind of thing, but you have a scalable scalable way of doing it across the board. So this is going to be in November. Should be really, really fun, and everyone gets an extended trial of Hex. and their platform, and we’ll see, you know, what the leading voice in the industry, advise people to kind of, like, build up their context layer. I think that is the number one most Requested, sort of, topic, and the most confusing one, because it’s just so new that no one has a standard, or, you know, everyone’s trying to still figure it out. One… one story I like to tell people is that, I was at a Snowflake conference probably 2 months ago, no, 4 months ago, 3 months ago, and half of the booths that are selling software and stuff in the data
analytics space is how to manage your context. So, when you see that many vendors competing against each other, you know that it is still a really, really new field, but it’s super, super common pain point. And then the last one here is a 5-week course if you wanted to understand and learn even more around, for example, metrics, thinking like a… like a… like a senior data scientist, and building up the… Foundational knowledge of doing robust and high-quality analysis. But not you doing it, you’re delegating to AI, but then it helps you build out the foundations for how to judge if this is good, this is bad, and with this, we’re hoping to help people to be self-sufficient, even if you’re not in the data space, or if you’re in the data space, we distill as much of our
our experience and best practices as much as possible in a course, almost like a crash course, so people can really accelerate their delivery and productivity in the AI world and delegating all the execution pieces to agents. Cool. Okay, so that’s it. I think we’re right at time. Can probably take a few questions if there’s any. But yeah, this is the… this is the content for this week. Hope to see you all in the next one. Let’s see. I haven’t been monitoring chat, so I’m not sure if… Attendee, you have a… you have your hands up. You are on mute if you’re… Oh, okay, no problem, no question. Does anyone else have questions? If you do, feel free to raise your hand. Attendee, you said you had a question earlier. Okay, where is it? Do you want to come off mute?
Attendee: I can also, speak directly. So I was just wondering how do you apply this method, to zero to one phase? You know, in early stages, when you have more, like, a hypothesis about the product market fit, but don’t really have proof yet.
Hai Guan: Hmm, yeah, I think that’s a… that’s a great… that’s a great question. I mean… yeah, it’s certainly gonna be tougher and just be agile about it. I would say, like the promise of you know, the concept of the promise still holds whether you have a product already or not, right? Like, you probably have a hypothesis that by building this product, this is the pain point we’re solving for, for the customer. And what is that pain point? And, you know, what is that moment when that pain point is solved, for example, would give you some candidates for what that could be. Now, this is just what it could be. Once you start launching a product, once you start to, you know, look at you know, traffic or whatever, you’re gonna start to learn pretty quickly whether this is the right problem to solve, whether this is the right promise, whether you need to pivot, and stuff like that. So, like, that metric would almost be like a placeholder, if you will.
But the act of doing it would give you the exercise of… of the complementary exercise of being very clear about, what you’re solving for, for people. I know, especially during times when it is all hands on deck, just crunch time, you know, making the product work, there is the aspect that, you know, people would tend to skim over, kind of like, hey, what exactly are we solving for for the customer? This is one good forcing function for it. Don’t get married to the metric itself. But be very clear about, you know, like, what, what your, what value you’re trying to deliver.
Attendee: Okay, thanks.
Hai Guan: No problem. Hopefully, hopefully that helps. Yeah, a lot of, again, back to the, you know, like, the more that we work with AI, the more that we’re in the Agentic space now. the more it is that a lot of these are, you know, like, human judgment problems and processes, alignment, influencing, and deliberation, stuff like that, where an AI is just not going to be able to give you an answer, if you will. So, you know, like, more… in this new world, there’s actually more need for humans. Alright, cool. We are…
Attendee: Question. Sorry, this is Attendee. Just wondering about the two last courses, the Enterprise Context Management. This is more for people, I mean, for real enterprises, like, how big the enterprise I mean, you said something already, but is it… is it something like, how… how… or how small can it be?
Hai Guan: Yeah, it can be as small as one person. It could be as big as multi whatever Walmart, let’s call it like half a million workers. It doesn’t matter. Enterprise just means company in this context.
Attendee: Business, okay. Cool.
Hai Guan: Yeah, like, you can always build up your own context from the bottom up, right? Like, that’s one way of doing it, but for. company-wide context management. That’s what this is referring to.
Attendee: Yeah. Right. Okay. Cool. Thank you.
Hai Guan: No problem. Cool. All right. Oh, and Attendee, don’t sign up via this code. We have another code for you since you’re an alumni. All right, cool. Thank you, everyone.
Attendee: Very cool. So where is the where can we get the other links or?
Hai Guan: Yeah, I’ll tag you, I’ll tag you in our… in our group.
Attendee: Okay, perfect. Thank you.
Hai Guan: Cool. Thanks, everyone. See you all next week, hopefully.