In February a product designer posted on r/UXDesign that an interviewer had asked, "What's the North Star of our business?" and they froze. Someone on r/StartupsHelpStartups had a ready answer instead. Their product is an AI agent, so they count conversations users start: "The higher the number, the healthier the metric." That count rises when the agent helps someone. It also rises when the agent confuses someone into asking three more times.
The metric stands in for what the customer gets
You cannot measure directly whether customers got what they came for, so you pick a count that tracks it. Trouble starts when the team treats the count as the goal, because then anything that moves the count looks like progress.
At one company Shane worked for, anyone who opened an email from the company counted as a weekly active user. iPhones can report an email as opened on their own, so, as he put it in our June workshop, "if you just send more emails, suddenly your weekly active users go up." Hai Guan's test for this, from the metrics week of AI Analytics for Everyone: "If a big number going up can coexist with the business getting worse, you have a vanity metric."
Seven questions to test a candidate
- Does it capture customer value?
- Does it reflect your strategy?
- Does it lead revenue, or trail it?
- Can your team move it?
- Can someone outside the data team understand it?
- Can you measure it?
- Is it free of vanity?
Questions 1, 3, 4 and 7 are deal-breakers. Miss any one and the candidate is out, however it does on the other three. Revenue fails question 3. Daily active users usually fails question 1, because a login counts whether or not it did anything for the person. The example Shane used in the workshop was Airbnb's nights booked: "a booked night is real value. Someone's actually staying somewhere. It leads the money instead of trailing behind it."
A worked example on NovaMart
NovaMart is the made-up online store we use in our courses, with a few years of orders. In the June workshop we chose a North Star for it live.
Weekly active buyers sounds like the obvious candidate. It is a headcount, and it causes a problem one step later: the first input you would put under it is the number of buyers, which is the North Star under another name. Weekly completed orders avoids that. It splits into three inputs that are each something different: completed orders equal buyers, times orders placed per buyer, times the share of placed orders that complete.
We ran it through the seven questions with our open-source North Star tool. It passed 5 of 7, with no deal-breaker missed, and the tool called it weak in two places. A returned or late order counted the same as a good one, so the suggested fix was to count only orders with no issue: on time, not returned, not escalated. And nothing in it was specific to what NovaMart sells.
Then we asked what drove it in 2024. We reran the calculation for this page and got the same numbers. The three shares of growth add up to 100.
| Per month | Jan to Dec 2024 | Share of growth |
|---|---|---|
| Completed orders (the North Star) | 1,037 to 6,215 | Grew 6.0 times |
| Buyers | 985 to 5,833 | 99% |
| Orders placed per buyer | 1.22 to 1.26 all year | 2% |
| Share of orders completed | 83.8% to 86.6% all year | -1% |
Almost all of the growth came from more buyers. People did not buy more often, and checkout completion did not change. The average order fell from $80 to $73. Only 34.9 percent of buyers ordered more than once in the year. The Plus membership, meant to bring people back, was on 3.4 percent of orders, and Plus buyers averaged 1.52 orders in the year against 1.48 for other buyers. Shane's read in the session: "you're growing by kind of, like, renting new customers. You're not keeping any of them." The headline number looked like a great year. The input to work on next is how often people buy again.
Checks before you commit to it
Check the arithmetic. The inputs should multiply back to the North Star. Hai's rule from the metric tree lesson: "If your children don't reconstruct a parent within rounding error, then one of your numbers is wrong."
Check that it leads. Line the candidate up against revenue by month for the last year or two. If revenue turns before the candidate does, you have picked a lagging number.
Pick a guardrail, a second metric that catches the team gaming the first. For completed orders it is average order value, since discounts can raise the order count while each order is worth less. Hai's line on this: "The pair is the metric."
Write the definition down where people and AI tools both read it. "Completed" has to say what happens to returns and cancellations. An AI analyst asked for a count without that will pick a definition for you: on the same file, four fresh ChatGPT chats answered 99, 99, 2,268 and 4,633 active customers, using three definitions between them. Why AI gives different answers has the full test. The written version is a metric contract.
Doing this with AI
The North Star tool runs inside Claude Code and is free. /north-star audit grades a candidate against the seven questions and refuses DAU or MRR with the reason. /north-star drivers splits the metric into inputs on your data and reports which one moved. It will not decide what your customers value or which input to work on next, and its numbers need the same checks as any AI analyst's.
The June session is recorded with a full transcript: Develop North Star Metrics with AI. On October 7, Sravya Madipalli runs Metrics 101: Define a North Star with AI, free and live on Maven. If you are a PM choosing one for your own team, what to do yourself and what to leave to the data team covers the rest.