How do I know if a drop is worth investigating?

Compare it to the metric's range over several months. In the course example, checkout conversion had sat between 3.8% and 4.6% for six months, and the new number was 3.4%: outside the band, so worth explaining. Also check what you are comparing against. At the May workshop Hai's example was revenue down 12% on a week that had itself risen 12% for a known reason. That is a return to normal, with nothing to investigate.

Can AI do root cause analysis?

It can do the slicing, and fast. It still needs a person to check the cause it names. In our ChatGPT tests we said "The Northeast refund spike is the new courier. Confirm and quantify." It refused to confirm, then attributed "roughly 55%" of the spike to a cause the data had no field for.

What is a metric tree?

A top-line number broken into the numbers that multiply or add up to it. Revenue becomes orders times order value, and orders becomes visitors times conversion. When revenue drops, the tree tells you which branch to cut first. The top reply in the r/analytics thread was right that building the tree is the hard part: every branch needs a definition people agree on.

Why did my metric drop?

Someone with nearly 20 years in product and marketing asked r/analytics in June how to find out quickly why a metric moved. Their experience: "whenever a metric changes, people freak out and start hunting for the root cause." Some blame the market, some dig through email, some ask IT what got deployed. "Usually Google Analytics tells you that something happened, but not what happened."

The replies had the right pieces: a tree of the metrics under the top number, a daily trend, a log of what changed and when. One asked the question we would start with: "Did someone change how the metric is calculated? Is the latest data actually loaded?"

Check the data before the business

Hai Guan teaches this in week 3 of AI Analytics for Everyone with a story from NovaMart, the fictional online store the course runs on. Checkout conversion drops 20% in a week. The analyst runs the query, the query is right, and they post the number to the team channel. Three hours later data engineering replies: the pipeline dropped mobile web traffic for 48 hours. By then the PM has briefed her director and roughly 40 people are debating a drop that never happened. In Hai's words, "The SQL was right, the number was wrong because the data underneath the number was wrong."

He runs five checks before trusting any number, and each takes about 30 seconds:

Volume. Is the denominator normal? In the story, checkout sessions were 22% below their four-week average, a red flag before the rate is even computed.
Range. Is the value inside its usual band?
Sum. Do the parts add up, and is each part normal? Mobile web usually brings 38,000 to 42,000 sessions a week. That week it brought 3,000.
Trend. Drift or cliff? Changes in how people behave tend to drift. A cliff that starts at one hour and stays points at a release, a pipeline or a tracking change.
Source. Is this the right table for the question, and was it healthy during the window?

The definition gets the same treatment. If someone changed a filter, or the number came from an AI analyst that picked its own meaning of active, the metric can drop with nothing in the business moving. Ask for the definition and the SQL behind the number before anything else. Our page on why AI gives different answers shows how far one question can spread.

Cut the drop until you can name who dropped

The next lesson uses the same 20% drop, and this time the data is fine. Hai's rule for this step: "You do not earn the right to ask, why did this happen until you've earned the right to say this actually happened."

Then cut it three ways, in order. Time first: conversion goes from 4.1% to 3.3% at 2 PM on a Tuesday and stays there. Segment next: desktop is flat at 5.2%, mobile falls from 3.4% to 2.1%. Then the funnel step: every step is flat until the one from delivery details to payment, where mobile users leave. The question started as "why did checkout conversion drop?", which by Hai's count has roughly a thousand possible answers. It ends as "why are mobile users leaving between delivery details and payment since Tuesday at 2 PM?", which has about five.

At the root cause workshop in May, Hai added how to pick which dimension to cut first: the one that explains the most of the change in absolute terms. If revenue is down $5 million and one region accounts for $4.5 million of it, go there, even if some small acquisition channel is down 80%.

List causes that fit all three

A cause is worth testing only if it explains the timing, the segment and the step at once. In the course example the first guess is the new checkout design. It launched six weeks earlier, so it cannot explain a cliff at 2 PM Tuesday, and it goes off the list. What stays: a release that broke the mobile payment form, a payment processor throttling mobile, a third-party script timing out on mobile, a promotion that ended Tuesday.

Test each one. A mobile release went out at 1:45 PM. Processor errors are flat. The payment page's third-party script went from 400 milliseconds to 3.8 seconds at its 95th percentile, on mobile only. The release had added a fraud check script that blocks the page while it loads, and on a slower phone connection people take a three- or four-second freeze for a broken page. Desktop, on faster connections, did not notice. The test for whether you are finished: "If you cannot name a mechanism, you don't really have a root cause yet, so keep going."

The write-up is three lines: the cause, the evidence, the recommendation. Naming the script gives engineering a smaller fix than rolling back the whole release.

Where an AI analyst helps

The slicing is the slow part. Shane once spent two months at Nextdoor on an investigation into why sessions were down, circling back to new dimensions as more data came in. An agent can check several hypotheses in parallel with sub-agents and keep the findings in one place.

At the May workshop Hai asked Claude Code, running our open-source AI Analyst repo, for a root cause analysis of a June spike in payment support tickets on the NovaMart data. It confirmed the spike first, noting that payment attempts had grown smoothly over the same months, so the spike did not come from more people paying. Then it drilled through the dimensions and named a defective iOS app release, version 2.3.0, with tickets falling back to baseline after the fix. In a February session on the same data it had checked experiment assignment first, ruled it out, and found every one of the 2.3 payment tickets came from iOS.

That dataset has the story built into it, so the agent was looking for something that was there. Real drops are messier, and the agent will name a cause with confidence either way. Hai's comment on his own demo was that "we still have to check the work". Run the five data checks on its headline number and ask whether its cause fits the timing, the segment and the step. Our guide on checking an AI data analyst's answer has the rest.

If the drop came right after a launch you tested, read how to run an A/B test for the guardrails that should have caught it. If the explanation has to reach someone who can act on it, getting people to act on your analysis covers the readout.