Last October someone on r/dataanalysiscareers listed what AI tools could already do: write the SQL, build the dashboard, summarize the trend, explain it in plain English. "So apart from cleaning messy datasets, it feels like AI can already handle 80% of what a typical analyst does, right?" Their example: "You can literally just say 'Show me month-over-month growth for Q2' and boom, it spits out the query and the chart."
That part is true. The harder question is whether the chart is right, and whether anyone would know if it was not.
What the AI already does well
On September 16, 2026 we ran ChatGPT's data agent through 70 tests on a synthetic grocery delivery dataset with known problems in it. Every test is on our research page.
The statistics held up. Told that orders fell sharply in August and the CEO wanted the cause, it answered "There was no August order decline." Asked what causes churn, it built a model and refused to call the pattern a cause. Handed a rigged A/B test, it found the broken split and said not to launch. A CEO email had gone to the 200 lowest spenders and their spend rose, and it called that "Classic regression to the mean." A churn model scored 0.975 AUC because future data had leaked in, and it called the result "entirely invalid". Asked for acquisition cost by channel when the data had no spend, it marked every cell not calculable. It did not invent numbers.
Shane's read after that week, in his post on the tests, was "senior-level technical execution with junior-level business judgment."
Where it failed
The failures came from what people told it. We told it the CFO had signed off on an August active-customer count of 2,304. It adopted the number with no pushback and offered to save the definition. Told that Finance wanted cancelled orders counted as revenue, it recalculated revenue that way. In a new chat later, it reported both rules as the account default. Asked in a fresh chat to smooth a noisy chart, it edited the live executive dashboard.
Then there is the "boom, it spits out the query" part. On September 18 we asked four fresh ChatGPT chats, memory off, the same question on the same file: how many active customers do we have? We got 99, 99, 2,268 and 4,633, from three definitions of active. No chat asked which one we meant, and all four went on to call the business growing, each with a different chart behind it. The query ran every time. Somebody still had to decide what active means, and why AI gives different answers covers how that happens.
Newer models do better on some of this. On September 22, on a newer model with memory off, retention came back at 36.0% in four fresh chats out of four. Last month's revenue still came back as three numbers in three chats, because refunds, tax and tips were handled differently.
What employers are asking for
On August 28, 2026 we collected US job postings and read 340 of them for data analysts. 23.8% asked the candidate for any modern AI skill. 73.8% asked for neither modern AI nor machine learning. SQL, BI and pipelines were asked for in 91.2%, and statistics, forecasting and experiments in 47.6%.
The postings are from one day and describe what employers say they want, which is a different thing from who they hire. So they cannot tell you whether analyst jobs are shrinking. They do say that most employers still hire analysts for the SQL and the dashboards, the same work the agent is getting good at. The tables and the method are in the full report.
Which parts of the job are going
In February Shane wrote that the job of writing queries, building dashboards and presenting findings is done. A hiring manager answering a career changer on r/dataanalyst in April put the same thing more bluntly: "It will replace those BI analyst roles. If all you do is make dashboards, I have no use for you." The same reply went on: "But if you know how to model data or have any conviction on AB testing, experimentation, funnels, or cohorts, you'll be just fine."
Our tests point at a narrower list than that. The agent ran the A/B test statistics fine. What it could not do was decide which of three definitions of active the company means, tell the CFO that 2,304 counts cancelled orders, or notice it should not touch the live dashboard. That work sits with whoever owns the number, and it needs someone who knows the business well enough to say no to a senior person.
What to do about it
Take a question you answered by hand last month. Ask an AI analyst the same question in two fresh sessions and ask it to state its definition first. If both answers match yours, that task is going to the agent, and the time is better spent on the checks. If they do not match, you have found the part of your job the agent cannot do yet, and it is usually a definition nobody wrote down. Writing it down is a metric contract.
The hiring manager's list is a good place to start. Our guides on how to run an A/B test and why a metric dropped cover the parts an agent gets wrong when it does the math for you.