AI Analyst Lab · Questions and answers

AI analytics FAQ

Answers about using AI for analysis, giving it the right business context, and checking its work. Find a question or jump to a topic below.

Find a question

Type a word or phrase to see matching questions.

AI analytics

Why does an AI analyst need context if it can already query the data?

Give an agent access to good data and a clear question, and it can do a lot. It can inspect tables, write queries, and make charts. Trouble starts when a question depends on something the tables cannot tell it: which definition your company uses, which source you trust, or what a number means for a decision. The query can run perfectly and still answer the wrong question.

Is the data itself context, or is it the input to the analysis?

Data is one type of context, and it is the evidence an analysis works with. An agent also needs to know what a table represents, which records count, which joins are safe, and how your company interprets the result. When an answer looks wrong, check both the underlying data and the instructions used to analyze it.

Do these practices depend on a particular analytics tool?

No. Whatever tools your team uses, people need agreed definitions, trusted sources, a way to inspect how an answer was produced, and a way to test changes. Your tools affect where those things live and how you maintain them.

What if two teams use different definitions of the same metric?

Ask what decision each team is making. For example, one revenue report may include returns while another excludes them. Agree on which definition applies to each use, then record its owner, calculation, and scope. If a question could mean either one, the agent should ask before calculating.

Business context

Choosing where knowledge goes

What belongs in company-wide guidance, a domain guide, and a semantic model?

Put broad rules in company-wide guidance: what the business does, data conventions, recurring mistakes, and preferences that apply across many questions. A domain guide holds the detail for one area of work, including definitions, sources, joins, filters, and how to interpret results. A semantic model holds established measures, dimensions, and relationships that should be calculated the same way each time. The guide can explain when to use a measure from the model.

Can a small team keep everything in one Markdown file?

Usually, no. A rule about one subject does not need to follow every conversation. Keep the short, widely applicable rules separate from detailed guidance for particular topics. Then the agent can retrieve what the question needs without carrying every definition at once.

What should go in company-wide context?

Start with the business, where production data lives, conventions that apply across datasets, mistakes you see repeatedly, and how answers should be checked and presented. Keep it short enough to be useful for almost every question. The detailed definition of one metric belongs with that domain or in a semantic model.

How do I make a domain guide easy for an agent to find?

Give it a clear name and short description using the words people actually ask with, including common names and synonyms for the subject. Put the detailed instructions in the body. Then ask a representative question and inspect whether the agent found and used the guide. Publishing it alone does not prove it will be retrieved.

Definitions and decisions

What should I document for an important metric?

Write down who or what is counted, what qualifies, the reporting window, exclusions, source, and calculation. Add the checks that make a comparison trustworthy, such as data coverage or a change in tracking. If people will break the metric down by team, country, or product, explain how each record is assigned and how the pieces relate to the total.

When is a semantic model worth building?

When a calculation or relationship is established and people should reuse it. The model defines the source, grain, measures, dimensions, and relationships. An open-ended question such as “why did this change?” may still need a domain guide and additional analysis. You do not need to formalize every possible question before someone can investigate it.

If the model has the calculation, what does the guide add?

The model calculates. The guide explains which calculation to use for a question, which comparisons make sense, and what caveats matter. That distinction lets you maintain one approved calculation without trying to fit every business instruction into the formula.

Where should a method for “why did this change?” questions live?

Usually in a guide for the subject or type of investigation. It can tell the agent to check the definition and data coverage, compare the right periods, and look at relevant groups before naming a cause. Keep the method open enough that the question can lead somewhere you did not anticipate.

How should we handle a metric that includes an estimate?

Keep measured and estimated values distinct. State how the estimate is made, which inputs it needs, where it can be used, and what to do when an input is missing. If the estimate only makes sense at the company level, do not divide it among individual users or segments. Label the result so no one mistakes an estimate for an observed count.

Should an agent ask for clarification or send a question to a person?

Yes, when the answer depends on a term that has more than one meaning or a rule nobody has approved. For example, “exclude internal users” sounds simple until you ask which accounts are internal. Give the agent the approved rule, or tell it to ask. A human owner should make a new business decision; the agent should not invent one.

Trusted sources

Choosing trusted data

How do we tell people and agents which tables to trust?

Describe what each source contains, who owns it, and whether it has been reviewed. Mark approved sources clearly and remove retired or exploratory sources from the agent’s default choices when your tools allow it. Permissions determine what a person or agent can access. After changing these signals, inspect which source the agent actually chose.

What if existing documents and data definitions disagree?

Check the dates, owners, populations, and decisions each source served. One may be old, or both may be correct for different uses. Resolve that with the people who own the definitions before turning either claim into shared guidance. Record which sources are approved for which use and mark their status clearly, so the same conflict does not return in the next analysis.

Reusing what exists

Do I need to copy a definition from another tool into a guide?

If an approved calculation already lives in a maintained model, point to it. Put the business instructions the model does not capture in the guide. Copying the formula into several documents creates several places that can drift apart.

If I turn a company document into a guide, will it stay in sync?

Drafting a guide from a document is usually a one-time copy. The original may remain available through a live connection, but that does not automatically update the copied guide. Give the guide an owner and a link to its source. Decide who checks for changes and reviews updates.

Can a past notebook or analysis be useful context?

Yes. It can show the original question, sources, method, and reasoning. Check whether it still applies and make its scope clear. A useful past analysis is evidence to reuse; a one-off result does not automatically become the official metric definition.

Reviewing changes

Why keep version history if we have the current definition?

When a number changes, people need to know whether the business changed its definition. History shows what changed, when, and why. A diff gives reviewers something concrete to discuss; an earlier version gives you a way to inspect or restore the previous rule if the change causes trouble.

Retrieval and evals

Finding and applying context

What is the difference between retrieval, routing, and RAG?

Retrieval is finding relevant material. Routing is deciding where to look or which method to use for this question. RAG means the agent looks something up before writing an answer. These terms describe steps you can inspect. Finding the right document is only one step; the agent still has to apply it correctly.

The right guidance exists. Why did the agent miss it?

First inspect what the agent actually read. If it never found the guidance, check whether the title and description match the words in the question. If it found the guidance but missed an approved calculation, make that connection clearer. Ask the same question again and inspect the next run.

Can an agent find the right guidance and still miss an instruction?

Yes. Look at the query and answer, not just the list of sources it opened. If it missed a rule, make the instruction clearer or put an established calculation in a semantic model. Test the original question and a related one before publishing the change.

If an agent flags missing context, does that mean a document is missing?

It means the conversation needs a closer look. The question may be ambiguous, the rule may be absent or missed, a source may be inaccessible, or the data may not be enough to answer. Read the question and the agent’s work before editing anything. Repeated problems across conversations help you decide what to improve first.

What should we check before changing context?

Open the work behind the answer. Which source, definition, query, filters, and reporting dates did the agent use? Did the data cover the period? Find the step that failed, change that part, and ask the question again. A wrong table and missed guidance need different repairs.

Testing answers

What is an eval, and how is it different from trying a question once?

Trying a question once is useful while drafting. For an eval, write a case with the question and what a good answer must do, such as use a particular definition or source. Add cases to a suite and run them against the current or proposed context. Read the answers and methods, then rerun the suite after a change to catch regressions.

What should our first eval cases cover?

Use questions your team already cares about. Include a metric with a known value, a definition the agent should state correctly, and a question that tests a method or rule. Add a real failure you found in someone’s analysis and one question that already works, so a change does not quietly make it worse.

Is checking the final number enough?

No. An answer can have the right number for the wrong reason. Check the source, population, filters, and calculation. Sometimes the correct response is to say that the data is incomplete or an estimate cannot be made yet.

Are user feedback and eval cases competing ways to measure quality?

Use both. Real conversations and people’s feedback show what they tried to do and where they got stuck. Evals test whether the agent applies a known definition, source, or method on questions you can rerun. Start with questions people actually ask; check the answer and its work, then track consistency, time, and credits where those matter to your team.

Working with a team

Who should write and maintain this context?

The data team knows how the sources and calculations work. The people using the answer know what decision it supports and which exceptions matter. Bring both into the review, name an owner, and revisit the context when the business or data changes.

Where do we start if we already have years of models and documents?

Start in phases with a question people are asking now. Find the existing model, document, or past analysis that could answer it; check its owner and whether it is still current. Resolve conflicts with those owners, then put the approved rule in the right place and test the question. Repeat with another question and source. You can build useful shared context from the assets you have without cataloging ten years of work first.

Is there a plan we can follow over the next month?

Yes. Our 30-day plan starts with a data teammate, brings in someone who uses the answer, then connects existing company knowledge and prioritizes what to improve next. You can use it with the analytics tools your team already has.

Can AI Analyst Lab help our team put this into practice?

Yes. We work with teams on AI analytics beyond this context workflow, including how to make analyses useful, share the work, and check the answers. Book a conversation and tell us what your team is trying to do.

Need help with AI analytics at your company?

AI Analyst Lab works with teams on their analytics workflows, data context, and how they check AI-produced answers. Tell us what your team is trying to do and we can discuss where to start.

Book a conversation