Shane Butler: Okay, we’re talking about context management. So when a new analyst joins just company, they need to have a lot of the context around like these kind of underlying choices the company has made around. certain definitions of metrics and how to pull them, and the same thing goes for an Agentic analytics system as well. Like, an AI analyst, just like a human analyst, is gonna be able to select a table with some, whatever, financial or monetary values. write SQL that runs and works. It’s gonna be able to, like, return a number. Around revenue. But that number could be based on a definition that our team doesn’t actually use. And so to evaluate whether that number is correct, we have to understand the choices behind it, just as we would if a human generated it. So a lot of those choices are just things we we need to explain before the analysis even starts.
So our new analysts, you know, say our human analysts or AI analysts, they need to understand what we mean by revenue. What if we make that question a bit more specific. That’s a really good place to start. So we could say, instead of saying, like, what’s the revenue last month, we could say, give me net revenue for September, broken down by day. Now we’ve told them a little bit more about what we want back. But they still need to know how our company calculates net revenue, which sources to use, and which, like, orders belong in it. That’s information we’re preparing when we do context engineering. So we’re making the company definitions and analytical methods available to AI, so it can apply them to the work we’re asking it to do. Our question… Right now, supplies the reporting period.
But the company knowledge is going to explain which date field to use for that period. It also explains, like, the exclusions, what not to include in this. If the question leaves a choice kind of, like, open to interpretation, we went away for our agent, just as a human analyst would do too. To come to us and ask us, for… Kind of confirmation or more information around that kind of abstractness, so it doesn’t just guess. So we can explain those things. A lot of the way I like to think about when I’m thinking about context engineering is just, like, how would I explain this stuff in a conversation with my team or with stakeholders. Other people across the company that have opinions about this same sort of knowledge. So a first step is kind of, like, talking through that with people who need to use it.
just helps us start aligning those definitions, regardless of whether you’re using AI or not. This is a really important, part of getting, you know, aligned, reliable numbers that are going to be shared across a company. Then we need somewhere to maintain those explanations. This, I think, is where, kind of, like, the ball gets dropped a lot right now in terms of humans. Like, maybe they happen in an ad hoc conversation, live in a meeting, they’re in someone’s personal notes, maybe they’re in Slack. Hopefully they’re documented somewhere, but that’s not always the case. If you want to make this accessible to AI, maintaining those explanations in a structured way is really important. So if finance changes a definition later on, we want to have somewhere to update it. And a record of how that’s changed over time.
So anyone can access it. So where would we start finding that knowledge. our retailer. So before we sit down and write instructions, it’s helpful to think of where a lot of this information can potentially already live. I kind of gave a few examples of just, like, the sort of, like, internalized places it can live, but there is probably some documented, stuff we can already look at. It’s really helpful to look at what the company’s already using. So, say finance produces a monthly revenue report, and there’s a query behind it showing how they calculate that number. That’s… that’s one source of information. Say the company has some, like, policies around how it operates, documented somewhere. For sure, they would have this in terms of like, you know, what’s the refund policy? There’s another source of information around kind of business rules.
That’ll come in handy when we’re trying to determine how refunds work in a calculation and why. Hopefully there’s a data catalog of some sort. Maybe there’s not, maybe it’s not fully exhaustive. Hopefully a starting place for, you know, data descriptions that explain the order data. And of course, there will be, for sure, code. In the application itself, so, like, say you’re… ordering stuff online, there has to be code to create the ability to order everything online. That code in the application records, you know. the purchases, as well as, like, how those records get processed, even if it’s not stored in data anywhere yet. That is in a usable way for the data team. There’s code that makes those transactions possible. So we can learn different things from each of these. The query, from, like, the finance reporting, that’s gonna show us
filters and the calculation that gets used regularly. The refund policy explains when a customer can actually get their money back. The data descriptions explain what the records contain, and which columns mean what in a table. And then the code. That can explain why some records have already been removed before the report even queries them. So all the upstream kind of transformations that happen before it even gets to this sort of query analysis part. It holds that logic behind what builds the data. These are really good starting points for, context. We still need to investigate whether these choices apply to the new questions that we’re trying to get answered, and confirm them with, like, the human people responsible for them, but it’s a great place to start so you’re not, you know, trying to reinvent the wheel or build something up from nothing.
So for, you know, for your own company. it’s helpful to think about, like, work people already rely on, I think is a nice place to start. Maybe a report that gets used regularly on a weekly or monthly basis. Is a really nice place to begin as you start to think about, you know. How do I start getting you know my feet under me in terms of building some context for a reliable Agentic Analytics system without trying to boil the ocean? So let’s actually… let’s actually pause a moment for a minute here. So you’ve… you’ve… I kind of want us to think through that a little bit, and apply this to our own jobs, so you’ve got something to think as we go through this with our kind of imaginary. outdoor retailer. So in the chat, don’t hit enter yet, but write down…
one question people regularly ask at your question… at your company. One question people regularly ask at your company. And then… With that question, What’s the report or document or query? That you find often helps answer it. So don’t enter it yet, we’ll give everyone, like. 20 seconds to write that down. A question. That gets regularly asked. And the corresponding, kind of, Way it gets answered. You don’t need to share any confidential information. If you don’t want to. Okay. Does everyone have something? Okay, go ahead and hit enter, and let’s see, we have in chat. All right, Attendee, customer success teams always ask about attributed and influenced GMV data. And how that’s derived. What’s GMV data if you wanna write that?
Attendee: Gross merchant as well. So this is typical for a SaaS company, an end-user-facing SaaS company. So, for example, let’s say if you have a plugin that plugs into a Shopify store, for example. How does… How does a vendor, provide value, as in, justify value. My app led to… my app led to this much amount of attribution, and it’s… It’s not a science, it’s basically an art where you look at multiple data points, but at the end of the day. You can only say with a certain amount of prediction, or you can say with only a certain amount of certainty, whether a particular order was influenced through that influence, either directly attributed, influenced through that plugin, or completely organic.
Shane Butler: Yeah, so that’s like a pretty challenging thing even for a human analyst to do with like, they have, you probably have to have a lot of. context around where that’s all coming from, how it’s calculated, how one thing gets attributed to one and not another.
Attendee: That is true, and also it’s a case-by-case basis, because we have noticed in several instances where the business intelligence data that the company collects locally is different from the Google Analytics data that’s sent to the dashboard. So, there are discrepancies, and it depends There is a lot… there is so much variability that there is a strong human element to it, to investigation and justifications. There is a strong human element to it.
Shane Butler: Yeah, I can imagine onboarding someone new, there’s a ton of context that has to be taught to them, and especially with a Agentic Analytics system, as well. Nice, thanks for sharing, Attendee. Attendee, how many ACH connections were established by customer every month? Attendee, where do I find our product user guides release notes? Why don’t these numbers match? Yeah, that’s just… that’s just… that’s basically the job. Why don’t these numbers match? You’ll just spend 30 years doing that, and then you get to retire and live your life. Okay, so thanks, thanks, thanks everyone for sharing. These are great. It’s like, in our case, like, maybe someone asks for this, like, imaginary case, maybe someone asks about renewals regularly. And there’s a report, like, the customer success team uses. Maybe there’s a revenue question and a finance query behind that, like, a monthly report or a sheet that people look at.
But, you know, you don’t have to drop in the chat, but it helps if you start Thinking of a thing, thinking of a place to start as you want to start applying this to your own job. Because this is what we’ll be doing. In the kind of live workshops to come. Okay, so let’s… let’s… Go ahead and keep it moving. We can kind of think about these choices. We need to understand For those work… that work we picked. So let’s go back to our retailer’s monthly revenue report. So, when we open, say, a report, we’re gonna see a number. we see that number, we’re going to want to go look at the query behind it, and that is kind of like the first part, at least for me, where I’m going to start to try to understand
How finance has historically arrived at some. revenue figure, right? You’re kind of doing that, exploratory investigation work, trying to parse out someone else’s SQL query, which is always fun. You might find that, like I said before, it excludes, like, test orders from our QA team. Maybe it uses… maybe there’s a bunch of different date fields that, could be leveraged, but it uses a particular, one. Those choices are going to be really useful for any future analysis we do. Whether we’re AI or we’re a human. Suppose we also find something kind of weird, where it’s like, there’s a filter in there that only includes customers who received some particular promotional campaign, when we’re saying, looking at last… last week’s report. That might have been appropriate for the original report. But not for our current question. So before using the query for company-wide revenue, we kind of need to understand all of the filters and decisions behind it, and whether they belong in our new analysis. There’s multiple ways to make existing work.
like this available to, agents. Hex supports this through projects as context. I dropped a link down there in the slides, which I’ll share later on. But… Having the ability for an agent to refer to existing analytical work. And its logic, by pointing it towards past projects, is a great way to start building out that context of. How things work for your business. So for the report you chose, you know, you’ll want to think about how can you share that calculation from previous analysis, along with the explanations of what it was created for. So if… In our case, it’s, like, last… month’s finance report was really focused on a meeting around a single, like, promotional campaign, and that’s why that filter’s in there. We want to add context around those kind of caveats of the use case, so as it’s extended to
Questions that don’t care about that commercial campaign. We don’t include that filter. If the query… Leaves anything, like, unexplained, right? If we don’t know why certain filters are in there, certain things are joined. I find it’s just really helpful to go back Ask the owner, you know, you can come up with your hypotheses, but, you know, ask… whoever’s in there, whoever created it, why that’s in there. If there’s, if you’re seeing, like, in the results that things are filtered out, but you’re not seeing it in the query itself. Then you might need to go further upstream to how the data itself was created, which is our sort of next source of context. So let’s say… We open up this query. And… We notice from the numbers that it’s like filtering out some sort of users, but we don’t actually see that filter in the query itself.
Like, why are test orders missing from the report, for instance, if the query doesn’t exclude them? So one possibility is that the transformation code sort of kind of like. data modeling code, say if you use, like, DBT or whatever, to create these tables, actually remove those records before the reporting table was. even created. So that’s upstream from the query. Looking at that code ourselves. And giving AI access to inspect it can help us understand what’s already happened. to the data we’re using prior to any analysis. Sometimes we need to go even earlier still. Sometimes we need to go beyond. before that ETL layer up to like the application code itself. So now we’re thinking about like what’s actually being tracked. And how? When does the application record an order? When someone attempts a purchase, or when the transaction is actually completed?
So those… those get us into, like, the product records and the actual, like, event tracking. And this is all information you can share, with your AI analyst. As a regular analyst or, like, a data engineer, you’re probably looking through this all yourself if you’re really trying to understand a product area, right? You’re not just looking at past queries, you’re looking at all of the pipelines that created the table, and you’re working with your engineering team to look at the underlying application code to see how tracking events are fired. That’s all great context to add, not only to give to a human, but to give to an Agentic system as well. So if we’re trying to understand what the records mean, we need context much further upstream than just the query itself. In this case, we’d be using code as context. So…
You’ll probably know what a repository is. It’s where a team maintains its code. That could be application code or data modeling, data transformation code. Transformation and modeling code. It’s going to explain how records were filtered or combined. application’s gonna explain what was recorded in the first place. So that code tells us what the system implements. in our course, in our case, it’s, like, retailer’s site, and as you all know, AI is pretty freaking good at reading and understanding code, so it’s another great source of context, to provide it. We can ask… We can then ask, you know, any relevant owner of that code. Whether the behaviors actually fit the analysis we’re trying to do, or if we have to. add something different. Once we understand that source, though. We effectively want to make available any sort of descriptions around it to make it as interpretable as possible.
Attendee, no, this isn’t necessarily, the SQL behind the query. This is more like… You know, the application code, the code of your actual product, that is firing off events when a user, does certain things. So, basically, those events later on gets translated into, the kind of raw data. Yeah, like a website workflow. That eventually evolves into, like, the raw data that you will query yourself. But it’s a great source of information. So… Lost my spot. Here we go. So let’s make it a little more specific, like, say a customer buys a trail shirt. A water bottle and a day pack from our retailer. We’ve got a record for… The whole order. in our, like, fact orders table. And we also have separate records for each item within it. And our fact order items. So, granularity orders tables, each row represents the whole order.
Its total belongs to the order. In the order items table, each row represents one item within that order. an item’s amount is going to measure something different than the order’s total, right? If the dollar value Of a single short, a single shirt. It’s the dollar value of a single shirt, a water bottle, or a data pack. That’s what’s going to be in the order items, while the order total is going to be all those combined. So if we’re trying to query, like, what’s the order value, we want to have some descriptions of like what each table’s granularity is, how things should be added up. What each row represents. And in that way that we don’t end up doing some, like. I don’t know, duplicating rows or miscalculating things. So, we can explain the table, we can explain the columns, if there’s other columns in there, like status columns, what values it contains, which values mean an order was canceled.
Which mean it was refunded. We want to provide as much information, and just make that as interpretable as possible. You probably already know this stuff from working with the data. It might be very obvious to you. Honestly, something as simple as this is probably very obvious to. an Agentic analytics system with how powerful the models are, but I think you can probably imagine we all have Some tables at our company that are terribly abstract. that we just want to add as much detail and descriptions as possible, so whether a newly onboarded human analyst comes in, or an AI analyst comes in, they can really understand what everything means and how things join together. Like, an example of this, basically. This is not… probably not a mistake many of us would make, but, like, say… Say we got an order for $100, trail shirt costs $30, water bottle $20, and a day pack $50.
If we weren’t to look at, like, the underlying order item data, and were to do some, like, quick join, where we joined the order items to the total orders table, and then aggregated that order total column from the orders table, you’d end up 3X-ing your overall order value to 300. In reality, for this question, if we want to know, like, total order value, we wouldn’t even need to join to the order items table. That might be relevant to a totally different question, but it’s just easy for us to kind of, like, make mistakes like this when we don’t know what the underlying, kind of. grain and column meaning of these tables are. Again, this is kind of like a pretty simple example. That I actually don’t think AI would make a mistake on. So… Now that we’ve seen why the choice of, kind of, sources, different sources of, information matter, like our code, like our policies, like our queries, how would our new analysts know which available sources the team
Has actually reviewed, like, which ones they actually are trusted. There might be a reporting table. finance relies on. Maybe there’s, like, an experiment analysis someone worked on a couple weeks ago. Maybe there’s, like, a bunch of, like, old reports in your company’s, like, Notion repository, or whatever sort of, like. Company wiki you have. All of these kind of, sources, these kind of… example analyses could look really relevant to our revenue question at hand. Doesn’t mean we want to… Just like it doesn’t mean… just like we don’t necessarily want to take all these and just, like, give them to a newly onboarded human, we don’t necessarily want to, like, give them all to an AI, as equally weighted sources of information, either. So having some way to kind of… certify or endorse these kind of reviewed pieces of work helps, both humans and agents prioritize where they should really start. So descriptions can explain
what each of these sources is intended for, and then an endorsement of some, you know, you could do, like. This was reviewed by… XYZ, this hasn’t been reviewed, you could do, sort of, like. medallion tier of, like, gold, silver, bronze, in terms of what is the most, like, reliable sources. But together, this and the descriptions give a lot more information about which work, the team recommends using. We still need to check whether the source fits our… request. Like, the monthly revenue summary might be useful for totals. But it may not contain the detail needed for some sort of product investigation. But at least in this way of endorsements, we know we don’t have some old report that we’re counting on that’s using a definition that the company has changed. So, endorsements really help us prioritize reviewed work.
Explanation helps us understand what the work’s appropriate for. Yeah, Attendee, I think a big part of managing the challenge of changes to context, is solved by having owners. Like, I think a lot of… Some of the jobs… some of the data team’s jobs, some of, like, the stakeholders’ jobs is going to evolve in terms of, like, having specific owners for… contexts that need to be Reviewed on some, maybe, cadence, but at least importantly, like, if that person knows that their team, like, if the finance team changes some… definition, then you have someone who’s in charge, like, hey, I own, the kind of… management of this context, and I’m going to update that in the… places that need to be updated, or I’m gonna reach out to… the team who updates it and tell them what the change is. So I think it’s like.
it’s definitely more stuff to maintain. I think this is actually work that we should have been doing, you know, long before AI, and really good data teams have done this long before AI. But it’s also, like, you know. It’s kind of boring work. It’s the can that gets kicked down the road, and it’s really easy to lose track of. I think also just, you know, having dates around when the last, like, update happened is a nice way to know, like, hey, is this pretty stale? Like, has no one looked at this for 6 months, or a year, or 3 years? But definitely some, you know, human element there, and of course, you can… you can automate, like, some sort of reminders with that. So who do we talk to at the retailer? before turning, the report into guidance, for everyone else. So this is kind of actually what I was just talking to you about, Attendee. So, finance can help confirm the revenue definition, and the decision is intended to support, the data team
can explain how the source itself is built. The person maintaining the report can then explain, kind of, its filters and assumptions. So it’s… you want to have, like, designated owners for all these things. At a smaller company, honestly, those responsibilities might actually all sit With the same person. But it’s important to have an actual person you can go to in order to get answers. So if two reports disagree. You want to be able to bring that question and that calculation to the people who are kind of in charge of working that out and aligning. Like, you know, I’m sure data people here have been in lots of meetings before where you’re trying to align on… You know, why these two numbers don’t match. I think someone, yeah. Attendee, I don’t know if I’m pronouncing your name right, but why don’t these numbers match up? I’m sure we’ve been in a lot of meetings before where you have to get all the right people in the room. So that doesn’t change. It’s probably more important now than ever, but it’s nice that we have this forcing function too.
Kind of actually make it happen. you Yeah, we kind of need to understand. Which is the actual thing that applies if one report might have an outdated rule, that someone’s basing their opinions on, or they might both be, honestly, useful, correct interpretations of a metric, but for different purposes. And so that can be kind of described as well. Once you’ve confirmed a rule. Like I said to Attendee, you also need to decide who’s going to, maintain the explanation, so when the business changes. Someone needs to update the knowledge the agent’s reading, just as someone would need to update the information another employee relies on. So the question, like, you wrote down earlier, I think it’s helpful to think about who could help confirm the meaning. If you haven’t got a name yet. Finding that person’s probably going to be one of your first actions you take in terms of, you know.
managing context. Okay. Man, we’re already at 1043. Okay, I’m only halfway through my slides. I kind of made these slides, and then I looked at the number of them, I was like, yeah, this is, like, the right number of slides, and then I was… also realized I was thinking in the context of, like, the two-hour courses that Hai and I teach, not a one-hour lesson. So what I’m actually gonna do… I’m gonna go through a few slides here. We’re about halfway through. I’m gonna add a part 2. To this, next week. I actually queued up a Lightning lesson just in case this happened. So let me drop a link in the chat. You guys can all kind of sign up for this now. But we’ll do a part two. Next Thursday… 10 to 11. You can sign up here.
And I’ll keep going for another… 5 minutes, and then we can kind of open up for questions. But we’ll get… we’ll get more into this next week, too. But I don’t want to rush through it. I’d rather we just have all the content available for you all. So, we’ve identified… Useful work, and the people who can confirm it. Once we’ve agreed on that meeting. Where do we put that knowledge so AI can actually use it? There’s several kind of different jobs here, right? Explaining what one order rail represents. That belongs pretty close to the data, probably. In his description. Identifying a source that the team has reviewed. is kind of what an endorsement helps with. We might also need company information that applies much more broadly to any analysis and Hex that lives in kind of like workspace context.
A detailed procedure for answering a very specific type of question. You can create a kind of, like, a… A document that acts as a guide. If we’ve got a calculation the team wants to reuse as a defined metric. A semantic model can specify that calculation and any relevant data relationships. We’ll come back to this shortly. Previous projects or code can provide references the agent can inspect as it does the work. Umm. A lot of these choices, in terms of, like. what needs explaining and where it can live, whether it’s Hex or any other product you use, or if you’re kind of building out your own repo as well, like we do in some of our courses. That choice just depends on what you’re trying to explain. So, a table description, a particular procedure, a metric calculation, they each give the agent, different information in a different way.
So let’s start with instructions. We’ll talk about which ones we’d want available for, kind of, like. every question, and then which ones we’d want specific to, like, a single question, like our revenue question. So, for the retailer, for our outdoor retailer, some information might apply across a lot of analyses, like what the business does, or conventions used across all data. Other information may be super specific, like the procedure for calculating revenue from our orders table. Whether you’re using Hex or building this yourself, or using any sort of product, you’re going to want to have some distinction between these two types of information, these two kind of use cases. So… Like, in Hex, workspace context is included in every conversation. That’s a really useful place for company information and conventions that are relevant broadly across any sort of question you or your stakeholders might have.
And a sort of like guide library can be retrieved when It’s only relevant to a very specific decision or question that’s being made that needs that information, but isn’t needed in other questions. So, say, when answering a revenue question, the agent might use a detailed revenue guidance, just like if I’m going to, as an analyst, want some, like, how to answer revenue, I might have a guide that I go read specifically on that. But someone investigating a different part of the business, they wouldn’t necessarily need to use that revenue guide. They probably still want that broader company context, but they can ignore all this sort of irrelevant information to their business. They may need a totally different set of instructions. Maybe they’re trying to understand, like. how NPS is calculated or something. When you write your context down.
You basically just want to ask, like, you know, which questions actually need that context. That helps you decide whether it belongs in a more, kind of, broadly supplied context, or in a guide for a particular kind of work. Which brings us to another question. If we put the revenue instructions in a guide. How would an agent even recognize that it’s the guide it needs? So let’s imagine we saved… like, you know, instructions, we wouldn’t do this, but imagine we saved instructions under something, like, terribly abstract and generalized, like sales notes. And the description just says something like, information about sales or sales revenue. you know, what would we expect to find in that document? It could be about forecast, it could be about sales targets, it could be about calculating revenue from orders. the name doesn’t give a new analyst. Like, if you were searching through a repository.
of your company’s information. It doesn’t give a human a lot of information around what it contains, human or AI. It just doesn’t help much in deciding whether to open it up for this question. So a guide’s name and description Really helps the agent decide when it’s relevant. Something more like revenue and orders gives it a lot more to work with. The description can explain that it covers revenue, sales totals, refunds, and order reporting. It could be much more exhaustive than this, even. I think the best way to think about this is to use the words. people at your company actually use when they ask these questions. Include those, like, actual human words from those conversations, and explain what the guide helps them to do. That’s gonna help a lot with, the agent being able to, Select.
when it’s relevant. That being said, we still need to have some sort of system to verify… verify that it’s selected. We still need to inspect whether the relevant guides were consulted in the analysis, and once the agent opens it. Do the instructions explain how to actually do the work? So, what I’ll do here, I think we’ll end after this. These next two slides. We’ll talk briefly just about what a guide might look like. So, if we just say something like, In our guide, like… Sorry. Use the right revenue data. That’s, like, terribly general. No one’s gonna write that. But the agent still has to do a lot of work. in terms of, like, which source we mean, how we calculate from it, is having to reason through a lot of choices we could have explained to it. And the more reasoning it has to do without us explaining to it is the more opportunities for it to kind of guess, the wrong thing.
So we want to be much more specific. We want to name the orders table. Explain the exclusions. say that each qualifying order should contribute its total once. We can be… we can be far more detailed than this and explain the reason behind all these instructions, too. You know, we could give examples of errors that might occur. Like we said earlier, like, if an order is joined. Three times its total can appear on all three rows and overstate the total order. That explanation is going to help someone understand when and why instructions matter, whether they’re human or AI. It also gives us something specific to evaluate, so we can inspect the source selected, the filters applied, and whether the calculation actually adds up one total per order, as our rules kind of define. So in a way, it kind of gives us the basis of a rubric to evaluate against, too.
So let’s put those pieces into, like, a kind of short guide. This is a very, very basic version of a guide. You can actually have take your kind of guide this way and give it to AI to kind of make it more robust, align it with your team. But, you know, take a moment to read through it. Think about what would help a new analyst understand. the work. I think a lot of the times when I’m trying to think about context, I just think about like, well, would it help like a new person joining? In this case, it explains which request it applies. To what needs confirming. And where to find the calculations, so… If the request hasn’t specified the reporting period or whether it means gross or net revenue. The agent should confirm that with the person.
For this example, we’re using a net revenue definition that excludes canceled, refunded, and test orders. Your team would need to confirm its own definition before you’re using, like, whatever instructions at your own work. Then it tells the analysts how to add those values, and explains the kind of, like, example item join problem we talked about. And then the comparison at the end uses the retailer’s daily revenue summary. So we’re giving something for it to kind of like try and tie out to or compare against. We’d effectively need that summary to cover the same period. and definition, but if the totals disagree, now we have something to investigate in terms of trying to understand why that is. We might be comparing different definitions. Or data updated at different times. At the end of the day, we’re basically just trying to Draw out a procedure.
That the agent can refer to, so it has less guesswork. Umm. For a calculation, the team uses regularly. We can decrease that guesswork even more by defining the metric itself for use through semantic models. I think we’ll stop there, though, since we have 5 minutes. And we’ll start with next Thursday. on part two. Yes, we’ll start getting into… Defining a metric that your team. can reuse. Okay. In the meantime, I think we just dropped maybe, like, next… next… last 5 minutes for questions. I will shout out very quickly, some of the upcoming stuff if you haven’t joined yet. So next… next week. Next Thursday, we’ll continue with Part 2. of… How to teach your AI. How your company works. I just dropped the link in the chat again. We’ll do that same kind of time, 10 to 11 Pacific.
On October 30th, We’re gonna get into… how to, fix the mistakes your AI analyst keeps making. Rachel’s gonna join us again for this one, and we’re gonna… walk through kind of one of the context consultant skills they’ve built over at Hex. And then… feel free, that’s free. Go ahead and scan the QR code if you want to join that for free. If you can’t make it live, we’ll send you out the recording afterwards. And then November 14th, we’re going to have our hands on. Workshop from 7 to 11. Pacific. I think that’s a Saturday. It’s only 25 bucks, with the code context. 25. If you scan that QR code, it will automatically apply that promo code already for you. This one will be… This should be pretty cool hands-on. We’ll we’ll all be getting into our own hex workspaces
you can leverage… Some… Data that we’ll provide for you, or even work on your own company data. But, yeah, any questions in these last 5 minutes here? Hi, any questions that you saw? Come up in the chat.
Hai Guan: Yeah, Attendee has a question around, “Is too much context a problem?” Is context cleanup part of the engineering components?
Shane Butler: Yeah, I think too much context, like, is this is kind of what I was getting to with, like, routing. to specific kind of like guides for a specific questions. You don’t want to. You don’t necessarily need to like. feed your agent all the context around your business for a specific question. You can have that kind of like broad context that you know you need for every question. And then for a specific question like the revenue one here. you can have it read the relevant guides to those questions. We’ll get into this, next week, but then that can also lead them to read into, like, the relevant semantic models for that question, so it’s kind of, like. Routing into a single place. You don’t want to necessarily feed everything, but it is, it’s also interesting, like the models are getting.
Better and better at handling. very large swaths of context, so I don’t know how that changes in the future. I don’t know, Rachel, if you have any input on this question as well.
Rachel Herrera: No, I think… The too much is what you were describing, like, in one go, yes, but I think if you’ve got it organized in such a way, and your rag is, like, doing proper retrieval, and you’ve… kind of put, like, the term they’re using now is, like, front matter, which is effectively a description of the free text context that you create, like, the markdowns. If that’s all set up accordingly, to your point, like, the models are getting really good at being like, nope, that’s not important, don’t need to read it, versus, yes, that is important, I should take a look at it. And that really helps. I think the… the failure mode is stuffing it all in a single file, and hoping that your agent reads the entire file correctly.
Shane Butler: Yeah, 100%.
Hai Guan: Attendee. Sorry, go ahead.
Shane Butler: I was gonna say, what’s a good way to spot a context gap? or insufficient context. We’re going to talk a little bit about this next week. I think Hex, you guys have an article on around, like, context exhaust. Kind of like, it’s an iterative thing. You’re always gonna have gaps. It’s kind of like one of these things where it’s like. If you… If you… Create a system that answers all the questions that anyone’s ever asked. They’re just gonna start answering… asking new questions that the answers to those old questions start spawning off. But kind of, like, going through iterative phases of, like, trying to do the analysis, seeing what it gets wrong. And then… talking to the people around, like, hey, why did they get this wrong? And then adding the new context, having a way to evaluate doing AI evals to see, like, how accurately you’re answering questions within different parts of the business are all great ways to find.
those gaps, I think there’s, like, always gonna be… There’s always going to be room for anything, for, like, people are always going to ask… answer new, interesting questions when they’ve answered all the stuff. They asked before, and you have to keep building it out. I don’t know, Rachel, if you got more. More thoughts on this, how to identify gaps.
Rachel Herrera: Yeah, I think that becomes the, like, core… One of the deliverable… use cases that data teams are, like, highly focused on is, like, closing those gaps through whatever process you know, is it reading agent messages? Is it reading thinking? Is it reading tool calls? Is it running evals to sort of regularly understand? Is there any regression? And then closing those gaps sort of becomes, like, the context management lifecycle that teams are going to have to get really comfortable with going forward.
Shane Butler: Yeah, Attendee, I don’t know if you’re already signed up for our workshop, but we’ll be, for sure, focused on this in the hands-on workshop. You know, we’ll play through, like. Creating some context, you know, starting from scratch, and then identifying gaps. You know, filling in those gaps, rerunning our evals, see what gaps still exist, and going through it again. What else? I know we’re a minute over. Any other questions? Reading code assumes code’s correct. Yeah, exactly. I mean, I think anything… I put a post up today that’s like the Anakin Padme. A meme that’s like. Oh, I rolled this out to my entire company, and it’s like. you fixed all the data foundations, all the underlying code, right, before you did that. Like, it’s just… if you build this stuff on… No matter what the product, if you’re using
Claude’s dashboards, they just released, or ChatGPT’s data agent. If you’re not even using AI, if you’re building Tableau dashboards, or Power BI, or whatever, or a Google Sheet, like, if it’s based on… faulty foundations and bad code and data, then it’s just, like, you’re just gonna get wrong answers faster. So, certainly still. strong component there. I think this could be the forcing function to finally clean up all those data foundations, because you’re gonna release this, and your CEO’s gonna be using it, and they’re gonna be like, what the fuck, Shane? Why is this thing all messed up? And it’s like, hey man, our data stuff’s screwed up, like… It’s been screwed up for years. And they’re like, okay, I fixed that so I can answer my question. I don’t have to ask you every time.
Sam? I think it’d be a good forcing function. I say this always in our… in our sessions, that If Judgment Day happens, and Skynet goes sentient, the robots take over, we battle them. We win the war against AI, but we have to shut them all down. It’s a bummer. at least we’ll have shored up our data foundations in the process along the way, so now that human analysts can have trustworthy reports, we all go back to regular human reporting. So, you know, this is a win-win. There’s no losing to this. To making strong data foundations.
Hai Guan: And there’s probably no better appetite in history of data work that There’s this window now to actually get it going.
Shane Butler: Yeah. If you’re a data leader, this is the time to be like. I need the resources to do this to make this work for all of you. Cool, alright, we’ve stayed 5 minutes over. Thank you all for joining. I will send out an email. I’ll send out an email with a link for the follow-up part 2 for next week. Sorry, I… I overestimated my time significantly for today’s session, but we’ll get through it next week, and that’ll be fun. October 30th, hope you’ll join us then, and hope you’ll join us on November 14th as well. Yeah, in the meantime, feel free to hit us up on LinkedIn or Slack. I’ll I’ll drop our Slack channel in the email as well. And or just email me. You can email us at hello at AIanalystlab.ai. But thanks all for joining! Appreciate you.
Giving us the time. As always. And we’ll see you next week!