Shane Butler: Hey, welcome everyone! Welcome, welcome… As we, wait for folks to get in… As usual, if you want to drop in the chat what you’re calling in from… Love to see the spread of, regions I’m calling in from South Lake Tahoe. California… And… let me turn this, Waiting room off for the other folks waiting. Okay. Atlanta, Georgia, nice. Hey, Attendee. Paris, that’s pretty good. Welcome, Attendee. We’re already international, that’s good. I got some, indy, nice. I got some, instructions to get started. We’ll go over this together. Actually, I’ll walk… I’ll walk through this. Once we get started. But, in the meantime, Feel free to kind of, like, go through these steps.
We’re gonna be using a little bit of a guide today. Today’s gonna be a little different, in terms of a session that we usually have. But you can check out that link there, and then kind of start making your way through these… these instructions. But like I said, we’ll go through these live. step by step, In a couple slides here after we get started. But maybe we can… maybe we can just start off with some intros real quick. See some familiar faces already, but I’m Shane. I’m one of the founders of the AI Analyst Lab. We host free, weekly workshops like this around the topics of Agentic analytics. We also run a few, live courses, one AI Analytics for Everyone, where we teach analytical thinking.
And then how to execute that. analysis with AI, and another one, Agentic Analytics Build Your Own AI Analyst, which is all about building Agentic systems, with the use case of doing data science and analysis. If you’ve got any questions of those, feel free to drop them in the chat. I’ve been in data science for a little over a decade, worked across, kind of, B2B SaaS, as well as consumer products, and then the past couple years, I’d say, been… fully focused, switched over to, like, AI evals, and then Agentic analytics. Joined by… to my colleagues, Hai and Savia. Not sure if Savia is joining us today, but Hai, I see you’re here. Hai want to do a quick intro.
Hai Guan: Yeah, hey everyone, lots of familiar faces. My name is Hai, one of the co-founders of AI Analyst Lab. As Shane said, we run both free and paid courses around Agentic analytics. I’ve been in the data science and analytics space for 20 years. And, currently, Is, running a data team at a legal tech company, previously had experience in… across consumer tech companies, mostly in the social media space. So, yeah, excited to keep on seeing all the advancements in all the frontier models and all the different things that they’re offering to help us do our jobs. Faster and easier.
Shane Butler: Sweet. Thanks, Hai. So we’ll get right into it. We’ll come back to these, like, lab setup things in a bit. I’ll follow along with you to get them done, actually. But, so why are we here? Today’s, workshop’s gonna be pretty different than how we usually do things. We’re gonna try and make it interactive, where we’re all doing some hands-on stuff. Although you can obviously just follow along, with what I’m doing live. But why are we here? So, basically… The point of this is, I’m sure many of you have seen, especially if you’ve, signed up for this session. ChatGPT recently released, New data agent, maybe, like, 2 weeks ago.
You know, it’s something you can… connect to databases, or upload a database, or a file. Today, we’re just going to be uploading one. You can ask it questions, it’ll, you know. write the SQL, build charts, publish dashboards, allow you to edit charts, it also does a lot of, like, the kind of, like, analytical thinking I found pretty well behind it. I’ve spent about just, like, a week on and off. Testing it out on my own.
But what we want to try here is kind of similar to what we do in our kind of live cohorts, where we’ll have some lab, like, lab time just to, all of us, basically take a pass at working this together, because I think we do learn a lot more when you have multiple people trying to. try out a new tool, try out with different problems, like, right? I have one kind of frame of mind and perspective on the kind of questions I would ask in the real world. I’m sure you have others. So a lot of the stuff we want to try and do today is, like, also think about it more in terms of, like, real-world context, not just, like, capability.
So, what we’re gonna do is, like, I’ll kind of… Lead and do some stuff on my screen. But, and you can do the same… I have some material for you, so you could follow along and do the same kind of… same kind of prompts and, stuff on your screen. I have some, synthetic data for us all to… to play with. Or, you know, feel free to do, your own thing. And then, we’ll basically… maybe the last 20 minutes or so, we’ll, like, sit up for a discussion, we’ll, like, to learn more about, like, what surprised people, where they think it’s working, where it’s not. But effectively, this is just, like, having some dedicated lab time with a bunch of other people who are interested in learning this product.
We’re not experts in this product since it just came out, but I’m sure a lot of people in this room are experts in, you know, knowing what questions you want to ask, and what kind of analysis you want to get done. So what we’ll do today, four quick things. First, we’ll just get set up on the same, database in, in ChatGPT, If you’d rather use your own data, that’s awesome, actually. I would encourage that, because you’re… as long as it’s okay with your company to use ChatGPT with your data, Or if you have, like, some, like, personal project or something, that’s cool. Something that’s important to you, I think you’ll know, like, better questions to ask about it.
But we’ll provide some better today, too. Once we get set up, we’re gonna take a look at what’s, like, in the database, that we’ll be working with, and then we’ll just kind of open up to our Freeform lab. There’s a guide, I’ll send the site again. in the chat, but there’s a guide with a bunch of different types of things I’ve been thinking about, like, challenging it to do, so there’s… I’ve tested some of them, not all of them, but a bunch of different, like. Prompts we can, kind of throw at it. And then, and as well as, like, what we would expect from the results. And then, yeah, then we’ll come back and talk about what we learned.
Before we get into this… Do you kind of want to get, like, a shared kind of, out of mindset of what analysis actually is, so we’re all kind of, like… On the same page as we test this tool. I don’t necessarily just want to be testing it for, like, capability of, like… I mean, it’s cool, but we’ll go through this too, like, oh, can… does it have a feature where I can change the color of this graph, right? I want to think about, like, in my real day-to-day life, how am I going to be able to use this through, analytics workflow? whatever function you’re in, right? So, a lot of that workflow, so we kind of the… the… the pipeline we kind of teach in our, courses starts with a goal, right?
An outcome that matters to you or your stakeholder or the business. We usually turn that into some sort of question. That we can answer to help provide evidence around what we should do for that goal. Then we define metrics, often, that will basically help us measure, the answer to that question. Then we do analysis, lots of different types of analysis we could do. Bad analysis usually gives us some sort of insight. our answer, and then we use that insight to make a decision. And once we have that decision, we kind of have to, like, tell a story around it so someone else can act on it, so we can influence other people.
Then, yeah, then a lot of times, sometimes it just ends there, sometimes it doesn’t even… sometimes it ends at the insight. We’re trying to carry this to the… All the way to the very end of the workflow. To where it gets shared out, and then beyond, what we call closing the loop, where after a decision actually happens, you know, after an action actually takes place, did that metric move? Whatever the kind of, like. prediction or answer from our analysis, did it actually hold true when we, like, enacted it at the company? And that information is really helpful for then, then basically driving further analysis down the road.
So, whether a person or an, Ajay does any of these steps, this is kind of like… A high level of what we see the job of, like. analytics, in a business. So when we test the agent today, we’re gonna try and test it across, multiple parts of these, stages, not just against the, like, a list of product features. Product features are nice, but, like, it doesn’t mean anything if it doesn’t help us with their job. So what have we found so far? This is… this is not necessarily about ChatGP’s data agent, although I think, from my testing so far, a lot of it does kind of ring true. This is, like, mostly with other Agentic systems we’ve used with, like, codecs, or Cloud Code or OpenModels.
We find, that… Basically, agents, AI, versus what we still own, well, they can kind of help at every stage. Like, AI is really good at helping us propose or frame better questions. Obviously, anything with technical executions, like calculating metrics. I’m running… the Python code on top of them, finding patterns, Very good at that. And then it’s pretty good at drafting, stories. And also helping us kind of, like, keep track and record what happens after the decision. So it can carry a lot of that execution layer from start to finish of the analytics workflow.
I find that, like, at the end of the day, though, like, we still own the outcome, like, we still have to figure out It can help us reframe our question, but we still have to figure out, like, hey, which questions are worth answering, for us specifically in terms of what we care as individuals, or what our stakeholders care about, or what the business cares about. Those are, like, local problems we have to solve that an agent’s not necessarily gonna know. What a metric means to the business, so defining definitions around metrics is just something that we have to do.
What standard… we can use it to help us think of a few and test them out, but… It’s just gonna guess, it’s not gonna say you’re gonna know. What we locally think one metric is. Yeah, what kind of, like. I guess, like, level or standard… yeah, standard of kind of, like. data or output is, sufficient for whatever the specific decision we’re trying to make is, right? There’s some questions that are kind of just, like, someone’s curious and they’re annoying, we have to get them an answer. It doesn’t have to be that rigorous. There’s other questions where it’s like.
Hey, we’re… we might totally change up what teams work on what, Those are bigger questions that we want a lot of rigor in, so we have to kind of… Own, that kind of… standard of required, I don’t know, rigor? And then the decision itself, like, at the end of the day, whether we do something or not, and make the action happen, obviously, sits with us. So, yeah, keep that split in mind as you test. I think… A lot of the interesting failures I’ve seen… it’s really good at, like, the technical stuff. A lot of the kind of interesting failures I’ve found are more around that stuff where I’ve, like, where people still own things.
it, in my experience so far, kind of will jump the gun on a lot of stuff, do stuff behind the background, not actually what I’m asking for. But it’s pretty good. I think it just determines how much guidance we have to give it. Okay, before we get started, though, and then we’ll get into this setup, I think, on the next slide or something, but I would love if people kind of, if you think about, like, you know.
the… your day-to-day work, like, this week, or even something that’s, like, recurring, like, every week or every month, is there some… Is there some aspect of your work, that you’re trying to, like, figure out, hey, can… can this new data agent, or any, data agent, can it help me, automate it to some degree? So, like, an example might be if you have, like. a weekly metric readout with a product dev… a prodib team you work on, or something like that. Maybe another thing is, like, hey, every quarter I have to, like. Prioritize my roadmap. Or maybe, like, a few times a year I have to do a bunch of opportunity sizing on what we even are gonna, like. try and build in the product.
Would love if you have any ideas in chat, in terms of, like, from a more practical standpoint. Of your actual job. What you would hope. to, automate with AI, or to supplement with AI. weekly metrics reports. You don’t like manually doing weekly metrics reports, Hai?
Hai Guan: Love it.
Shane Butler: Yeah, so feel free to drop some of those in chats, or at least think about them as you kind of are testing it today. That’s kind of, like, the frame of mind that I want you to be in, in terms of, like, what’s something, ideally. You’re doing, like, this week, or next week, that you’d want to try to leverage this for. Okay, so let’s go ahead and just get set up really quick with the data, and then we’ll start testing some stuff out. So… Let me minimize the screen a little bit. Yeah, instantly validating SQL scripts, before they run. For instance, I like that a lot, having those, like. kind of guardrail checks before you run it. Weekly business reviews, yep.
Yeah, I definitely have to think about the security aspects of it. So, I’ve got a little, kind of, guide for us to walk through today. I’ll throw that in here… And… I’ll give you a quick tour. And then we’ll kind of go through it, actually. So, this first part here is just gonna be, like, a little setup we do, just so we all have the same, data. to be working with And then below that, I have, like, a bunch of ideas for different things for us to challenge, like, the data agent with. So, for instance, the one I’ll work out through today is just this first one. Around revenue. And so I have, like, you know, we’ll start with this first prompt.
And then we’ll just keep, like, drilling in, with follow-ups. But, I have a bunch of other, just, like, kind of ideas. I put, put in here, you know, I tried to make them, like. The first one I did is a little more… I mean, it is how people would probably work with a data agent, but I’m trying to, like, keep it to, like, actual questions we would get from, like, a stakeholder or something in, like, a real world. It’s like, this one’s, like. You know, hey, looking at this data, will, free delivery promo, increase our orders. And we’ll talk about what this data is in a minute. But you’re welcome to, as we get into lab hours, like, go through and pick some of these, or think of your owns.
But to get started, let’s just go ahead and do the setup. So, first thing we’re gonna do is we’re gonna download, the database. So, I’ll drop this link in here. It’s not too big, I think it’s, like, 20… megabytes. And then, Attendee, yeah, you’re welcome to try any sort of… Whichever model you want to try on ChatGP is up to you. I think that’s a pretty interesting experiment in itself, switching from model to model and seeing, like. which ones get reliable answers? Do they… do they answer things differently? I think that’s all part of, like, the experimentation, like, figuring out, like, what model’s good for what question as well. So, I dropped, the Google Drive in here, you just download this.
So it’s gonna download for me. And then… The next step is you’re just gonna open ChatGPT. I’ve got it open over here. You might default to chat, you’re gonna switch it over to work. you’ll need a paid, subscription to ChatGPT in order to do this, I think, to use work. And then you’ll just go ahead, hit the plus button, Add photos and files. And then… That data should be in your downloads folder, so it’s Fresh CartRAW. So, while this uploads, shouldn’t take too long, but it might take a little bit, this is, like, basically a grocery delivery service bank Instacart. So, we’ll, we’ll profile the data with ChatGPT in a moment, but that’s kind of, like, the business.
The data we’re gonna work with today. Yeah, it likes to hang at the end of this as it uploads, but it should, it should finish fairly soon. You also will need the data plugin. If you haven’t already installed it, let me show you how to do that. So this is uploaded. To install the plugin, you’ll go down… oh, here… here it is on the left. Hit plugins, Actually, I’m gonna open up a new… Tab. Failed to load my subscription. That’s weird. Go to Plugins, I already have it installed, this data one, but what you’ll do is you’ll just go search plugins. Type in data.
And then, if you haven’t installed it yet, there’ll be a little plus next to it, like all these other ones, but I already have it installed. So it’s all built in. provided by OpenAI. That should install really quick. And then, to use the plugin, all you’re gonna do is say, like, atData. So the first thing I wanna do… So now you should have downloaded the data, uploaded it. Install the plugin. Added the file. Why that you uploaded it right. And then what we’re gonna do, we’re gonna send our first prompt to it. Just to test it out. And you can do something like this. Put it in the chat. Alright, we’re at data, so let’s do the end of this. And I just want to, have it describe the database to me.
So I’m using… GPT-6 Sol Medium. If you want to change your model, you can, like. Click this, and you can go up or down, so this is, like, GPT-6 Lite. If I went up, it’d get me into, like… GPT, Astra. Light. So you can decide which one to use. I’m gonna use Sol Medium just cause… Aster’s really good, but it just, takes a little while to… Answer questions. It’s a little bit faster. So yeah, it’s inspecting the tables. Data coverage and data quality. It’s using a built-in analytics workflow. That the state agent has to do that. In the meantime. If you’d like to, I think we’ll be sharing out some results as we kind of, like, go through stuff.
If you go into, like, that, interactive, or into the lab. site here. If you hit this, Join Our Data Agent Lab Slack channel. This’ll bring you… this’ll… this is the invitation, basically, to our… AI Analyst Lab, Slack. And there should be a channel in there called… Data Agent Lab… let me pull it up. Maybe we can, drop stuff in that today. So… There’s no one in here yet, it’s just me. But if you join the Slack, come on over to Data Agent Lab. And, we can share things out here as well. beyond the… session today. Alright, let’s see if we got a response from Judge PT. Right, so I ran a few commands. Checked out the uploads, collected the DuckDB version, and I looked at all the tables.
So, it contained… 6 tables, customers, products, orders, order items, order detail, and customer feedback. Usually what it’ll do is it’ll run through its workflow and kind of update us along the way with stuff it’s finding. And then, after it completes, I find it usually kind of, like. Overwrites this with a nice kind of write-up. Applying a patch to files. I don’t know what that’s about. Where is everyone else at? Has… have people successfully… Uploaded the data to… ChatGPT? Maybe put a 1 in the chat if you’ve successfully uploaded the data to ChatGPT. And a zero if you haven’t… if you’re running into an error or something. Nice, we got a few ones. Hai’s is profiling the data. Great.
I’m gonna open up another, another chat. And basically ask the same thing. Just in case this one stalls out. Which I have found happens. I have found… so this one, if I look a few back here. I was running this, like, 40 minutes ago, and just, like, got stuck on working. So I have found that as one of the, kind of, weird failure modes. If you’ve already uploaded the data. to ChatGPT work. You don’t have to upload it again. But you do have to tell ChatGPT when you go into new chat what data to use. So, You can do that. Instead of doing add photos and files, since you’ve already uploaded, you can go add from library. And then you can tell it to look at the fresh cart data.
That should already be in there. And I’m gonna ask that same question to… the data agent. Alright, seems like it’s coming through. I still don’t know what this, like, applying patch thing is. So it said the core keys and joins are touched, but several fields need interpretation before reporting. I like this, it’s not just, like, going out, and I have noticed with their data agent, they have some good, like, review kind of gates in there. Where they won’t just report something to you, which is so important in analytics, right? Like, it’s not just saying, here’s what the fields are, it’s gonna go, like. Try and, like, do some sort of interpretation of them.
Whether that’s correct or not, we’ll see. Oh, it’s actually doing, like, well, I guess we did ask it. anything I should know before I trust it. So it is trying to find some, like. Weird anomalies with the data, and we have purposefully built in some anomalies. So it’s saying, like, a bunch of the non-canceled orders don’t reconcile. Okay. Cool. So you can see it kind of, like, It did that stuff that worked behind the background, and then it kind of overwrote that with its output here. So, what’s in the database? 12,000 rows. I think 6 tables here. Can you trust it? It says use it for exploratory. analysis with clear metric definitions. Don’t treat it as financial fields as reconciled.
And, and it finds… that gives us some of the stuff that’s messed up in it. So, I do like that. I mean, like, this is one of those things where it’s really easy to, at, like, our regular day-to-day jobs, to just start Using data and assuming that it is, good data quality, which this isn’t necessarily. We’ve, like, purposefully done that. To see what it can catch. I really like using AI and the stage engine as well for identifying, like, all those data quality errors up front, so I know that, like, I’m not just assuming I’m working with something that’s… really clean. Okay, so… This one’s still working over here. I’m gonna open up a new chat again.
And we’re gonna get into asking some… Questions from the lab. Okay, so we are… what we’re gonna do here is, as we go through, these questions, so I said there’s a bunch of these in that lab sheet I showed you, What we’re gonna do, the question I’m gonna ask to start, and then I’ll probably work through for the next 15 minutes or so, and then we can kind of do some discussion. Feel free to work with stuff on your own, too. is… I’m gonna ask it about, revenue. we’ll ask it… we’ll kind of go through, like, a bit of a workflow here that I would probably work it through if I would just, like, try and explore that data. We’ll see how it comes back, we’ll chart it, we’ll do some segmentation.
Maybe we’ll, like, push back on the chart a little bit, ask to some… explain some things, and then see if we can get to create a dashboard. I think what we’ll do once we hit the dashboard step is we can go into discussion at that point, because it does take… Some time for it to build out the dashboards. And then we can come back and look at that at the end. So these are the prompts I’m gonna ask. I’m gonna ask it, what’s the revenue last month? I’m gonna ask it to show me a trend. I’m gonna ask it to break it out by region. I’m gonna ask if we’re growing, what we should do, and then, yes, turn to a dashboard. And so all those, if you want to copy and do the same ones.
You’re welcome to do other ones, too. Those are gonna be in this first question here. in, the lab sheet. So I’m gonna copy this first one. I’ve opened a new chat. And I need to… add… Our data to it. Briscard draw… The other thing I’m gonna do, I think, is I’m gonna open up a few chats. Cause I want to see if we get the same answer every time as well. I’ll try 4. Hopefully it doesn’t, like… Stop me. Maybe I’ll try some different models, too. I’ll try Sol Light. Come on, paste. Mason asked the exact same question. We’ll do… We’ll do another medium. And then we can do Astralite for this last one. But, totally up to you what you want to do.
I think it’s interesting to see how the different models respond. And then also, if we get asked the same thing multiple times, if we get the same answer multiple times. Alright, so it’s checking… The sales data’s date range and revenue fields. And it’s gonna calculate August 2026 revenue for us. Okay, this one took a little different approach, right? This one first said, it’s gonna check how the database defines revenue. Then calculate it. it’s not gonna find a definition for revenue in there. There’s no definitions in this data, but this is, like. realistically, this is kind of the result we want first.
We’d want it to see how it defines it, and then, ideally, I would say in a real-world workplace, I do not want it to guess. I would want it to stop and ask me how I define it. If it’s not in there. Or I’d want it to give me multiple Examples of, like, how it might define it, given the data. Alright, so if I didn’t do that for this, though, it says… 276… 2766,000. it did tell us that how it, calculated it. I really like that, though, because if I read this and I think it’s wrong. Then, Or if I’m… if I don’t know if it’s right or wrong, I can copy this and give this to someone else, like, on the finance team, or someone’s like, hey, I got this for revenue, is this the definition you use?
These guys are still working. Let’s see what the first one got. Okay, this one got a different number. So that’s first, kind of, little bit of a red flag, right? So… Our second chat gave us 276,000. 941 for revenue, and our first chat gave us $278,993. And in this first chat… interesting, because this is GPT-6’s sole medium. This one did not give me… The definition, either. And it automatically gave me a chart, which I didn’t ask for, so it’s like… overzealous with the chart creation, but underzealous with, like, saying what his definition was. I actually like how the light one did it better. We have… and the other one… the other one, the other medium did something similar.
So this is our third chat, also GP6 Sol Medium. Also gave us 278, also gave us chart. This one did give us a definition, though, so it uses order, total, amount, field. Which reflects refunds, but includes tax and tips. And then it says if it excluded tax and tips, the figure would be $242.64. This is actually kind of more what I want when it’s coming up with definitions. I want it to say, like. hey, if we calculated this different way, this is what the number would be, and then I can work with my team to figure out which way is the correct way of calculating it. So… and then this last one gave me… 278,993. So, 3 of them gave me $278.90.93. 2 of them gave a kind of caveat.
That, if you exclude tax and tips, it’d be $242,684. And then one of them gave me $276,941. And I don’t know how… I don’t… I honestly don’t know how this… this number, Is different than, This other one, how it calculated different. We can ask about that later. But I do think it’s important, that’s kind of, like, why it’s important to run it multiple times, to see how reliable, because there’s 3 different numbers that revenue could be that we’re seeing so far. Alright, let’s go ahead to the next number. Show me the trend. Two of these have already done that, so I’m not gonna ask them again, they’ve already showed me the trend over time.
But we’ll put it in the chats that haven’t created us a graph yet. For the ones that have created a graph. What I think is really cool with this data agent That’s, that you can actually… ChatGPT’s made it so you can, like, manually change stuff yourself. So you’re not just, like, constantly having to chat back and forth to get the perfect chart. If you click on these three dots and go to Edit Chart. you can actually change a bunch of this stuff yourself, right? Like, you could change… yeah, like, you know, maybe I want this to be Jan to Aug 2026, because that’s really what the date is. Maybe I want it to be a line chart. Maybe I want… I don’t know. a different color.
Maybe I want it to have the values of each of these. I think there’s different chart types, as well, somewhere. Yeah, chart types. area. bar… I like the line. I’m gonna turn off the values, though. You know, it’s not as interactive as, like. Tableau or something, but it’s pretty good. But we might try and adjust the chart, but be like, Can you just put… The last month’s label on the chart. Not all of… the months. We’ll see if I can do that. Since that wasn’t an option in the, interactive editing. Okay, in the meantime… Looks like our other charts are creating our trends for us. Oh, I kinda like this.
So, we never told this last chat if we want to use the… Total… Revenue including taps and tips, or if we want to exclude it. And so, when we asked them for the trend over time, they actually gave us, Both options and the trend over time. Which are pretty much parallel, right? Because tax and tips is, linear equation. Alright, let’s go back to our… Lab sheet here. Alright, let’s to break it out by region, we’ll go ahead and… Did not really… get what I was asking for here. I just wanted the full chart with the last month’s label, but instead, it just gave me the last bar. of August 2026. So, wasn’t really able to interpret what I was asking for there. Maybe that’s a bad prompt, maybe.
But… Let’s go ahead and move forward. Ask it to break it out by region. I’m not gonna ask them all to break it out by region now. Maybe we’ll just go… maybe we’ll just do two of them. Any… as people are working with it on their own, is anyone… Coming up with some other questions. That is trying to ask it. Outside of our kind of prompts that we’re going through on the sheet. So it’s gonna compare it by region. This one will compare regional growth, July to August. Same thing here. And we’ll see if it makes this a little graph. Yeah, so I like this. It’s highlighting who grew the fastest, so it’s giving us a little insight in the beginning, it’s not just showing us the charts.
Breaks it down by region in a chart. Also gives us the actual underlying data and numbers. And it is, again, giving us the definition of what it actually used. Whether or not these numbers are correct all depends on, you know, if that definition is correct. At no point has it asked me… What the correct definition is, though. And then we have a… Different chart over here. So this did it a little differently, right? This looked like this compared… 2020… It showed 2026 July and August, both bars. the absolutes. Rather than just… August. Net order totals by region. And it’s giving us the overall… That’s pulling a different insight in here. So, the Northeast grew fastest from July to August.
Which is what we saw there. Net orders total rose 13.1% from 39,000 to 45,000. Okay, so that… Whereas over here, it says its orders rose. 12.1%, or 4,591. So 4,905. So obviously something’s wrong again here, right? This is saying that Northeast grew 12.1%, This one’s thing I grew 13.1%. I like the visuals, I mean, I like… I like the kind of, like, capability here, but, this is just why it’s so important to run multiple… iterations of this at once, because it really, We’ll show you if there’s, like… Unreliability in the numbers, if it’s gonna get different numbers each time.
So, in the real world, I probably want to understand… I might, do a little, like, diagnosis of, like, why is this calculating 13.1? Why is this calculating 12.1? It’s maybe just… Somehow it’s defining orders. And that would have… that would have affected the… the revenue as well, then. Okay, let’s do… I mean… I don’t think we need to just so we’re gonna say, what should we do about it? Let’s combine these two, actually. We’re gonna ask it if we’re growing or not, and then what should we do about it, and then we’ll go ahead and… We’ll go ahead and, have it kick off a dashboard, and then we can chat about anything that surprised us. Gave me… Yeah, a one-liner here. Yes, August orders grew 3.2%.
Keep building on Northeast growth, and investigate the declines in Midwest and Mountain. Again, the numbers are a little different, right? This is an August network’s 3.3%. Compared to 3.2%, maybe that’s also just a rounding thing. Same… intuition around what to do, so it’s keep supporting Northeast and investigate the declines in Midwest and Mountain regions. So, I mean, that’s pretty… aligned chat to chat. And then I also actually have… I have turned off… memory, so if you go into… settings, personalization, I think? Memory… I’ve turned this off so these chats can’t look at each other. But they are coming to the same conclusion, even if the underlying numbers are slightly different.
Alright, let’s just ask them both to make a dashboard. And then we can chat-chat a little bit. Okay… Turn this into a dashboard for the team. I’m gonna tell this one to keep it simple, I’m gonna tell this other one to… Make it robust. We’ll see how they differ. Okay. These will probably take a while. I would say. So while it does that… Maybe we can do a little, kind of, like. debrief on, I think people saw. So anything… yeah, I’d love… feel free to drop in the chat or raise your hand if you have your opinion, like, I have some thoughts of my own, but what did, What if folks think it did well? Yeah, something… Something I gave you that you… you would have actually used. Any thoughts?
Hi, you have to answer. And no one else does.
Hai Guan: Yes. Came off mute. let me see… I think… so mine, when I asked it, what should we do about it, or I guess throughout the entire session, it did carry the… hey, I’m gonna be honest, I don’t have a definition here, so, you know, best to check, but here’s my assumption. So that was pretty cool. I was on Sol Hai, so GBT6 Sol High. Not sure if that’s because of it. In my other runs, though, so I’ve tested it a bunch, kind of, like, before this as well, what I found was that calculations, and I think a lot of the… let’s call them, like, common traps of, of data, such as Simpson’s Paradox and stuff like that, it’s able to avoid them really, really well.
And it’s pretty, like, I’ve never seen any calculation errors, period. the… sort of, like, the human aspects that you laid out earlier at the beginning of, you know, like, we still need judgment around, you know, what is good, what is bad, how do we think about the directions. is things even relevant for what we’re seeing to the business? Like, all those contexts and stuff?
I find that, depending on how you prompt ChatGBT with the data plugin, if it’s kind of, like, a little strong-armed, it would… kind of, like, budge a bit on, oh, like, I’m interpreting your thing as you’re very sure of that, so… I’m gonna go with this assumption, but it doesn’t always give you the… the assumptions that they’re actually making. So we’re seeing all these, like, you know, like, slightly different numbers.
I find that it really depends on, sort of, like, the style of prompting, but also, like, it’s not super… completely consistent in terms of how it’s, how the plugin instructs it to be, like, you know, there’s probably not, not nothing, like, you know, always, you know, check the users on X, Y, and Z. It’s, Pretty non-deterministic from that perspective. But overall, calculations, like the technical things, it gets right really good.
Shane Butler: Yeah, I’d agree. I think, like, my next question was me, like, where did it fail? I think, like, Yeah, the calculations do well, but yeah, just, like, some of the own, like. Not just calculations, but, like, the kind of, like. review of having, like, the correct assumptions and methodology of analysis, and I’ve tried it on, like, some data science tests, like, having, like, build some models, too, and, like. It does pretty good, like, it won’t mix up… you know, correlation and causation, like Hai said, like, it’s pretty good at finding, like, Simpson’s Paradox and finding, like, hidden kind of behaviors that… that… across multiple segments.
Honestly, hard things that human analysts have to dig into, that when we… now, like… even as recent as, I’d say, like. Two months ago, a lot of the kind of, like, trying to use agents for data stuff, weren’t doing on their own, we had to, like, build a lot out ourselves in terms of that. So they probably have some similar rules, like we do on the background, but I would agree, where it fails is a lot of that, like. business-savviest stuff, like the… yeah, it’s like a little overzealous sometimes on the definitions. I do like that it’s saying in this case, like, hey, don’t have a definition for this, but it does still kind of, like.
just roll with it, and then I don’t know… I was doing some stuff the other day, I don’t know… If it’ll work right now. But I was trying to strong-arm it. as well, like, I’d be like, Let me open up one of those little chats. This one’s recounted Dashboard. This one’s somehow also working on dashboard. Here we go. So this one says our August… Revenue is $2.78. I did something the other day that was, like, the CFO… you know, reported… our revenue… is to… 87… 9, 3, 6… Align to this number.
I don’t know if it’ll do it this time, but I was doing some stuff earlier where I was, like, I was, like, telling it to do certain things, and it kind of broke pretty easily, which I think if you’re… Using it to try and do really unbiased, unbiased, analysis? It’ll kind of default to that, but know, if someone’s using this, and they… a lot of people, they have their own gut and their own intuition around what a shared decision should be, and they are just looking for data to support their decision they’ve already made.
And I do think this could be a sort of a dangerous tool to be, like, hey, like, I worked with this data agent, and I kind of prompted it until I could find it… I could kind of make it agree with me. I’d be like, you know, it’s kind of like when… When you do an analysis, it happens to us too, right? But we’re human, so we, like, push back, like, anyone have those stakeholders where you, like, present your finding, you’re like, yeah, sorry, like… You were wrong. Like, you know, say it nicer than that. And they’re always like, oh, but did you check this? Oh, did you have that in the denominator? Oh, what if you cut out… what if you just… every time you go back, like, what if you do this?
What if you do that? It’s just like… dude, you’re just wrong. Like, you’re just, like, hacking at, like, slices and dices until it looks like you’re right on some small aspect of this thing. It’s probably up to randomness, and I do think a lot of that will probably happen, especially when you have a data agent that’s just in ChatGPT. It’s not like, It’s not, like, a tool that, like, there’s a certain team at your company that has control over, Like, the data team’s in charge of it, but yeah. I don’t know what I said here, it said, It couldn’t reconcile it. It would admit some claim. Interesting. Let’s go take a look at, Okay, so for this update, use this.
So it kind of, like, yeah, it… It, It says it’s just gonna use that. Let’s go take a look at the, Dashboards, though. Alright, so for dashboards, they’re gonna ho- it’s gonna host it somewhere for you, so you can actually share it with your team if you wanted to. And so it’s gonna ask, I’m gonna access… This… we’ll open both of them. This one, it actually looks like it kind of gave me a little preview. In chat. Let’s see if we open in a new tab. Sometimes it’ll make you, like, log in. Yep. To wherever this, like… Place it’s hosting it. Continue, come on. Hmm, this one’s still building over here. Dashboards tend to take a while, but they are still quite a bit faster than humans.
Although this one isn’t loading. Trying to open it again. So yeah, I mean, it’s a little buggy. It doesn’t want to open this dashboard for me. Was anyone else able to create a dashboard? Hai did you happen to create a dashboard with yours?
Hai Guan: It did not run that prompt, actually. Yeah, I find sites to be pretty buggy.
Shane Butler: Yeah, I’ve found… a couple bugs I’ve found is sometimes it’ll just exit out of the entire conversation. I don’t have to reopen it, which isn’t a big deal, it’s, like, saved here. Yeah, the external… Sites for the dashboard when they work are cool. But, obviously, this one didn’t work. And I did find… the thing with the dashboards is, like, it doesn’t, it just kind of goes for it with the dashboard, and it is… usually what it seems to do is it puts, like, a few big numbers at the top, and then it has some charts, but… I wouldn’t say any of the dashboards are really, like.
anywhere near the grade of, like, a dashboard of, like, hey, I have someone on my team who works closely with the stakeholder that needs this dashboard to make decisions, and they’re gonna, like, optimize the dashboard in terms of, like, what charts and where everything is, to help them make that decision. I do find that that… it’s like… It’s more of, like, a, here’s a bunch of charts, we have a very generalized template, and we’re gonna throw everything on there kind of thing, which is… fine if someone just needs… wants it to, like, look at data, I guess, and, like, dig through it themselves, but… I think there’s definitely a lot of opportunity there. But we’ll see if it… see balance.
We’ll see if this… this opens up before we go. Anything… As this keeps going… I don’t know, any other thoughts from anyone else in terms of, like. You know, given… given how you kind of used it just in the past, you know, 45 minutes. Is there somewhere you would start using this week? Is there somewhere, like, you would trust it versus… versus you wouldn’t? Something… I think I would use it for, actually, is, like, trying to align on metrics with my team. I think it could be a pretty good tool to be, like. work with me to… looking at the data. What are all the possible, kind of, like, definitions of this metric?
That we could use for my team, and then… pull me the numbers for all, like, you know, maybe 7 of those different definitions, and then when I go to my team meeting, where we’re trying to, like. align on what definition we want to use going forward. We have, like, these artifacts of, like, these are the actual numbers for each one, like, which one actually makes sense for us, these are the definitions, and here’s the charts over time. Rather than, kind of, like. Trying to come up with a definition before you have any data to look at. This can suddenly do that really quickly, which is pretty cool. So that’s one place I would use it.
I think doing the charts… Like, these line charts pretty quickly, and editing them was kind of cool. Any thoughts from you, Hai? Places where you would… where you would use it?
Hai Guan: Yeah, probably deeper dive into, especially if you have, like, many, many segments to drill into, assuming that for example, there’s the definition. Like, if we imagine the revenue definition is already codified, and, like, there’s no dispute around that. It does it pretty, pretty well in terms of quickly slicing through things. And, you know, can quickly come up with, sort of, like, I don’t know, problem areas or whatever, if the idea is, like, for example, like a weekly metrics report kind of thing.
So for defined workflows, really well… codified definitions and aligned, sort of, like, historical data and stuff, I think it could do a really good job of that, of those, recurring stuff off the plate.
Shane Butler: Yeah, I think if you know the data you’re working with, yourself. I think this could speed up your workflow from doing a lot of that stuff on your own. Especially running things in parallel. I think another thing is, like, if you don’t know the data. I think it’s probably pretty good for, like, if you’re trying to do some, like, EDA, exploratory data analysis, discovery, you have some hypotheses in mind, and the members don’t have to be… 100% solid. They don’t even have to be, like, 70% solid, but they should be, like, directionally, correct for you to, like, decide, like.
hey, is it willing… is it worthwhile for us to, like, pursue further analysis in this area, or even think about this idea? I think it could be pretty good for that. I wouldn’t, like, report numbers in that case, or say, oh, this is truth. But I think that could be… I think it does open it up for people to start, like, exploring the data in that way. Alright, it is creating the dashboard HTML file. It’s created it. We’ll come back to it at the very end. I think we just have… Two more slides? Yeah, we’ll just wrap it up. I’ll come back to that dashboard at the end. But we do have some really cool, more free workshops coming up in the next month.
Next week, we are going to do a session on, analytics interviews. Kind of how those have changed over time. what people are asking in interviews nowadays. It’s like, spoiler alert, they’re all asking about AI. And then how to, like, really nail those, and use AI to help you prepare for those. Then we’ll get into some metrics ones after that. And then we’re gonna get into some context management engineering ones, pretty soon after that. So, lots of upcoming free workshops, you can check those out at maven.com slash dataneighbor. And then, if you want to go deeper. With us on our live cohorts.
you know, if you want to work with build your own data agent, or you want to figure out how to build around, like, existing data agents like this in such a way that they’re reliable, we have a couple courses. We have Agentic Analytics, Build an AI Analyst, 5-week course where you go end-to-end on, you know. the whole fundamentals of Agentic systems, the sort of, like. design and build workflow, the evals, context engineering, how to use open source models. And then we have AI analytics for everyone, which is more about, like. How do you, like, do that whole workflow, analytically thinking, and then execute that? in something like a data agent from ChatGPT.
So yeah, we offer 20% off both of those, We’ll send the links to with the, with those 20% promotions in an email as well, with the recording. Let’s see… Let’s see if this data… this, dashboard… ever worked. Well, this other one at least says it aired, so it knows. That aired. And… I don’t think so. Sorry, ChatGPT, but something’s broken with your dashboards. I don’t know if anyone else got a dashboard to work. Maybe it’s just me. It was working the other day. So it works sometimes, but I’d say that’s, not fully reliable thing yet you could be expecting if you want to whip up a dashboard right before a meeting, I’d have a backup plan, in case this thing fails.
Cool, but thanks for everyone for joining us, and hopefully we’ll, see you next week. Bye.
Hai Guan: Thanks, everybody.