WEBVTT 00:00:00.160 --> 00:00:04.520 You switch to an AI model at half the price. A month later, the 00:00:04.540 --> 00:00:06.760 bill is higher. Do you switch back? 00:00:07.000 --> 00:00:10.320 Before I do that, tell me what I got for the extra money. 00:00:10.620 --> 00:00:14.420 Welcome to The AI Bill, Itemised. I'm Alex. 00:00:14.720 --> 00:00:15.740 And I'm Jordan. 00:00:16.039 --> 00:00:20.160 If you use AI to get work done, this show helps you decide 00:00:20.180 --> 00:00:24.400 what deserves your money. We look at models, tools, and the way 00:00:24.420 --> 00:00:28.240 you use them, what they cost, what they can do, and whether the 00:00:28.260 --> 00:00:30.060 results are worth paying for. 00:00:30.340 --> 00:00:34.220 Today, we're looking at a bill that went up after a price cut. 00:00:34.700 --> 00:00:38.960 We'll work out what changed, then decide what to check before cutting 00:00:38.980 --> 00:00:43.300 usage or changing models again because a bigger bill could mean 00:00:43.320 --> 00:00:47.300 more useful work. It could also mean you're paying for work 00:00:47.400 --> 00:00:48.460 nobody needs. 00:00:51.820 --> 00:00:56.100 We've built a fictional example so we can follow the whole bill. The 00:00:56.120 --> 00:00:59.860 numbers are invented, and the calculation is in the episode notes. 00:01:00.400 --> 00:01:02.280 This isn't a customer result. 00:01:02.540 --> 00:01:03.440 What's the job? 00:01:03.780 --> 00:01:07.680 Summarising documents for a team. Someone sends in a 00:01:07.700 --> 00:01:11.880 document and gets back a summary. They're supposed to be able to use 00:01:11.900 --> 00:01:15.900 that summary in their work. Last month, the invoice was 00:01:15.920 --> 00:01:20.100 $140. The team switches to a model whose 00:01:20.120 --> 00:01:24.380 input and output prices are both half as much. Same amount of 00:01:24.400 --> 00:01:27.860 work, same length of answer, and you'd expect $70. 00:01:28.680 --> 00:01:30.220 Then the invoice arrives. 00:01:30.580 --> 00:01:31.900 One hundred and sixty. 00:01:32.200 --> 00:01:36.360 Twenty dollars more than before. That's annoying, but I 00:01:36.380 --> 00:01:38.920 wouldn't change a working product over that alone. 00:01:39.260 --> 00:01:43.420 Neither would I. The useful question is why the expected saving 00:01:43.440 --> 00:01:47.440 disappeared. If the team doesn't understand the change, they can't 00:01:47.460 --> 00:01:49.300 sensibly budget for the next one. 00:01:49.620 --> 00:01:52.040 So let's see whether the work stayed the same. 00:01:52.600 --> 00:01:56.640 It didn't. The team processed 10,000 documents before the 00:01:56.660 --> 00:01:59.540 switch. Afterward, 16,000. 00:01:59.840 --> 00:02:03.020 That could be good. More people found a use for it. 00:02:03.340 --> 00:02:07.480 It could, and it explains part of the bill, but the new 00:02:07.500 --> 00:02:11.720 summaries also got longer. Twice as long on average. The 00:02:11.740 --> 00:02:14.360 documents going in stayed the same size. 00:02:14.620 --> 00:02:16.880 Did somebody ask for longer summaries? 00:02:17.180 --> 00:02:21.340 We haven't established that. We know the length changed. We don't 00:02:21.360 --> 00:02:23.140 yet know whether people wanted it. 00:02:23.440 --> 00:02:27.680 Okay. More documents and more writing for each document. 00:02:28.060 --> 00:02:32.340 That explains how a cheaper model could produce a bigger bill. But was 00:02:32.360 --> 00:02:34.280 each summary actually cheaper? 00:02:34.620 --> 00:02:38.940 Yes, in this example. Generating 1,000 summaries used to 00:02:38.960 --> 00:02:43.120 cost $14. With the new prices and longer answers, 00:02:43.300 --> 00:02:44.500 it costs 10. 00:02:44.860 --> 00:02:49.140 So the unit cost came down. The total went up because the team 00:02:49.160 --> 00:02:50.200 did more work. 00:02:50.480 --> 00:02:54.560 And the longer answers used up part of the saving. If you follow the 00:02:54.580 --> 00:02:58.640 calculation in the notes, the extra documents and extra output 00:02:58.820 --> 00:03:02.960 explain the whole gap between the expected bill and the actual one. 00:03:03.400 --> 00:03:06.340 We haven't hidden a retry charge or a fee in there. 00:03:06.660 --> 00:03:10.480 Then switching back could cost more for the volume we now have. 00:03:10.800 --> 00:03:15.100 It could. Looking only at the invoice total might push you toward the 00:03:15.120 --> 00:03:16.080 wrong decision. 00:03:16.620 --> 00:03:20.960 There's still something I don't like about our comparison. You said 00:03:21.020 --> 00:03:25.160 1,000 summaries cost less. Are they 1,000 useful 00:03:25.180 --> 00:03:25.860 summaries? 00:03:26.200 --> 00:03:27.260 We don't know yet. 00:03:27.520 --> 00:03:31.580 Because I can imagine both versions of this. Maybe the longer 00:03:31.600 --> 00:03:35.820 answer includes something people were missing, or maybe people 00:03:35.840 --> 00:03:39.280 now have to dig through twice as much text to find the point. 00:03:39.520 --> 00:03:43.520 And those lead to different decisions. For this team, a useful 00:03:43.540 --> 00:03:47.520 summary needs to capture the main point and the important caveat 00:03:47.700 --> 00:03:51.820 without inventing anything. That's the standard we'd use to judge the 00:03:51.840 --> 00:03:52.460 result. 00:03:52.780 --> 00:03:56.440 Then I'd want someone to look at the summaries against that standard. 00:03:57.040 --> 00:04:00.580 I wouldn't call the extra output waste just because it's longer. 00:04:00.940 --> 00:04:04.820 Nor would I call it valuable just because people generated it. The 00:04:04.840 --> 00:04:08.780 provider can charge us for an answer whether anyone uses it or not. 00:04:09.060 --> 00:04:11.640 Can the bill tell us which answers were useful? 00:04:12.000 --> 00:04:16.060 Usually not by itself. We need some evidence from the work, 00:04:16.460 --> 00:04:20.580 a review result, a completed task, or feedback from the person 00:04:20.600 --> 00:04:24.800 using the summary. If all we have is the invoice, usefulness 00:04:24.880 --> 00:04:25.880 stays unknown. 00:04:26.260 --> 00:04:30.500 So we've answered why the money moved. We haven't answered whether it 00:04:30.520 --> 00:04:31.600 was well spent. 00:04:31.920 --> 00:04:36.160 Right. Those are separate questions, and we haven't priced human 00:04:36.180 --> 00:04:40.620 review time either. The dollar figures we've discussed are model charges, 00:04:40.820 --> 00:04:42.900 not the whole cost of running the product. 00:04:43.480 --> 00:04:47.780 My first instinct was to leave it alone because the increase was small. 00:04:48.280 --> 00:04:52.540 Now, I'd keep the model for the moment, but check what the longer answer is 00:04:52.560 --> 00:04:53.480 doing for us. 00:04:53.820 --> 00:04:54.860 How would you check? 00:04:55.180 --> 00:04:59.540 I'd take a small set of documents that resemble the work people actually send 00:04:59.560 --> 00:05:03.780 in. Include a few difficult ones. Run those through the 00:05:03.800 --> 00:05:07.980 current setup, then through a version that asks for a shorter summary. 00:05:08.560 --> 00:05:10.860 Keep the model and the documents the same. 00:05:11.140 --> 00:05:15.400 And decide what counts as a good summary before looking at which version costs 00:05:15.420 --> 00:05:15.900 less. 00:05:16.280 --> 00:05:20.040 Yes. Does it capture the point? Does it keep the important 00:05:20.060 --> 00:05:24.380 caveat? Can someone use it without fixing it? If it drops an 00:05:24.400 --> 00:05:26.780 important detail, I wouldn't ship it. 00:05:27.080 --> 00:05:31.000 Count every billed attempt too. If someone regenerates an 00:05:31.020 --> 00:05:35.040 answer, include that cost even if they discard it. Some 00:05:35.060 --> 00:05:39.360 retries succeed, some don't. Either way, they're part of what it 00:05:39.380 --> 00:05:40.740 took to finish the job. 00:05:41.060 --> 00:05:45.020 I'd also note whether the reviewer has to spend longer fixing one 00:05:45.040 --> 00:05:49.020 version. I might save on the model and create more work for a 00:05:49.040 --> 00:05:49.640 person. 00:05:50.300 --> 00:05:54.560 That gives us a useful first test. A small sample can reveal 00:05:54.580 --> 00:05:58.520 problems. It won't tell us exactly how every production task will 00:05:58.540 --> 00:06:02.768 behave. If the shorter setup looks promising- Try it on a 00:06:02.788 --> 00:06:05.648 limited share of real work and check again. 00:06:05.908 --> 00:06:09.908 And if the longer version is clearly more useful, keep it and 00:06:09.928 --> 00:06:14.168 budget for it. The point of the check is to make that a decision we can 00:06:14.188 --> 00:06:14.988 explain. 00:06:15.548 --> 00:06:19.588 You can start this on your own stack without investigating everything at 00:06:19.608 --> 00:06:23.888 once. Pick one recurring job, answering a support 00:06:23.908 --> 00:06:28.108 question, writing a document summary, extracting information from a 00:06:28.128 --> 00:06:31.788 form, something where you can describe a useful result. 00:06:32.128 --> 00:06:36.248 Then pick comparable periods. Don't compare a quiet 00:06:36.268 --> 00:06:40.468 week with a launch week and assume the model caused the difference. If 00:06:40.488 --> 00:06:42.768 the work changed, write that down. 00:06:43.708 --> 00:06:47.888 Look for what you spent, how many original jobs you tried to complete, 00:06:48.068 --> 00:06:52.148 and how many results met your standard. Count the original job 00:06:52.168 --> 00:06:55.808 once, even if getting it done took several model calls. 00:06:56.248 --> 00:06:58.428 Keep all of those calls in the cost. 00:06:58.728 --> 00:07:02.748 And if the usage dashboard only gives me requests, not completed 00:07:02.768 --> 00:07:03.308 jobs? 00:07:03.668 --> 00:07:07.848 Mark the task count as missing. Ask whoever owns the workflow 00:07:07.968 --> 00:07:12.208 whether they can connect requests to jobs. You can still investigate the 00:07:12.228 --> 00:07:16.528 spend. You just can't honestly calculate cost per useful job yet. 00:07:16.828 --> 00:07:20.208 That's a better first finding than dividing by the wrong thing. 00:07:20.528 --> 00:07:24.288 Once you can count useful results, divide the model spend, 00:07:24.448 --> 00:07:28.388 including discarded calls, by the original jobs that produced a 00:07:28.408 --> 00:07:32.568 useful result. If none did, report that directly. 00:07:33.128 --> 00:07:36.908 Write down what you know, what's missing, and the next check. 00:07:37.468 --> 00:07:41.388 You can start there even before you have enough data for a calculation. 00:07:41.948 --> 00:07:46.208 Back to the opening. The model got cheaper, but the bill went up. 00:07:46.588 --> 00:07:47.608 Do we switch back? 00:07:47.908 --> 00:07:52.008 Not on that evidence alone. In our example, each summary 00:07:52.028 --> 00:07:56.148 became cheaper to generate. The team processed more documents, 00:07:56.468 --> 00:08:00.608 and longer answers consumed some of the saving. The next check is 00:08:00.628 --> 00:08:03.348 whether those answers are worth the extra output. 00:08:03.648 --> 00:08:07.668 That's the question I'd take into a budget meeting. What useful 00:08:07.688 --> 00:08:11.408 work did we get for this spend, and what would we change next? 00:08:11.728 --> 00:08:15.808 That's what we'll keep investigating on the AI Bill. When a model, 00:08:15.968 --> 00:08:20.048 a tool, or the way you use it changes, we want you to be able to 00:08:20.068 --> 00:08:24.188 judge the trade-off. For this episode, pick one workflow 00:08:24.268 --> 00:08:28.468 and write down what a useful result looks like. Then check whether 00:08:28.488 --> 00:08:31.588 your cost data can tell you what it takes to produce one. 00:08:32.408 --> 00:08:36.348 Start with one piece of work you actually care about. Thanks for 00:08:36.368 --> 00:08:36.948 listening.