1
00:00:00,160 --> 00:00:04,520
You switch to an AI model at half the
price. A month later, the

2
00:00:04,540 --> 00:00:06,760
bill is higher. Do you switch back?

3
00:00:07,000 --> 00:00:10,320
Before I do that, tell me what I got for
the extra money.

4
00:00:10,620 --> 00:00:14,420
Welcome to The AI Bill, Itemised. I'm
Alex.

5
00:00:14,720 --> 00:00:15,740
And I'm Jordan.

6
00:00:16,039 --> 00:00:20,160
If you use AI to get work done, this
show helps you decide

7
00:00:20,180 --> 00:00:24,400
what deserves your money. We look at
models, tools, and the way

8
00:00:24,420 --> 00:00:28,240
you use them, what they cost, what they
can do, and whether the

9
00:00:28,260 --> 00:00:30,060
results are worth paying for.

10
00:00:30,340 --> 00:00:34,220
Today, we're looking at a bill that went
up after a price cut.

11
00:00:34,700 --> 00:00:38,960
We'll work out what changed, then decide
what to check before cutting

12
00:00:38,980 --> 00:00:43,300
usage or changing models again because a
bigger bill could mean

13
00:00:43,320 --> 00:00:47,300
more useful work. It could also mean
you're paying for work

14
00:00:47,400 --> 00:00:48,460
nobody needs.

15
00:00:51,820 --> 00:00:56,100
We've built a fictional example so we
can follow the whole bill. The

16
00:00:56,120 --> 00:00:59,860
numbers are invented, and the
calculation is in the episode notes.

17
00:01:00,400 --> 00:01:02,280
This isn't a customer result.

18
00:01:02,540 --> 00:01:03,440
What's the job?

19
00:01:03,780 --> 00:01:07,680
Summarising documents for a team.
Someone sends in a

20
00:01:07,700 --> 00:01:11,880
document and gets back a summary.
They're supposed to be able to use

21
00:01:11,900 --> 00:01:15,900
that summary in their work. Last month,
the invoice was

22
00:01:15,920 --> 00:01:20,100
$140. The team switches to a model whose

23
00:01:20,120 --> 00:01:24,380
input and output prices are both half as
much. Same amount of

24
00:01:24,400 --> 00:01:27,860
work, same length of answer, and you'd
expect $70.

25
00:01:28,680 --> 00:01:30,220
Then the invoice arrives.

26
00:01:30,580 --> 00:01:31,900
One hundred and sixty.

27
00:01:32,200 --> 00:01:36,360
Twenty dollars more than before. That's
annoying, but I

28
00:01:36,380 --> 00:01:38,920
wouldn't change a working product over
that alone.

29
00:01:39,260 --> 00:01:43,420
Neither would I. The useful question is
why the expected saving

30
00:01:43,440 --> 00:01:47,440
disappeared. If the team doesn't
understand the change, they can't

31
00:01:47,460 --> 00:01:49,300
sensibly budget for the next one.

32
00:01:49,620 --> 00:01:52,040
So let's see whether the work stayed the
same.

33
00:01:52,600 --> 00:01:56,640
It didn't. The team processed 10,000
documents before the

34
00:01:56,660 --> 00:01:59,540
switch. Afterward, 16,000.

35
00:01:59,840 --> 00:02:03,020
That could be good. More people found a
use for it.

36
00:02:03,340 --> 00:02:07,480
It could, and it explains part of the
bill, but the new

37
00:02:07,500 --> 00:02:11,720
summaries also got longer. Twice as long
on average. The

38
00:02:11,740 --> 00:02:14,360
documents going in stayed the same size.

39
00:02:14,620 --> 00:02:16,880
Did somebody ask for longer summaries?

40
00:02:17,180 --> 00:02:21,340
We haven't established that. We know the
length changed. We don't

41
00:02:21,360 --> 00:02:23,140
yet know whether people wanted it.

42
00:02:23,440 --> 00:02:27,680
Okay. More documents and more writing
for each document.

43
00:02:28,060 --> 00:02:32,340
That explains how a cheaper model could
produce a bigger bill. But was

44
00:02:32,360 --> 00:02:34,280
each summary actually cheaper?

45
00:02:34,620 --> 00:02:38,940
Yes, in this example. Generating 1,000
summaries used to

46
00:02:38,960 --> 00:02:43,120
cost $14. With the new prices and longer
answers,

47
00:02:43,300 --> 00:02:44,500
it costs 10.

48
00:02:44,860 --> 00:02:49,140
So the unit cost came down. The total
went up because the team

49
00:02:49,160 --> 00:02:50,200
did more work.

50
00:02:50,480 --> 00:02:54,560
And the longer answers used up part of
the saving. If you follow the

51
00:02:54,580 --> 00:02:58,640
calculation in the notes, the extra
documents and extra output

52
00:02:58,820 --> 00:03:02,960
explain the whole gap between the
expected bill and the actual one.

53
00:03:03,400 --> 00:03:06,340
We haven't hidden a retry charge or a
fee in there.

54
00:03:06,660 --> 00:03:10,480
Then switching back could cost more for
the volume we now have.

55
00:03:10,800 --> 00:03:15,100
It could. Looking only at the invoice
total might push you toward the

56
00:03:15,120 --> 00:03:16,080
wrong decision.

57
00:03:16,620 --> 00:03:20,960
There's still something I don't like
about our comparison. You said

58
00:03:21,020 --> 00:03:25,160
1,000 summaries cost less. Are they
1,000 useful

59
00:03:25,180 --> 00:03:25,860
summaries?

60
00:03:26,200 --> 00:03:27,260
We don't know yet.

61
00:03:27,520 --> 00:03:31,580
Because I can imagine both versions of
this. Maybe the longer

62
00:03:31,600 --> 00:03:35,820
answer includes something people were
missing, or maybe people

63
00:03:35,840 --> 00:03:39,280
now have to dig through twice as much
text to find the point.

64
00:03:39,520 --> 00:03:43,520
And those lead to different decisions.
For this team, a useful

65
00:03:43,540 --> 00:03:47,520
summary needs to capture the main point
and the important caveat

66
00:03:47,700 --> 00:03:51,820
without inventing anything. That's the
standard we'd use to judge the

67
00:03:51,840 --> 00:03:52,460
result.

68
00:03:52,780 --> 00:03:56,440
Then I'd want someone to look at the
summaries against that standard.

69
00:03:57,040 --> 00:04:00,580
I wouldn't call the extra output waste
just because it's longer.

70
00:04:00,940 --> 00:04:04,820
Nor would I call it valuable just
because people generated it. The

71
00:04:04,840 --> 00:04:08,780
provider can charge us for an answer
whether anyone uses it or not.

72
00:04:09,060 --> 00:04:11,640
Can the bill tell us which answers were
useful?

73
00:04:12,000 --> 00:04:16,060
Usually not by itself. We need some
evidence from the work,

74
00:04:16,460 --> 00:04:20,580
a review result, a completed task, or
feedback from the person

75
00:04:20,600 --> 00:04:24,800
using the summary. If all we have is the
invoice, usefulness

76
00:04:24,880 --> 00:04:25,880
stays unknown.

77
00:04:26,260 --> 00:04:30,500
So we've answered why the money moved.
We haven't answered whether it

78
00:04:30,520 --> 00:04:31,600
was well spent.

79
00:04:31,920 --> 00:04:36,160
Right. Those are separate questions, and
we haven't priced human

80
00:04:36,180 --> 00:04:40,620
review time either. The dollar figures
we've discussed are model charges,

81
00:04:40,820 --> 00:04:42,900
not the whole cost of running the
product.

82
00:04:43,480 --> 00:04:47,780
My first instinct was to leave it alone
because the increase was small.

83
00:04:48,280 --> 00:04:52,540
Now, I'd keep the model for the moment,
but check what the longer answer is

84
00:04:52,560 --> 00:04:53,480
doing for us.

85
00:04:53,820 --> 00:04:54,860
How would you check?

86
00:04:55,180 --> 00:04:59,540
I'd take a small set of documents that
resemble the work people actually send

87
00:04:59,560 --> 00:05:03,780
in. Include a few difficult ones. Run
those through the

88
00:05:03,800 --> 00:05:07,980
current setup, then through a version
that asks for a shorter summary.

89
00:05:08,560 --> 00:05:10,860
Keep the model and the documents the
same.

90
00:05:11,140 --> 00:05:15,400
And decide what counts as a good summary
before looking at which version costs

91
00:05:15,420 --> 00:05:15,900
less.

92
00:05:16,280 --> 00:05:20,040
Yes. Does it capture the point? Does it
keep the important

93
00:05:20,060 --> 00:05:24,380
caveat? Can someone use it without
fixing it? If it drops an

94
00:05:24,400 --> 00:05:26,780
important detail, I wouldn't ship it.

95
00:05:27,080 --> 00:05:31,000
Count every billed attempt too. If
someone regenerates an

96
00:05:31,020 --> 00:05:35,040
answer, include that cost even if they
discard it. Some

97
00:05:35,060 --> 00:05:39,360
retries succeed, some don't. Either way,
they're part of what it

98
00:05:39,380 --> 00:05:40,740
took to finish the job.

99
00:05:41,060 --> 00:05:45,020
I'd also note whether the reviewer has
to spend longer fixing one

100
00:05:45,040 --> 00:05:49,020
version. I might save on the model and
create more work for a

101
00:05:49,040 --> 00:05:49,640
person.

102
00:05:50,300 --> 00:05:54,560
That gives us a useful first test. A
small sample can reveal

103
00:05:54,580 --> 00:05:58,520
problems. It won't tell us exactly how
every production task will

104
00:05:58,540 --> 00:06:02,768
behave. If the shorter setup looks
promising- Try it on a

105
00:06:02,788 --> 00:06:05,648
limited share of real work and check
again.

106
00:06:05,908 --> 00:06:09,908
And if the longer version is clearly
more useful, keep it and

107
00:06:09,928 --> 00:06:14,168
budget for it. The point of the check is
to make that a decision we can

108
00:06:14,188 --> 00:06:14,988
explain.

109
00:06:15,548 --> 00:06:19,588
You can start this on your own stack
without investigating everything at

110
00:06:19,608 --> 00:06:23,888
once. Pick one recurring job, answering
a support

111
00:06:23,908 --> 00:06:28,108
question, writing a document summary,
extracting information from a

112
00:06:28,128 --> 00:06:31,788
form, something where you can describe a
useful result.

113
00:06:32,128 --> 00:06:36,248
Then pick comparable periods. Don't
compare a quiet

114
00:06:36,268 --> 00:06:40,468
week with a launch week and assume the
model caused the difference. If

115
00:06:40,488 --> 00:06:42,768
the work changed, write that down.

116
00:06:43,708 --> 00:06:47,888
Look for what you spent, how many
original jobs you tried to complete,

117
00:06:48,068 --> 00:06:52,148
and how many results met your standard.
Count the original job

118
00:06:52,168 --> 00:06:55,808
once, even if getting it done took
several model calls.

119
00:06:56,248 --> 00:06:58,428
Keep all of those calls in the cost.

120
00:06:58,728 --> 00:07:02,748
And if the usage dashboard only gives me
requests, not completed

121
00:07:02,768 --> 00:07:03,308
jobs?

122
00:07:03,668 --> 00:07:07,848
Mark the task count as missing. Ask
whoever owns the workflow

123
00:07:07,968 --> 00:07:12,208
whether they can connect requests to
jobs. You can still investigate the

124
00:07:12,228 --> 00:07:16,528
spend. You just can't honestly calculate
cost per useful job yet.

125
00:07:16,828 --> 00:07:20,208
That's a better first finding than
dividing by the wrong thing.

126
00:07:20,528 --> 00:07:24,288
Once you can count useful results,
divide the model spend,

127
00:07:24,448 --> 00:07:28,388
including discarded calls, by the
original jobs that produced a

128
00:07:28,408 --> 00:07:32,568
useful result. If none did, report that
directly.

129
00:07:33,128 --> 00:07:36,908
Write down what you know, what's
missing, and the next check.

130
00:07:37,468 --> 00:07:41,388
You can start there even before you have
enough data for a calculation.

131
00:07:41,948 --> 00:07:46,208
Back to the opening. The model got
cheaper, but the bill went up.

132
00:07:46,588 --> 00:07:47,608
Do we switch back?

133
00:07:47,908 --> 00:07:52,008
Not on that evidence alone. In our
example, each summary

134
00:07:52,028 --> 00:07:56,148
became cheaper to generate. The team
processed more documents,

135
00:07:56,468 --> 00:08:00,608
and longer answers consumed some of the
saving. The next check is

136
00:08:00,628 --> 00:08:03,348
whether those answers are worth the
extra output.

137
00:08:03,648 --> 00:08:07,668
That's the question I'd take into a
budget meeting. What useful

138
00:08:07,688 --> 00:08:11,408
work did we get for this spend, and what
would we change next?

139
00:08:11,728 --> 00:08:15,808
That's what we'll keep investigating on
the AI Bill. When a model,

140
00:08:15,968 --> 00:08:20,048
a tool, or the way you use it changes,
we want you to be able to

141
00:08:20,068 --> 00:08:24,188
judge the trade-off. For this episode,
pick one workflow

142
00:08:24,268 --> 00:08:28,468
and write down what a useful result
looks like. Then check whether

143
00:08:28,488 --> 00:08:31,588
your cost data can tell you what it
takes to produce one.

144
00:08:32,408 --> 00:08:36,348
Start with one piece of work you
actually care about. Thanks for

145
00:08:36,368 --> 00:08:36,948
listening.