How to Know It Still Works Next Month

Questions this answersQuestions this answersfaq

can we just use a published AI benchmark?can we just use a published AI benchmark?why not a public benchmark?

They measure someone else's work. None of them contain your customers, your product names, your state's rules or the way your team writes. A model can score brilliantly on a public test and still be wrong about your catalogue, and that is the only question you actually have.They measure someone else's work. None of them contain your customers, your product names, your state's rules or the way your team writes. A model can score brilliantly on a public test and still be wrong about your catalogue, and that is the only question you actually have.Public benchmarks measure general capability on tasks that are not yours, and the widely published ones are old enough to have leaked into training data. Neither problem applies to thirty cases drawn from your own transcripts, which is why a small private set beats a large public one for deciding whether to ship a change.

how many test cases do we need?how many test cases do we need?how large should the set be?

Thirty is a real answer. Fewer than about twenty and a couple of lucky results move the score enough to fool you. The bigger risk is not size, it is building the set out of easy cases because they were quicker to write down.Thirty is a real answer. Fewer than about twenty and a couple of lucky results move the score enough to fool you. The bigger risk is not size, it is building the set out of easy cases because they were quicker to write down.Thirty to fifty for a first set. Below roughly twenty, ordinary variance swamps the effect you are trying to see. Composition matters more than count: include the failures that prompted the work, keep the ratio of hard to easy honest, and hold back a portion you do not look at while tuning.