Why can't AI tools count my reports?

Here is a simple test. Pick a type of report your company fills in regularly: an incident report, a near-miss report, a customer complaint form, a delivery check. Count how many were filed last quarter. Then ask an AI tool connected to your documents the same question.

In many cases, the answer will be wrong. Not slightly wrong: confidently, specifically wrong. "Your records show 3 near misses in the third quarter." There were 7.

This is not a fault in one product. It follows from how most of these tools work.

How most AI tools answer a question

When you ask a question, the tool does not read all of your documents. That would take far too long. Instead it does something closer to what you would do with a search engine (the method is called retrieval, or RAG):

1. It looks for the passages that best match your question: the paragraphs or pages most likely to contain the answer.
2. It takes the best few of those passages.
3. It writes an answer using only what is in them.

For most everyday questions, this works very well. "What notice do we need to give Northwind?" The answer is in one clause of one contract. If the tool finds that clause, it can answer correctly and show you where it came from.

Why counting breaks it

Now ask: "How many near misses did the Hamburg office report in the third quarter?"

The answer is not in any one passage. It is spread across every near-miss report Hamburg filed that quarter, possibly dozens, among hundreds from other offices and other quarters. To answer correctly, something has to look at all of them.

A tool that reads only the best few matches sees a few reports, counts those, and writes an answer. It has no way of knowing what it did not read. So it does not say "I only looked at five". It says "3". Confidently.

The same problem appears with any question that needs the whole picture:

• "How much did we spend with this supplier last year?"
• "Which month had the most customer complaints?"
• "How many of our vehicles are due for inspection in November?"
• "Has the number of incidents gone up since we changed the procedure?"

These are exactly the questions managers ask most.

What to look for instead

When you are choosing an AI tool for your company's records, ask it to count something you already know the answer to. Then check three things:

1. Is the number right? Not close: right.
2. Does the answer say what was counted? Which office, which dates, which kind of report. If the tool read your question differently from what you meant, this is where you will see it.
3. Does it admit what it cannot do? If a tool cannot count reliably, the honest answer is to say so, not to produce a number.

How AskNeil handles it

Neil answers everyday questions the way described above, with the sources shown for every answer.

For the reports and forms that matter most to your company, Neil goes further, and takes every single one into account. So counting, totalling and comparing questions get exact answers, and the answer states what was counted:

During set-up, you tell us which of your reports matter most, and we prepare Neil to count them. From then on, the questions you would normally answer by building a spreadsheet can simply be asked.

In short

Most AI tools read the few passages that best match your question. That is excellent for finding a clause and unreliable for counting anything. Before you trust a tool with your reports, ask it something you can check, and make sure the answer says what it counted.