- Look beyond the demo answer. Ask how the system handles your records as they really are: scattered, duplicated, and partly in the heads of people who will leave.
- The essentials: your existing permissions, sources for every answer, honesty when the answer is not there, and a clear way to handle versions that disagree.
- The differences show in four tests: a counting question, two versions of a document, a question your records cannot answer, and a leaver's email.
- Ask who sets it up, who looks after it, and whether it can grow from answering questions to handling routine work.
Almost every AI system for company knowledge gives a good answer in a demo. The demo documents are tidy, the questions are chosen in advance, and the person asking knows exactly where the answer is.
Your records are not like that. They are spread across email, shared drives and chats. They come in several versions with almost the same name. Some of the most important answers sit in the mailbox of someone who left last year. A system that will serve you for years has to work with records as they really are.
Twelve things to look for
Your records, as they are
1. It works with what you already have. Email, shared drives, document libraries and chats, without moving or renaming anything. And not only today's records: old email archives and exports from your business systems often hold the history people ask about most. And in every language your company works in: staff should be able to ask in their own language, find documents written in another, and use the system itself in their language.
2. It is not tied to one platform. Companies change their email and file systems, merge with others, and acquire companies on different platforms. A system that works with more than one platform protects your choice later.
3. It follows the permissions you already have. Each person should get answers only from what they can already open, checked before any answer is written, and kept up to date as people join, move and leave.
Answers you can trust
4. Every answer shows its source. The document, the email or the chat, and the exact passage, one click away. An answer you cannot check is an answer you cannot act on.
5. It says when the answer is not there. "This is not in your records" is one of the most valuable answers a system can give. Be wary of a system that fills gaps with general knowledge from the internet, so that the answer sounds right but is not your policy.
6. It handles versions that disagree. "Prices_FINAL_v3" and "Prices_FINAL_v3_new" exist in every company. A good system tells you when two documents disagree, names both with their dates, and says which is newer, instead of quietly picking one.
7. It can count. "How many complaints did we have in March?" needs every complaint, not the few that best match the question. Most systems read only the best few matches, and get counting questions wrong with confidence.
People, now and later
8. Knowledge stays when people leave. When someone resigns, falls ill or leaves without notice, their mailbox often holds the most complete record of what they agreed. Look for a simple, controlled way to let a successor search it for a set period, with every answer from it clearly marked.
9. New starters find answers from day one. Without interrupting colleagues, and without anyone reading their questions. People ask freely only when they know their questions are private.
10. Managers can see what is missing. Which topics come up most, and which questions the records cannot answer, so you know exactly what to write down next. Topics and trends, never individual conversations.
Looking after it, and growing with it
11. Someone sets it up and looks after it for you. Connecting systems, checking permissions, choosing what stays out, and keeping it all working as things change. Ask who does this work, and who you call when something is wrong: a call centre, or the people who build it.
12. It can grow from answering to doing. Today you want answers. Tomorrow you will want routine work handled: filing, chasing, reminders, first drafts. Choose a system that can take on defined tasks, set up and looked after for you, rather than one you will need to replace.
Four tests to run before you choose
Do not rely on the vendor's questions. Bring your own documents and run these four tests, in a demo or a trial.
- 1A counting question
Ask how many reports of one kind were filed last quarter, and check the number against your own count. If counting has to be set up first, ask how long that takes and what it costs.
- 2Two versions
Include two versions of a price list or procedure that disagree. See whether the answer notices.
- 3An answer that is not there
Ask something your documents do not cover. The right answer is to say so.
- 4A leaver's email
Ask about an agreement made only in a former colleague's email. See how, and whether, it can be found.
One more choice: where it runs
Most companies are well served by a cloud service with strong permissions and clear contracts. Some have contracts, regulators or policies that require their records to stay in their own building. If that could ever apply to you, ask now whether the system can run on your own servers, and whether anything at all would leave the building.
In short
Every AI knowledge system looks good in a demo. Choose one that works with your records as they really are, shows its sources, admits what it does not know, handles versions and counting, keeps knowledge when people leave, and is set up and looked after for you. Then test it yourself, with your own documents and your own hardest questions.