Use case
Company knowledge search
Staff ask a question in plain language and get an answer drawn from your own documents, with citations, limited to what each person is already allowed to see.
The problem
How the work goes today
The answer exists somewhere: a policy PDF on SharePoint, a Confluence page, a precedent in the DMS, a ticket someone closed last year. Finding it means trying three search boxes and then asking the colleague who has been there longest.
There are often several versions of the same document. People find an old one, act on it, and nobody notices until something goes wrong. The newest version is not always the one search ranks first.
New joiners take months to learn where things live, and senior staff spend part of every day answering the same questions in chat. When those people leave, much of that knowledge goes with them.
28.6%
Drop in median time to resolve an issue after LinkedIn's support team started using retrieval over past tickets linked in a knowledge graph, measured over about six months in production.[1]
17–33%
Share of benchmark queries on which legal research tools built on retrieval still gave a hallucinated answer, across more than 200 preregistered questions. The vendors disputed parts of the method. Retrieval reduces made-up answers but does not remove them.[2]
98%
Share of Morgan Stanley financial-adviser teams actively using its internal assistant for finding information. Reported by OpenAI, the model supplier, and not independently audited.[3]
How it works
Step by step
Highlighted steps are where a person decides. Everything else runs on its own and is logged.
Systems involved
- Document stores (SharePoint, Google Drive, file shares)
- Wikis (Confluence, Notion)
- Legal and professional DMS (iManage, NetDocuments)
- Ticketing (Jira, ServiceNow, Zendesk)
- Identity and permissions (Entra ID, Okta)
- Chat surfaces (Microsoft Teams, Slack)
- Search and vector stores (Elasticsearch, OpenSearch, pgvector)
- 01Person
Approve sources
Content owners decide which SharePoint sites, Confluence spaces, DMS folders and ticket queues are indexed, and security signs off the permission mapping.
- 02System
Connect and sync permissions
Connectors pull documents and their access lists from each source and keep both in sync on a schedule.
- 03System
Parse and index
Documents, including tables and scanned PDFs, are parsed, split into passages, tagged with owner, date and version, and indexed for keyword and vector search.
- 04Person
Ask a question
A signed-in user asks in the web app, Microsoft Teams or Slack.
- 05Agent
Retrieve within permissions
The search index returns only passages the user can already open in the source system, then re-ranks them by relevance, recency and authority.
- 06Agent
Answer with citations
The model writes an answer that links each claim to the passage it came from, or says it couldn't find enough to answer.
- 07Person
Review regulated outputs
Answers that will go into client advice or policy statements are checked by a qualified person before they leave the team.
- 08System
Collect feedback
Wrong-answer flags and missing-document reports are routed to the owner of the source content as tasks.
- 09System
Audit
Each query, the documents retrieved and the answer given are logged for audit under your retention policy.
Measurement
What we measure
Agreed before anything is built, with a baseline taken from how the work runs today.
- Answer correctness
- Share of 100 to 300 real questions, written with subject experts before launch, where the answer matches the agreed gold answer. Re-run on every change.
- Citation accuracy
- Share of citations in a sample that actually support the sentence they are attached to, checked by a reviewer.
- Abstention quality
- Share of unanswerable test questions correctly declined, reported next to the share of answerable questions wrongly declined.
- Permission-leak rate
- Answers that cite a document the asking user cannot open, tested with low-privilege test accounts before every release. The target is zero.
- Time to answer
- Median time from question to a satisfactory answer, compared with a baseline timed on the same questions using existing search and colleagues.
- Content freshness
- Share of answers that cite a document past its review date, used to prompt owners to update or retire content.
Pitfalls
What usually goes wrong
- Someone sees a document they shouldn't
- Filter by the source system's access lists at retrieval time, not through instructions in the prompt, and test with low-privilege accounts before each release.
- Old versions give confident wrong answers
- Curate sources with their owners, rank recent and authoritative documents higher, and show the document date next to every citation.
- Tables and scans come out garbled
- Use layout-aware parsing and include your worst real documents in the evaluation set.
- Quality judged on impressions
- Write a set of real questions with gold answers before launch and re-run it on every change to sources, prompts or models.
- People trust every answer
- Put citations where they are easy to check and treat 'I couldn't find this' as an acceptable answer.
Build or buy
When a ready-made tool is enough
Custom work isn't always the answer. Here's how we'd decide.
- Microsoft 365 Copilot, Glean, Google Gemini Enterprise, Atlassian Rovo and Guru are enough when most content lives in mainstream SaaS tools with good connectors and standard permissions.
- Custom work pays off when the important sources are niche or legacy (a DMS, file shares, databases, ERP records), or when data residency calls for an on-premise or private model.
- It also makes sense when retrieval has to be tuned and evaluated on specialist material such as legal precedent, engineering specs or regulation, or when answers need to feed other workflows and agents.
- On a large workforce, compare per-seat licence costs with a usage-priced stack you run yourself.
Industries
Where this applies
Sources
- 1.Xu et al. (LinkedIn), Retrieval-augmented generation with knowledge graphs for customer service QA, SIGIR 2024 (Apr 2024)
- 2.Stanford HAI, AI on trial: legal models hallucinate in 1 out of 6 or more benchmarking queries (May 2024)
- 3.OpenAI, Morgan Stanley customer story
Figures are as reported by each source on its publication date. Examples are public industry cases, not AI Sewa clients.
Next step