Development
How to Prepare a Knowledge Base for an Internal AI Assistant

Prepare a knowledge base for an internal AI assistant by fixing source ownership, access rules, document quality, and evaluation questions before indexing. A model cannot reliably compensate for contradictory policies or determine which employee is allowed to read a document unless the system supplies that boundary.
The first deliverable should be a small, trusted collection with clear limits. Importing everything is an ingestion milestone, not evidence that the assistant is useful.
Prepare the knowledge base around real questions
Choose one bounded audience and task. A support assistant answering product configuration questions is different from an engineering assistant retrieving incident procedures. Start with questions the intended users actually ask and identify the authoritative source for each answer.
Use a worksheet like this hypothetical example:
| Question | Authoritative source | Owner | Access | Review trigger |
|---|---|---|---|---|
| How do we roll back a release? | Release runbook | Platform lead | Engineering | Deployment change |
| What is the refund policy? | Approved support policy | Support lead | Support | Policy change |
| Who owns this integration? | Service catalogue | Engineering manager | Staff | Ownership change |
These are planning examples, not claims about a particular company's policies. If two sources disagree, resolve ownership before the assistant must choose between them.
Make each document independently understandable
A useful document states its subject, intended audience, current status, and important prerequisites. Replace unexplained references such as “do the usual reset” with a linked procedure. Preserve headings, code blocks, warnings, and the relationship between a command and the environment where it is valid.
Remove navigation noise and duplicate exports. Keep older versions only when the use case requires historical answers, and mark them clearly. A current procedure and an obsolete copy should not look equally authoritative.
Do not cut a warning away from the action it qualifies just to fit a uniform chunk size. Test document boundaries against the questions the assistant needs to answer. Microsoft's retrieval-augmented generation overview explains why relevant retrieval, access controls, and bounded context are central to the system.
Preserve permissions through retrieval
Record the source document's permission model alongside its content. Enforce access before retrieved passages reach the model. A final answer filter cannot reliably undo disclosure that has already happened in model context or logs.
Test with at least two identities that have different access. Include a document the less privileged identity must never retrieve, a recently revoked permission, and a source that changed groups. Verify search results, citations, conversation history, caches, and operational logs.
Deleting a source or revoking permission must have an operational path through the index and caches. Define who owns that path and how quickly it should take effect. Avoid promising instant revocation until the implementation demonstrates it.
Build a small evaluation set before ingestion grows
For each test question, store the permitted identity, expected source, essential facts, and acceptable refusal behavior. Include:
- A direct question with one authoritative answer.
- A question requiring two permitted sources.
- A plausible question with no supported answer.
- An obsolete procedure that should not win retrieval.
- A restricted document requested by an unauthorized user.
- A document containing instructions that conflict with the assistant's operating rules.
Retrieved text is evidence, not authority to change system behavior. OWASP's prompt injection guidance describes the risk from malicious instructions embedded in external content.
Measure retrieval separately from answer quality. If the right passage never enters context, changing the answer prompt may hide the underlying retrieval problem.
Assign a maintenance owner
Every source needs a responsible owner, a refresh mechanism, and a deletion policy. Track index lag and failed updates. Keep an incident procedure for an assistant citing the wrong policy or exposing a restricted passage.
The project documentation approach helps establish discoverable sources before introducing an assistant. AI should consume a maintained knowledge system rather than become the only place where institutional knowledge can be found.
How much content should the first version include?
Enough to answer a bounded collection of real questions with traceable sources. There is no universal document count or chunk size that proves readiness. Expand after the evaluation set shows useful answers, appropriate refusals, and working permissions.
For help defining that first collection and its acceptance criteria, scope an internal AI knowledge review.