Techbanq

AI · retrieval

The model is not what makes this hard

A retrieval system answers from the documents you give it. When it answers wrongly, the cause is almost always in those documents — three versions of a policy with no way to tell which is current, a table that became prose, a file the asker should not be able to read.

Serving Saudi Arabia from our Riyadh office and our engineering hubs in Lahore and Karachi

Operational analytics dashboard displayed on a desk monitor
  1. 01

    Market

    Saudi Arabia

  2. 02

    Written for

    CTO, Head of Digital

  3. 03

    Decision

    Build a retrieval system over internal documents that staff will trust

  4. 04

    Languages

    EN · AR

Enterprise delivery credentials

  • ISO

    ISO/IEC 27001

    ISO/IEC 27001 certified

Certifications & registrations

The pipeline

From an enterprise document to a cited answer

The model is the last stage and the least of the work. Everything that decides whether an answer is right happens before it.

  1. 01

    Enterprise sources

    The document system, the shared drive, the ticket history, the policy set — with an owner named for each and superseded material excluded.

    • SharePoint
    • Drive
    • ITSM
  2. 02

    Ingestion

    Parsing that keeps structure: tables stay tables, headings stay headings, and scanned Arabic is read rather than skipped.

    • OCR
    • Parsers
  3. 03

    Permissions

    Every chunk carries the access of the document it came from, evaluated against the asker at query time rather than at indexing time.

    • Entra ID
    • ACLs
  4. 04

    Indexing and retrieval

    Hybrid keyword and semantic search with Arabic normalisation and stemming, because a query typed the way people type it has to match.

    • Vector store
    • BM25
  5. 05

    Model

    Answers only from what retrieval returned, and says so when that is not enough rather than assembling something plausible.

    • LLM
  6. 06

    Answer or action

    Read-only by default. Where an action follows, it runs through the same permission model as the retrieval that informed it.

    • Workflow
  7. 07

    Citations and audit

    Document and version behind every answer, and a log of who asked what, which is the first thing an internal auditor requests.

    • Audit log

Refusal path

Where the corpus does not answer the question, the system says so and names who does. Measuring refusal is as important as measuring accuracy: a system that never refuses is inventing.

The corpus

Four properties of your documents decide the outcome

The first is whether the corpus knows what is current. Most document stores contain several versions of the same policy, a draft that was never marked as one, and a superseded procedure that reads authoritative. A retrieval system has no way to prefer the current one unless something in the corpus says which it is, and a confidently delivered answer from a two-year-old document is worse than no system at all, because staff act on it.

The second is whether retrieval respects permissions. If the index is built once over everything and queried by everyone, the system will eventually answer a question using a document the asker could not open — a salary band, a disciplinary record, a contract with commercial terms. Retrieving under the asker's own permissions rather than the indexer's is an architectural decision, and it is far harder to add afterwards than to build in.

The third is structure. A large share of the answers people actually want live in tables, forms and appendices — a fee schedule, an approval threshold, an entitlement matrix. Extracted naively, a table becomes a run of text in which the relationship between a row and a column is gone, and the system will confidently return a number from the wrong row. Recovering that structure is unglamorous work and it moves the result more than any model choice.

The fourth is language. Where a corpus is genuinely bilingual — an Arabic policy with an English annexe, a form with both — retrieval has to work across both from a question in either. That is a decision about how the index is built rather than a setting to enable, and testing it needs questions in both languages against documents in both, which is not what a default evaluation set contains.

How it is built

Corpus first, model last

STAGE 01

1 · Corpus audit

What is in the store, how much is duplicated, how much is superseded, what carries a permission and what has structure that naive extraction would lose. The output is frequently useful to the organisation on its own. A picture of what the system would be answering from.

STAGE 02

2 · Evaluation set from real questions

Fifty to a hundred questions staff actually ask, with the correct answer and its source recorded — plus the questions the system must refuse, because refusal behaviour is what makes it trustworthy. A measurable target, agreed before anything is built.

STAGE 03

3 · Preparation and indexing

Deduplication, version resolution, structure recovery, and an index built to be queried under the asker's permissions rather than a service account's. The part that determines most of the result.

STAGE 04

4 · Retrieval tuning and refusal

Measured against the evaluation set, with the model chosen last because it is the cheapest variable to change and the least decisive. Refusal is tuned as deliberately as answering. A system with a known accuracy on your own questions.

Failure modes

What a pilot usually gets wrong

  1. 01

    Answering from a superseded document

    The most damaging failure because it is invisible: the answer is well-formed, sourced and wrong. Staff lose confidence in the system permanently after two or three of these, and rebuilding that confidence is harder than building the system.

  2. 02

    Answering a question it should refuse

    Either because the source is outside the asker's permissions, or because the corpus simply does not contain the answer and the system reached for something adjacent. Both are fixed by measuring refusal explicitly rather than treating it as an absence of success.

  3. 03

    Losing the table

    A fee schedule flattened into prose, and the system returns a number from a neighbouring row. Common enough that any pilot whose evaluation set has no table questions in it is not measuring the thing that will break.

  4. 04

    Working in one language and not the other

    An English question retrieves the English annexe and misses the Arabic policy that governs it. Only found by evaluating in both languages, which is why the evaluation set is built before the system rather than after it.

Scope boundary

Where we put a system to work

What we do

  • Agree the evaluation set before building, so the pilot has a measurable result rather than an impression.
  • Build retrieval under the asker's permissions, and show the source with every answer so it can be checked.
  • Report the accuracy we measure on your corpus, including where it is not good enough to deploy.

Where the corpus contains personal data, the obligations on processing and residency come from the data protection regulator and apply to the index and the model provider as much as to the original store. Establish that before choosing where the system runs.

Primary source

Common questions

What buyers ask

Can we use a hosted model, or does everything have to run locally?+

It depends on what the corpus contains and what your obligations are, and that is a determination for your legal and compliance function rather than a technical preference. What is worth knowing technically: the retrieval layer and the index can sit inside your environment regardless, so the question narrows to whether retrieved passages may be sent to a hosted model. Several providers now offer regional deployment, which changes the answer for many organisations.

How large does the corpus need to be?+

Smaller than most people assume, and quality matters far more than volume. A few hundred well-maintained documents produce a system staff trust; fifty thousand documents of unknown currency produce one they stop using after a month. If the choice is between indexing everything and indexing the two hundred documents that are actually current and authoritative, index the two hundred.

What does a pilot cost and how long does it take?+

Six to ten weeks for one document set, and the largest share of that is the corpus audit and the evaluation set rather than engineering. Where the documents are already versioned and permissioned, the shorter end is realistic. Where nobody can say which version of a policy is current, that question has to be answered by the organisation before the pilot can produce a meaningful number, and it is worth answering regardless of whether the system gets built.

Next step

Start with a corpus audit

A pilot on one document set with an evaluation built from real questions, including the ones the system should refuse.