TL;DR: I built an open-source scoped app that answers questions from your knowledge articles and catalog items with any OpenAI-compatible model. Plain tables are the vector store; a scope-routing step keeps searches fast; profiles, prompts and model parameters are records; business rules keep the index in sync. It ships with a Virtual Agent topic, Flow Designer actions and subflows, and a REST API. MIT licensed: https://github.com/andreiguimaraess/rag-kit
Why I built it
Most "chat with your knowledge base" demos stop at the happy path: embed some articles, run a similarity search, send the top results to a model. On a real instance, the hard parts are elsewhere:
- The store keeps growing, and scoring every vector on every question gets slow.
- Different audiences need different slices of the content: only English articles, only the IT knowledge base, only items of one catalog, only what this user is allowed to read.
- Articles change every day. Retired articles must disappear from the answers, edited ones must be re-embedded, and nobody wants to run a nightly batch job.
- Prompts and model parameters change more often than code, and they should not require a deployment.
RAG Kit is my take on those parts, built only with what the platform gives us: GlideRecord, RESTMessageV2, events, Flow Designer and Virtual Agent. No MID Server, no plugin to activate, no external vector database.
How it works
1. A vector store in a table
The Vector Chunk table holds one row per chunk: the source record's class and sys_id (a document ID field, so it works for any table), the list of scopes it belongs to, the exact text that was embedded, the normalized vector as JSON, a SHA-256 hash of model + text, and a JSON snapshot of the record's populated fields. Long articles are split by paragraph, then by sentence, with overlap.
Because vectors are stored L2-normalized, cosine similarity is a dot product, which is cheap enough in server-side JavaScript for thousands of candidates.
2. Scope routing before similarity
This is the part I like most. Each knowledge base and each catalog becomes a Retrieval Scope (one button generates them). A scope has a summary (its name, description and the titles of its records) and that summary is embedded too.
A question is first compared with the scope summaries, which is a handful of vectors. Only the chunks of the best scopes become candidates for similarity scoring. "How do I connect to the VPN?" goes to the IT knowledge base and the technical catalog, and HR content is never scored. If no scope is confident enough, it falls back to all scopes, and the Retrieval Log tells you it did.
3. Profiles decide what a caller may search
A Retrieval Profile is a named search configuration that callers pick by code:
Profile Source Restriction
| kb_all | Knowledge articles | routed to the best 2 knowledge bases |
| kb_it | Knowledge articles | only the IT knowledge base (explicit scope list) |
| catalog | Catalog items | only items of the Service Catalog (source filter) |
A profile also carries the thresholds, top N, "max chunks per record" (1 means you never get the same article twice), the default prompt, and Respect record access. With that flag on, candidate records are checked with GlideRecordSecure as the calling user, so knowledge user criteria and catalog visibility apply to the answers. Callers can combine profiles (kb_all,catalog) and add an extra encoded query on top.
The final candidate set is profile filter + caller filter + selected scopes + ACLs, all done with database queries before a single vector is parsed.
4. Prompts and model parameters are records
A Prompt Template holds the system prompt, the user and passage templates (with placeholders like {{query}}, {{context}}, {{language}}) and every model parameter: model, API style (Chat Completions or Responses), temperature, top_p, max output tokens, reasoning effort, verbosity, response format and a free JSON field for anything else. Empty fields are not sent, so reasoning models that reject temperature just work. The default prompt answers only from the passages, cites them as [1], [2], and replies with a no-answer token when the passages do not contain the answer. Answers come back in the language of the question.
5. The index maintains itself
An after business rule on kb_knowledge and sc_cat_item queues an event; a script action calls an idempotent indexer that makes the store match the record:
- published or edited: chunks created or updated, and only changed chunks are re-embedded (the hash), so editing one paragraph costs one embedding call;
- retired, outdated or deactivated: chunks removed;
- deleted: chunks removed in the same transaction;
- updates that only touch view counts or ratings are ignored.
Scopes are maintained too: changing a scope's condition reassigns chunks, editing its description re-embeds its summary, and a daily job refreshes summaries. Adding another table (problems, HR articles, a custom FAQ) is a new Index Source record plus a one-line business rule.
6. One contract, many entry points
Everything goes through one Script Include, RagKitFlowApi, which takes loose string inputs, never throws and always returns the same output names. On top of it, the app ships:
- a Virtual Agent topic that answers with links to the cited sources and keeps a short conversation history (works great as the fallback topic);
- Flow Designer actions and subflows (Semantic Search, Answer Question, Generate Embedding) for things like suggesting articles on new incidents;
- a Scripted REST API (/rag/search, /rag/answer) for portals and external bots;
- a Retrieval Log row per call, with selected scopes and their scores, results, answer, model and token usage. Tuning thresholds without it would be guesswork.
Providers and secrets
Embeddings and chat each have their own endpoint, model, auth scheme and key properties, so you can use OpenAI, Azure OpenAI, a gateway, or two different providers. One OpenAI key serves both. API keys are password2 properties: encrypted at rest, decrypted only in server code, never logged. The repository ships them empty, and a secret check runs before every deploy.
There is also an offline mock provider (feature hashing for embeddings, "best passage" as the answer), so you can install it and see the whole pipeline before you add a key.
Try it on your PDI
- Import the update set from the GitHub release (or npm run deploy with the ServiceNow SDK).
- Run the fix script RAG Kit - Seed and index.
- Set your provider and key in RAG Kit > Properties and run the fix script again.
- Type ask a question in the portal chat.
The README and the docs folder cover the architecture, every property, provider recipes (OpenAI, Azure), tuning, use cases (agent assist, catalog finder, multilingual knowledge bases, portal deflection) and troubleshooting.
What I learned
- Routing beats brute force. Comparing the question with a few scope summaries first is cheap and makes the similarity step both faster and more accurate.
- Retrieval quality is a data problem. Good article titles and scope descriptions matter more than prompt tweaks.
- Log everything. The Retrieval Log made threshold tuning a five-minute job.
- Fluent is a great way to share apps. The whole app is code in Git (tables, Script Includes, ACLs, flows, the topic), built and deployed with the ServiceNow SDK.
Limits
It is a reference implementation, tested on a PDI with a few hundred articles and catalog items. Similarity runs in JavaScript, so it is meant for thousands to tens of thousands of chunks with scopes doing their job, not millions. Content you index is sent to your model provider, so check that against your data policies. If you have Now Assist, that remains the supported path; RAG Kit is for learning, prototyping, and teams that want full control over the model and the retrieval rules.
Feedback welcome
I would love to hear how it behaves on your data, what you would add (hybrid keyword + vector search? ATF tests? question rewriting for follow-ups?), and where it breaks.
Feedback and issues are welcome on GitHub: https://github.com/andreiguimaraess/rag-kit
Personal open-source project, not affiliated with or endorsed by ServiceNow.
