Vidzy RAG (Ask Vidzy)

Overview

Ask Vidzy answers questions about VidzyPlayer (False Logics) from a reviewed set of canonical questions and answers. By default it returns the closest reviewed answer with its source and related questions, and it adds an LLM step after retrieval only when that mode is switched on.

It is a standalone project with its own knowledge base and does not depend on the VidzyPlayer application code. I wrote all of it.

Context and problem

The design suggests the aim was consistent, reviewed answers to product questions, without exposing internal knowledge or paying for an LLM call on every query.

It likely serves two audiences: prospects and users of VidzyPlayer asking product questions in the public chat, and internal staff searching private notes with an API key.

My role

I built it alone: the retrieval design, the API, the chat UI, the knowledge base, the evaluation harness, CI and the deployment examples. Every commit in the repository is mine.

The history covers five days in late September 2026 and includes a tagged v0.1.0 public release; there has been no activity since.

Constraints and requirements

• The default path works without external credentials: embeddings run locally and no paid API or LLM key is needed.
• Internal knowledge must never reach the public chat, so public and internal content sit in physically separate collections instead of being filtered at query time.
• The project stays independent of the VidzyPlayer application code.

Research and planning

Before any vector storage, I wrote down a knowledge-base lifecycle: source material is extracted into facts, verified, classified as public or private, and only then written up as canonical docs and Q&A. Facts an AI coding agent derives from a codebase count as candidates to check, not as truth.

A validation script checks the knowledge base before it is ingested.

Solution and user experience

A visitor asks a question. The system embeds it locally, finds the closest reviewed canonical questions in the public collection, and the confidence policy picks one of three outcomes: a direct answer with its source and related questions, a list of related results, or a plain statement that there is no reliable answer.

Generative mode is opt-in. It reuses the same retrieval step and passes the retrieved public documents to an LLM through a LangChain prompt; if the provider, package or key is missing, it can fall back to the canonical answer.

Key features

• Public chat: reviewed canonical answers with sources and related questions.
• Confidence policy: a direct answer, related results or no reliable answer, decided by score thresholds, an ambiguity margin and a lexical-overlap safeguard.
• Scope isolation: the public chat route is hard-wired to the public collection; internal and combined search need an API key header.
• Optional generative mode: an LLM composer after retrieval, with a fallback to the canonical answer.
• Ingestion: heading-aware Markdown and JSONL loading with deterministic SHA-256 IDs and full-snapshot reindexing.
• Evaluation: an offline, labelled retrieval benchmark of 26 cases that needs no paid API.
• Knowledge lifecycle: facts, reviewed docs and canonical Q&A, checked by a validation script.

Architecture and stack

A question goes through a local embedding, then Chroma, then the score and confidence policy, and comes out as a canonical answer, related results or no answer; the optional LLM composer sits after retrieval.

• API: Python 3.12 and FastAPI, which also serves the compiled React app.
• Retrieval: LangChain with a custom BaseRetriever that keeps Chroma's relevance scores.
• Vector store: persistent Chroma with separate public and internal collections.
• Embeddings: local sentence-transformers (all-MiniLM-L6-v2), one model instance per process.
• UI: React, TypeScript and Vite.
• Generation: an optional OpenAI adapter through langchain-openai, off by default.
• Tests and CI: a pytest suite over the API, config, ingestion, retrieval and static serving, with GitHub Actions running the checks.
• Deployment: nginx and pm2 examples, with a pm2 process configured on the Vidzy droplet.

Implementation details

Scoring looks at the best canonical-question match, its gap to the runner-up and whether the question shares meaningful words with it. The lexical check stops a semantic neighbour that only shares generic product wording from being returned as an answer.

The thresholds and the ambiguity margin are configuration, not constants hidden in the response service.

Ingestion splits Markdown by heading and derives each document's ID from a SHA-256 hash of its source; reindexing replaces the whole collection with the current snapshot, so duplicates and stale records cannot build up.

Input is validated strictly: unknown fields, invalid scopes, long prompts and bad limits are rejected, the internal key is compared in constant time, interactive API docs are switched off under the production setting, CORS is allowlist-driven, and logs record error types rather than request bodies or secrets.

The benchmark mixes exact questions, independent paraphrases, terminology changes, ambiguous, unrelated and unsupported prompts, and no LLM takes part in scoring.

Challenges

• Avoiding confident wrong answers: the policy has to decline when the match is weak or two candidates are too close, which is why it combines thresholds, an ambiguity margin and the lexical safeguard.
• The benchmark's one top-1 miss is a terminology question that calls a popup an "offer message". The right popup results stay in the top three, but the policy declines rather than return the wrong top answer, and more reviewed popup phrasing is the documented next step.
• CI needed a valid cache context for the embedding model, fixed in a follow-up commit.

Trade-offs

I left that missed question to decline instead of lowering the threshold, accepting an unanswered question to avoid more false positives.

Serving reviewed canonical answers is predictable and costs nothing per query, but it likely covers only questions the knowledge base already answers; the optional LLM step trades some of that predictability for flexibility.

Screenshots

Links

• Source Source code github.com
• Demo Live demo ask.vidzyplayer.com

Skills demonstrated

• Retrieval-augmented generation design: a retrieval-first flow with a confidence policy that can decline.
• Vector search with Chroma and local sentence-transformers embeddings.
• Python API development with FastAPI and LangChain.
• Retrieval evaluation with a labelled offline benchmark.
• Access scoping and input hardening for internal knowledge.
• Testing and CI with pytest and GitHub Actions.