Web app · metrics · CI thresholds
Bot quality evaluation
“It seems better” is not an argument. With every change, the support bot goes through 40 control questions, and a release won’t reach production if the metrics fall below a threshold. The app repeats this check in the browser: compare keyword search, the neural network and hybrid, tweak the settings and see which questions got better and which broke.
Scenario: The Kvadrat Coworking support bot: its knowledge base, its 40 CI questions and 20 paraphrased questions.
Open the demo
The “As in production” configuration is evaluated right away. For the neural network and hybrid, the model loads on a button press — about 126 MB once.
The app runs in your browser: nothing is sent to a server.
What the demo shows
How to try it
Open the app
The “Custom” configuration is open with a lowered “I don’t know” threshold: the bot starts answering off-topic questions, and CI would block such a release. Press “As in production” — the thresholds pass.
Connect the neural network
“Load model”, then the “Semantic model” or “Hybrid” card. The set reruns, and below you see which questions got better or worse than in production.
Switch the set
On paraphrased questions keyword search sags, while hybrid finds more — but errs on off-topic questions. That’s how a configuration is chosen: by numbers, not by feel.
Add your own questions
A question and the document containing the answer — and the metrics are recalculated. The report can be downloaded in Markdown for a task or review.
No paid neural networks
The metrics here are real: the same search and the same questions as in the demo bot’s CI, and the neural network runs in the browser. No language model is involved — what’s evaluated is the search the answer depends on. In a production project these metrics are complemented by checking the model’s answers themselves: whether it answers from the fragments and doesn’t make things up.
The demo runs without OpenAI, YandexGPT or GigaChat API keys. Wherever a production system calls a model, there’s a local stub with the same interface — so the demo is free, always available and never makes things up.
What a production project adds
- Control questions from real conversations, not invented ones — labeled with the correct document
- A CI run for every change to the base, prompt or model — with thresholds that block the release
- Evaluating model answers: grounding in sources, refusing off-topic questions, resistance to prompt injections
- Metric history by release — you can see exactly what improved or broke the bot
Stack
- All demos by stack:
- React
- TypeScript
- In-browser neural network
- React 19
- TypeScript
- Transformers.js
- Web Workers
- Comlink
- Zustand
- Vite
Based on
Books and university courses whose ideas are at work in this demo.
Book · Robert C. Martin
The Clean Coder
Thresholds like in CI: if the metrics drop, a banner says CI wouldn’t let this release through — I don’t ship what I’m not sure about.
Book · John Berryman, Albert Ziegler
Prompt Engineering for LLMs
Every change to the bot’s settings runs on 40 test questions: hit@1, hit@3 and the share of correct refusals show whether it got better or worse.
Developer’s comment
Anton Goncharov
AI systems developer · Python
For business
- Every bot change is checked before release: if quality drops, the release stops, and customers never see it.
- The configuration is chosen by numbers on control questions, not by impression.
- The report can be attached to a task — it’s immediately clear what improved and what broke.
For developers
- The metrics are hit@1, hit@3 and the share of correct refusals on 40 questions from the bot’s CI, 6 of them off-topic, plus 20 paraphrased questions.
- The run happens in a Web Worker via Comlink, so the interface doesn’t freeze even with the semantic model.
- The comparison with production is per question, “before → after”: regressions are visible one by one, not just on average.
Tips
Tip 01
Include off-topic questions in the set: hit@k doesn’t see them, and that’s exactly where a bot makes things up.
Tip 02
A CI threshold should block the release, not print a warning: people stop reading warnings within a week.
Want the same for your business?
Tell me about your task — I’ll tell you what from the demo fits, how long it will take and what I’ll need from you.
Other demos in this section
All demos- live
Web app · agent diagram
AI agent builder
A support agent is assembled from blocks: intents, a knowledge base, API calls, checks and operator handover. Send a customer message and watch its path light up on the diagram step by step.
- React
- TypeScript
- live
Web app · question → SQL → chart · PostgreSQL in the browser
AI data analyst
A manager asks in plain words — “revenue by channel for Q3”, “which customers stopped ordering” — and gets SQL, a chart, a table and a conclusion. The sales database is real PostgreSQL running right in the browser.
- React
- TypeScript
- live
Web app · risks against a standard · Word with tracked changes
Contract review
A counterparty sent their own contract — the review compares it with the company’s standard, highlights risky clauses, explains why they’re dangerous and suggests wording. The output is a Word file with tracked changes and a disagreement protocol.
- React
- TypeScript
- live
Web app · tasks from chat · kanban · evening digest
AI agent in a team chat
People write in the team chat as usual — “Pete, fix the price list by Friday”. The agent creates a task on the kanban board, moves it through “On it” and “Done”, reschedules and reassigns tasks, reminds about overdue work in the morning and sends a digest in the evening.
- React
- TypeScript
- live
Web app · shared inbox · CRM · protection against bank detail fraud
AI inbox triage
The agent reads the sales team’s shared inbox: it understands what each email is, extracts fields, creates a deal, a ticket or a task in the CRM and drafts a reply. An invoice with new bank details or a phishing email is caught before anyone pays or clicks a link.
- React
- TypeScript
- live
Web app · marketplaces · moderation check · unit economics
Product listings for WB and Ozon
The agent turns a supplier’s text into two listings — for Wildberries and for Ozon: a title within the marketplace limit, a description with search keywords, attributes from the reference lists. Then it checks what listings usually fail moderation on and calculates what’s left from each sale.
- React
- TypeScript
- live
Web app · meeting transcript · minutes · kanban
Meeting minutes → kanban tasks
The agent reads a stand-up transcript and builds the minutes: decisions, tasks with an assignee and deadline taken from live speech, risks and open questions. Tasks go to the kanban board, and at the next meeting the agent moves them through statuses itself.
- React
- TypeScript
- live
Web app · personal data · healthcare · analytics
Anonymizing data before it reaches a neural network
The agent finds full names in any grammatical case, phones, passports, SNILS numbers, cards, addresses and diagnoses in emails and transcripts, replaces them before sending to an external neural network, and restores its answer to the real data on your side. It anonymizes a patient table for analytics so that nobody can be identified.
- React
- TypeScript
- live
Web app · automation · amoCRM, 1C, Telegram
An n8n-style automation pipeline
Three working scenarios of a furniture factory on a canvas, as in n8n: website leads go to amoCRM and to managers by lead temperature, YooKassa payments go to 1C with retries and an error branch, and every morning there’s a digest for the director. Trigger an event and watch records flow from node to node; the workflow can be edited and exported to real n8n.
- React
- TypeScript
- live
Web app · multi-agent system · content
A team of AI agents for content
Six agents prepare a publication from a brief: the editor-in-chief plans, the researcher finds facts in the knowledge base, the writer writes, the fact-checker and editor send the draft back for revision, and the SMM agent adapts the text for Telegram, VK, a newsletter, a blog and a carousel. You see the agents’ entire conversation, every edit and what would have been published without checks.
- React
- TypeScript
- live
Web app · HR · scoring and chatbot
CV screening and an interviewer bot
Twenty CVs for three vacancies are analyzed in seconds: employment periods, experience, salary and city, match with the requirements and a quote from the CV for every “yes” or “no”. What’s missing from the CV is clarified by a bot: it asks only the missing questions, follows up on vague answers and books an interview. Gender, age and family are left out of the assessment.
- React
- TypeScript
- live
Web app · RAG · neural network in the browser
Search sandbox
How a support bot finds an answer in a knowledge base: documents are cut into fragments, search runs by keywords, by meaning or as a hybrid, and every fragment found shows why it ranks above the others.
- React
- TypeScript
- In-browser neural network
- live
Web app · AI security · vanilla JS
Prompt injection defense trainer
Try to trick a support bot: coax out a secret promo code, extract the system prompt or get a refund without grounds. Six defense levels, direct attacks and attacks via emails, reviews and the knowledge base, and a check log based on the OWASP Top 10 for LLMs.
- Vanilla JS
- live
Web app · AI economics · vanilla JS
AI bot cost calculator
How much a bot will cost per month on GPT, Claude, YandexGPT, GigaChat or your own model on a rented GPU. Token-based calculation, all models compared on a chart, tips on paying less, and a real o200k tokenizer right in the browser.
- Vanilla JS