A multi-tenant platform where organizations put a retrieval-based AI assistant on their site and run it from a dashboard.
In production
The platform is live and serving client organizations.
Multi-tenant
Used by at least three organizations, one of them the company’s own website.
Client
- SolidJS widget
- Next.js dashboard
- TypeScript
Services
- FastAPI
- Celery workers
Data and cache
- PostgreSQL with pgvector
- Redis
- Object storage
AI
- LLM provider
- Speech to text
- LLM tracing
Messaging
- Push (Firebase)
- Transactional email
Step through a request one hop at a time, or switch to Trace and break to see which flows a service takes part in.
The whole system. Pick a scenario, then step through it. The view zooms to the services each step involves.
- Request
- AI pipeline
- Alerts
- Auth
Signed tokens instead of a session lookup
Chose
Custom JWT auth. The backend issues each visitor a guest token and verifies the token itself on every request.
Rejected
Looking up a session record in the database on every request.
Why
A visitor to the widget has no account, so a guest token fits. A token can be checked without a database round trip, so every message skips an extra query.
- Retrieval
Check a semantic cache before retrieval
Chose
The semantic cache is checked first. On a miss the question goes through vector retrieval and the LLM, and the answer is saved to the cache in the background.
Rejected
Sending every question through retrieval and the LLM.
Why
Similar questions can be answered from the cache, which avoids repeat LLM calls and keeps those answers fast.
- Answers
Answers that cite their sources
Chose
The LLM answers from the retrieved chunks and returns the sources it used. The sources are stored with each answer.
Rejected
Free-form answers with no sources attached.
Why
Visitors can see where an answer came from, and the dashboard shows the sources next to each chat.