Multilingual Voice AI Agent

Reception and front-desk teams field the same calls all day (availability, bookings, changes, simple questions), often after hours and across languages. Callers wait on hold, and routine requests eat the team's time.

Answer the phone around the clock with a natural-sounding agent that handles bookings and guest questions on its own, works across languages, recognizes repeat callers, and leaves the team a clean, structured summary, while escalating anything it shouldn't handle.

A streaming voice pipeline over real telephony. Inbound calls are transcribed as the caller speaks, grounded against a knowledge base, answered by an LLM, and spoken back with low latency: the first sentence of each reply is voiced while the rest is still generating, so the caller hears a response almost immediately. Every caller is recognized by phone number, with older turns rolled into an LLM-written memory summary so the agent recalls prior stays and requests. When the call ends, a tool extracts a structured booking record and emails a confirmation.

1. 1Caller dials in over the phone (Telnyx); audio streams to the agent in real time
2. 2Whisper transcribes each turn; the agent recognizes the caller from their number
3. 3Relevant facts are retrieved from the knowledge base (RAG) and added to context
4. 4An LLM generates the reply, streamed back as speech sentence-by-sentence
5. 5On hang-up, a tool extracts a structured record and emails a booking confirmation

• Real-time STT with Whisper (local faster-whisper or Groq whisper-large-v3-turbo)
• LLM dialogue via Groq (openai/gpt-oss-120b), streamed, provider-agnostic (Fireworks / Ollama)
• Streaming TTS with sentence-level output for low time-to-first-audio
• RAG over a knowledge base (Qdrant + BGE-small embeddings)
• Per-caller memory with rolling LLM summarization, keyed to phone number
• Post-call structured extraction (JSON mode) + email tool

• Answers real phone calls 24/7 with natural, low-latency conversation
• Multilingual: English, French and German, each with its own voice and prompts
• Remembers callers across calls (bookings, preferences, open issues)
• Answers from a live, re-indexable knowledge base
• Emails a booking confirmation automatically (more tools on the way)
• Deploy on-premise or in the cloud: run local speech-to-text and a self-hosted LLM to keep data on your network, or use cloud models for the lowest latency
• Live dashboard: calls, transcripts, caller memory and knowledge base

A FastAPI backend bridges the telephony provider (Telnyx Call Control + media streaming) to the AI pipeline over WebSockets. Inbound audio (mu-law, 8 kHz) is buffered with energy-based turn detection, transcribed, grounded via a Qdrant knowledge base, and answered by a streaming LLM; replies are spoken back through Telnyx TTS. Caller identity, transcripts and memory persist in SQLite. STT and LLM providers are swappable by config: local for privacy/latency or cloud for scale.

A Nuxt 3 monitoring dashboard shows live and past calls grouped by caller, full turn-by-turn transcripts, each caller's memory summary, and a knowledge-base manager to upload files and re-index, with a live connection status and call counter.

• Telnyx Call Control v2 (telephony + media streaming)
• Resend (transactional email for booking confirmations)
• Qdrant vector database; knowledge-base uploads (PDF, DOCX, JSON, MD)
• Groq / Fireworks / Ollama for LLM and speech

Containerized backend and dashboard (Docker Compose). The AI stack runs fully on-premise: local Whisper for speech-to-text, a self-hosted LLM (Ollama / vLLM) and a self-hosted vector store, so transcripts, knowledge and caller data never leave your infrastructure. Pair it with an on-site SIP gateway for a fully self-hosted deployment, or use cloud telephony (Telnyx) and cloud models (Groq) for the lowest latency. Run on a GPU box for faster local speech, or entirely on CPU.