B1 · update
Most AI voice assistants fail for two reasons: high latency from cascading API hops, and fragile turn-taking that freezes after the first reply.
Over the past few weeks, I’ve been building an open-source, fully local AI agent with a native full-duplex Speech-to-Speech (STS) pipeline. Here is the complete architecture breakdown of what’s running under the hood:
1. Unified Speech-to-Speech (STS) Pipeline
Instead of the standard 3-hop cascade (Client STT → LLM generation → regex sentence chunking → separate TTS API calls), everything runs through a single /api/voice/sts endpoint. Spoken input returns a direct, high-fidelity neural audio stream (Edge-TTS en-US-JennyNeural) in one unified hop.
2. Sub-10ms Conversational Reflex Engine
To hit instant human conversational timing, common greetings, confirmations, and interruptions bypass LLM inference entirely. A local reflex matcher resolves and caches replies in under 10ms with natural human conversational fillers ("Oh hey!", "Right!").
3. Zero-Delay Reasoning on Gemini 3.8 Flash
For novel reasoning queries, the pipeline calls Gemini 3.8 Flash with zero reasoning overhead (think: 4, 85-token cap). It delivers punchy, spoken-first answers with natural disfluencies in under 1.5 seconds.
4. Solving Chromium's WebSpeech Deadlock Bug
In browser voice interfaces, calling recognition.abort() tears down Chromium's WebSpeech engine, throwing InvalidStateError on subsequent turns and leaving the mic frozen. I replaced it with a clean instance lifecycle and an autonomous 1200ms duplex watchdog that keeps the listener armed across continuous back-to-back turns.
5. Local Self-RAG Vector Memory
Continuous conversation history is embedded locally on CPU using all-MiniLM-L6-v2. The agent recalls past project context, code snippets, and preferences across sessions without context window bloat.
6. Autonomous Agent Core & Self-Healing Proxy
Backed by 13 active MCP servers and 39 tools across local filesystem execution and code inspection, supported by a self-healing background proxy daemon on port 8081.
Open source code and architecture:
https://github.com/IshaanYK/gemini-1b-local-agent
#AI #MachineLearning #SpeechAI #VoiceAI #Python #SoftwareEngineering #OpenSource #FullStack #BuildInPublic