Engineering notes from the field: end-to-end voice latency, session memory and tool calling, voice and IoT integration, and the trade-offs between on-device and cloud deployment.
On a 256GB unified-memory M3 Ultra running a resident omlx inference service, TTFT, decode speed, long-context prefill, concurrency, and memory usage for two models — all measured numbers.
Users who say it feels fast are really reacting to the slowest of six chained stages: wake word, segmentation, recognition, generation, synthesis, and playback.
Cloud-only, device-cloud collaboration, and on-device-only: three paths, each with its own costs across latency, cost, privacy, and capability.
Pure dialogue models can not remember, compute accurately, or get things done. A three-layer structure: short-term context, long-term memory, and tool calling — miss any one and it does not work well.
"I said it, but nothing happened" is the most common problem in voice integration. Sending the command is only the beginning; status feedback and reconnection decide the experience.