Making a Voice Assistant "Remember and Get Things Done": Engineering Practices for Session Memory and Tool Calling

A voice assistant that only chats gets boring within three days. What makes it stick is: it remembers me and can genuinely help me get things done.

The Three Shortcomings of Pure Dialogue Models

• Can not remember: Once the context window fills up, earlier conversation gets pushed out, and the user has to repeat the background.
• Can not compute accurately: Inventory, prices, today's weather — the model can not possibly "remember" such real-time data; it has to look it up.
• Can not get things done: Turning on lights, placing orders, checking tickets — these actions must actually land in a system; generating text can not do it.

The corresponding solution is a three-layer structure: short-term context, long-term memory, and tool calling. Miss one layer and there's an obvious gap in the experience.

Layer One: Short-Term Context

Multi-turn conversation context can not be piled up indefinitely. A workable approach is "keep the last few turns verbatim + summarize and compress earlier content," saving tokens while preserving key information. Timing the compression matters: compressing only after hitting the limit loses the details being discussed.

Layer Two: Long-Term Memory

Remembering across sessions "who the user is, what they prefer, where we left off" is usually done with vector retrieval. But the key is not storage — it's the write decision:

• Which information is worth remembering (preferences, identity, long-term goals) and which is noise (one-off temporary instructions)
• When new information conflicts with an old memory, whether to replace, delete, or keep both
• Retrieve by relevance rather than stuffing all history back into the context

A counterintuitive point: More memory isn't better. Irrelevant memories mixed into the context dilute attention, slow responses, and easily lead the model astray.

Layer Three: Tool Calling

"Getting things done" depends on tool calling: the model decides which interface to call and what parameters to pass, then organizes the result into natural language for the user. Three engineering points must be handled:

• Permission boundaries: Which tools may be called and how wide the parameter range is must be constrained by the server, not trusted to the model alone.
• Timeouts and retries: External interfaces get slow and go down; you need timeout fallbacks and idempotent design to avoid "said it once, executed it twice."
• Result filling: After getting a structured result, convert it into conversational phrasing rather than reading out JSON.

Order Matters

Start with short-term context (the smallest change), then add long-term memory, and connect tools last — because tool calling introduces external uncertainty, and debugging without stable context is extremely painful.

The Xiaozhi voice conversation platform builds these three layers in as native capabilities and supports connecting enterprise systems as skills. If your voice assistant is stuck at the "can chat but can't be used" stage, feel free to contact us.