Speculative Response Generation for Live Cross-Language Dialogue: Hiding Language-Model Latency Inside the Interlocutor's Turn

Shijie Qu

KEOI Research Institute

KEOI Working Paper · October 2026

Abstract

Language models can formulate conversational responses, but their inference latency — typically measured in seconds — is incurred at exactly the wrong moment: after an interlocutor finishes speaking, when the socially acceptable response gap is shortest. This paper describes a mechanism that relocates that latency. While the interlocutor is still speaking, the system consumes streaming partial speech-recognition hypotheses — incomplete, incrementally updated transcriptions carrying word-level timestamps and confidence scores — together with the dialogue history and a persistent user profile, and generates a small set of candidate responses in the interlocutor's language in parallel with the ongoing speech. Each significant partial-hypothesis update re-triggers generation; candidates are refined, stale candidates are replaced, and the set is re-ranked, so that it converges as the turn's semantics accumulate and a final ranked set is ready before the listener's turn begins. A scoring function ranks candidates by contextual relevance, user-profile match, and predicted conversational utility, and a confidence gate suppresses presentation until the top candidate's combined score exceeds a configurable threshold, leaving the display in subtitle-only mode rather than distracting the listener with low-confidence guesses. The paper sets out the pipeline, the convergence behaviour that distinguishes the mechanism from one-shot reply suggestion, the ranking and gating design, and the disclosed alternative embodiments, including on-device generation with a distilled model and automatic identification of the target language. The mechanism is described as disclosed in U.S. Provisional Patent Application No. 64/170,247; no performance measurements are reported, and numerical values cited are the examples and ranges stated in that disclosure.

Keywords: anticipatory generation; streaming speech recognition; partial hypotheses; cross-language dialogue; latency concealment; response suggestion; human–computer interaction

1. Introduction

Every system that offers a speaker something to say faces the same scheduling problem. The offer is useful only if it exists before the speaker's turn begins; yet the information that would make the offer good — what the interlocutor is saying — is not complete until that same moment arrives. Conventional reply suggestion resolves the tension by waiting: "smart reply" features in electronic mail operate on complete, static text, where no deadline binds, and produce suggestions at leisure. Live spoken dialogue permits no such resolution. Between the end of one turn and the beginning of the next, natural conversation allows gaps that are typically a fraction of a second to a few seconds (Sacks, Schegloff, & Jefferson, 1974), and listeners who intend to respond fluently begin planning while the current speaker is still talking (Levelt, 1989). A large language model (LLM) whose generation takes seconds cannot start at turn end and finish within the gap. If response assistance for live dialogue is to exist at all, its computation must happen somewhere else in time.

Real-time speech translation, the neighbouring technology, does not face this problem because it does not attempt this task. Streaming transcription and translation are receptive: they render the interlocutor's speech intelligible to the listener, incrementally, as it arrives. They say nothing about what the listener might say back. The productive side of the exchange — formulating a reply in the interlocutor's language — is left entirely to the listener, under the tightest deadline in the interaction.

This paper describes a mechanism, disclosed in U.S. Provisional Patent Application No. 64/170,247 (filed October 6, 2026), that addresses the scheduling problem directly: it hides LLM inference latency inside the interlocutor's own turn. The core idea is to generate speculatively, on incomplete input, and to keep regenerating as the input completes itself. The system consumes streaming partial automatic speech recognition (ASR) hypotheses — transcriptions that are emitted incrementally, revised as more speech arrives, and annotated with word-level timestamps and confidence scores. From the latest partial hypothesis, the dialogue history of the session, and a persistent profile of the listener, it generates a plurality of candidate responses in the interlocutor's language, in parallel with the ongoing speech. Generation is re-triggered on each significant partial-hypothesis update, and the candidate set is refined and re-ranked, converging toward a final ranked set that is ready for presentation before the listener's turn begins. A confidence gate withholds the candidates from the display until the top candidate's score justifies showing them.

The mechanism is the processing core of a broader wearable conversation-assistance system disclosed in a companion filing (U.S. Provisional Patent Application No. 64/170,246), which embeds it in smart glasses with a near-eye display; the present paper isolates the generation mechanism itself, which is defined independently of any particular recognition, translation, or language-model component. The contribution is threefold: a formulation of response assistance as a latency-placement problem rather than a generation-quality problem (Section 2); a pipeline that generates candidate responses from partial hypotheses, dialogue history, and a user profile (Section 3); and an account of the convergence dynamics (Section 4), ranking and gating design (Section 5), and context and profile model (Section 6) that make speculative generation converge to something presentable at turn end. Sections 7–9 cover alternative embodiments, limitations, and conclusions. Throughout, the paper confines itself to the mechanism as disclosed in the patent filing; it reports no experiments and no measurements.

2. The Latency-Placement Problem

It is useful to state the timing structure explicitly. Let an interlocutor's turn have duration T. An assistance system that begins generation only when the turn ends must fit the whole of its latency — recognition finalisation, context assembly, model inference, ranking, rendering — inside the response gap G that follows, where G is short and socially enforced: a listener who leaves a long silence after a question has, interactionally, already answered badly. For contemporary LLMs, generation latency L for a non-trivial response is typically measured in seconds, so L > G in the general case, and post-turn generation fails by construction, no matter how good the model is.

The alternative is to notice that T — the interlocutor's speaking time — is a computation budget that existing systems leave entirely unused for the productive task. During T, a receptive pipeline is already running: streaming ASR emits partial hypotheses continuously, each a provisional transcription of the turn so far. A partial hypothesis at time t < T is an imperfect predictor of the completed turn, but it is far from uninformative: dialogue is locally coherent, turns project their own trajectories, and the space of plausible replies narrows rapidly as a turn accumulates. A generation process started at t risks working from a context the completed turn will revise; a generation process that restarts as the hypothesis updates converts that risk into a refinement schedule. The expected cost of an early, superseded generation is wasted computation; the expected benefit is that, by turn end, at least one full generation cycle has completed against near-final context. Because the generations overlap speech rather than the response gap, the mechanism adds no latency of its own to the exchange: generation time is fully overlapped with speech time.

This reframing has a consequence for how such a system should be evaluated conceptually. The primary design variable is not the quality of a single generation but the convergence behaviour of a sequence of generations: how quickly the candidate set stabilises as partial hypotheses lengthen, how gracefully it absorbs revisions when the turn swerves, and whether anything presentable survives the confidence gate at turn end. Sections 4 and 5 take up these questions in the mechanism's own terms.

The analogy to anticipation in simultaneous translation is instructive but partial. Simultaneous translation systems, including prefix-to-prefix frameworks that anticipate source-language content in order to emit target-language output before the source sentence completes (Ma et al., 2019), speculate about what the speaker will say next. The mechanism described here speculates one level further out: about what the listener will want to say in reply. The input is correspondingly richer — not the partial utterance alone, but the partial utterance situated in a dialogue history and conditioned on who the listener is — and the output is not a single committed rendering but a small ranked set of options, from which the listener remains free to choose or to depart entirely.

3. The Anticipatory Generation Pipeline

The disclosed pipeline (FIG. 1 of the filing) comprises five stages, numbered 410–450.

Streaming partial ASR (410). The front end emits incrementally updated transcription hypotheses of the interlocutor's ongoing speech, with word-level timestamps and confidence scores, without waiting for end-of-utterance detection. This is the property the rest of the mechanism depends on: the input is a living object that grows and is revised word by word, not a completed string delivered at turn end. Word-level timestamps anchor each element of the hypothesis in the turn's timeline, and word-level confidence scores mark which parts of the current hypothesis are firm and which are provisional — information a downstream generator can weigh when deciding how much of the reply space a partial turn already constrains.

Context assembler (420). The assembler combines the latest partial hypothesis with two further context sources. The first is the dialogue history of the current session: the prior turns of both parties, which supply the exchange's topic, its open questions, and the commitments each party has already made. The second is a persistent user profile of the listener (the wearer, in the wearable embodiment), storing the listener's identity, role, business domain, preferred formality, and frequently used expressions; the profile is built over time with the user's consent and stored encrypted. The assembler's product is the working context for generation: what is being said now, in the light of what has been said, addressed to a model of who will have to reply.

LLM generator (430). Prompted with the assembled context, the generator produces a plurality of candidate responses — two to four in the disclosed embodiment — in the interlocutor's language: that is, responses the listener is likely to want to give, already formulated in the language the reply must be spoken in. Generation is triggered not only at turn end but on each significant partial-hypothesis update — the disclosure gives, as examples, re-triggering every one to two seconds, or upon semantic-boundary detection in the accumulating speech. Each trigger assembles the context afresh from the newest partial hypothesis, so a generation cycle always works from the best context available at its start.

Ranker (440). The ranker scores the candidates produced by the generator — on the dimensions described in Section 5 — and orders the set.

Display (450). The top-ranked set is presented to the listener, on the wearable's near-eye display in the system embodiment, subject to the confidence gate described in Section 5. Presentation is timed to the turn structure the whole mechanism serves: the ranked set is ready before the listener's turn begins, so that at the moment the interlocutor stops speaking, the listener's attention can go to choosing and speaking rather than to composing.

4. Convergence: Refinement Across Partial Hypotheses

The mechanism's defining behaviour is not any single generation but the trajectory of the candidate set across a turn (FIG. 2 of the filing). Early in a turn, the partial hypothesis is a fragment. The disclosure's own illustration begins with the fragment "…meeting on": at this point (t1) the candidate set is rough and generic — the context constrains the reply space only weakly, and the generator's output reflects that. As more speech arrives (t2: "…meeting on Tuesday"), the context has materially changed: candidates are regenerated against the fuller context and re-ranked, and stale candidates — those fitted to a context the turn has since moved away from — are replaced. By turn end (t3), the set has converged: a final ranked set, generated against essentially complete context, stands ready for presentation.

Three properties of this schedule deserve emphasis. First, replacement is a normal event, not an error path. A candidate generated at t1 is a wager on the turn's trajectory; when the trajectory shifts, the wager is discarded and a new one placed. The mechanism therefore does not need the early hypotheses to be right — it needs them to be useful starting points whose successors improve monotonically in expectation as evidence accumulates. Second, convergence is driven by the input, not by a fixed schedule alone. Re-triggering is tied to significant partial-hypothesis updates — the disclosure's examples are a cadence of every one to two seconds or detection of a semantic boundary — so refinement effort is spent when the context has actually changed, and the final generation in a turn is naturally the one launched against the longest, most stable partial hypothesis. Third, the schedule is what makes the latency argument of Section 2 concrete: because generation cycles run throughout the turn, the cycle that matters — the last one — has the whole of the turn's remaining duration in which to complete, and the set it produces is already ranked and, subject to the gate, presented as the turn closes. The system trades early imprecision for zero added latency, since generation time is fully overlapped with speech time.

It is worth distinguishing convergence of the candidate set from stability of any individual candidate. Early candidates may be wholly replaced rather than edited; what converges is the set's fit to the turn as finally delivered. This set-level view also clarifies the role of candidate plurality: maintaining two to four candidates through the refinement process hedges the speculation, because a turn that defeats the leading interpretation of its prefix may still be well served by the second or third.

5. Ranking and Confidence Gating

A speculative system that displayed every intermediate guess would be unusable: the listener would watch half-formed suggestions flicker through the turn, and the distraction would consume the attention the mechanism exists to save. The disclosure addresses presentation discipline in two parts — ranking, which orders what may be shown, and gating, which decides whether anything is shown at all.

Ranking. Candidates are scored by a function considering three dimensions. Contextual relevance measures the candidate's fit to the assembled context — the current partial hypothesis situated in the dialogue history. User-profile match measures the fit to the listener: the profile's identity, role, and business domain, the listener's preferred formality level, and the expressions the listener frequently uses — so that a candidate reads as something this listener, in this capacity, would plausibly say, at the register the occasion calls for. Predicted conversational utility estimates the candidate's forward value to the exchange — whether adopting it would move the dialogue somewhere the listener is likely to want it to go. The paper states these dimensions functionally, as the disclosure does; it does not fix a functional form or weights for their combination, which the disclosure leaves open.

Confidence gating. In the preferred embodiment, candidates are displayed only when the top candidate's combined score exceeds a configurable threshold; otherwise the display remains in subtitle-only mode. The gate converts the ranker's ordering into a presentation decision with a deliberately conservative default: ambiguous speech — where partial hypotheses are unstable and candidates diverge — produces silence from the suggestion layer rather than a parade of guesses, while a turn whose trajectory has become clear releases its converged set. The threshold is configurable rather than fixed, because the right trade-off between eagerness and restraint depends on the deployment: a listener in a high-stakes negotiation may want only near-certain suggestions, while a listener in casual conversation may welcome speculative ones. What the mechanism guarantees is the form of the control — presentation is earned by the top candidate clearing a bar — not any particular setting of the bar.

Ranking and gating together define the listener's experience of the system across a turn: nothing during the uncertain early phase, then, as the partial hypothesis firms up and the candidate set converges, a single ranked appearance of the set at or before turn end — provided it clears the gate. If it never does, the turn proceeds unaided on the productive side, exactly as it would have without the mechanism, with the receptive subtitles unaffected.

6. Context Assembly and the User Profile

The generator's inputs determine the ceiling on candidate quality, and the disclosure's context design is worth stating as a design position in its own right. A candidate reply is a function of three things: what has just been said, what has been said before, and who must now speak. The context assembler (420) is where these are joined: the latest partial hypothesis supplies the immediate provocation; the session's dialogue history — prior turns of both parties — supplies the commitments, terminology, and open threads that a coherent reply must respect; and the persistent user profile supplies the speaker model.

The profile's disclosed contents are deliberately mundane and durable: identity, role, business domain, preferred formality, and frequently used expressions. These are the parameters along which real speakers vary and along which generic suggestions fail: the same interlocutor question — a price, a deadline, a request — calls for different replies from a buyer's representative and from an engineer, in different registers, with different stock phrases. Building the profile over time, with the user's consent, and storing it encrypted, treats the profile as what it is: a sensitive, accumulating model of a person's professional voice, assembled to be used at the moment of speech and protected accordingly.

The profile also does quiet work in the ranking stage, as Section 5 described: profile match is one of the three scoring dimensions, so the profile influences not only what the generator proposes but which proposals survive to presentation. And because the profile persists across sessions while the dialogue history does not, the mechanism separates the durable speaker model from the ephemeral conversational one — a separation the alternative embodiments exploit, as the next section describes.

7. Alternative Embodiments

The disclosure records three variations that mark the mechanism's intended range.

On-device generation. The generator may run on-device using a distilled model when offline, with a reduced candidate count. The trade is explicit: connectivity loss costs candidate breadth and, presumably, generation headroom, but the latency structure — the property the mechanism exists to secure — is preserved, because generation still overlaps the interlocutor's turn.

Automatic target-language identification. The language of the candidates is the interlocutor's language; the disclosure provides that the target language may be selected automatically via language identification on the interlocutor's speech, rather than fixed in advance. In multilingual settings where successive interlocutors — or successive turns — change language, the candidate set follows the speech it answers.

Session-scoped profile. For privacy-sensitive deployments, the user profile may be session-scoped, with no persistence: the speaker model is assembled within the session and does not outlive it. This trades the accumulated fit of a persistent profile for a deployment in which nothing about the listener is retained — the same mechanism, with the profile's lifetime as a policy variable.

8. Discussion and Limitations

The mechanism's wager is that a turn's prefix, a dialogue history, and a speaker model jointly constrain the reply space enough, early enough, for speculative generation to converge to something worth showing. Three limitations bound that wager, and this paper states them plainly.

First, prediction can simply fail. Some turns are constructed to surprise — a question that pivots in its final clause, an answer that reverses its opening. For such turns the converged set may be confidently irrelevant, the gate may still release it, and the listener discards it and composes unaided. The design's defence is asymmetry of cost: a discarded candidate set costs attention only if it was displayed, which the gate exists to minimise, and the fallback state — no usable suggestion — is the status quo ante, not a degraded one. But the frequency of such turns in real dialogue is an empirical matter on which the disclosure, and therefore this paper, offers no data.

Second, the mechanism inherits the errors of its inputs. Partial ASR hypotheses early in a turn are unstable; a misrecognised word can steer a generation cycle into a context the speaker never created, and the refinement schedule must then steer it back as the hypothesis corrects itself. Convergence is the designed answer, but convergence speed relative to hypothesis-correction speed is again an empirical question the paper does not resolve.

Third, nothing here is measured. This paper describes the mechanism as disclosed in the patent filing. It reports no user studies, no recognition or generation accuracy, no latency measurements, and no deployment results. The numerical values that appear — candidate sets of two to four; re-triggering exemplified at every one to two seconds or at semantic boundaries; a configurable gating threshold whose value is not specified — are the ranges and examples stated in the disclosure, not experimental findings. The contribution claimed is the mechanism's design: the placement of generation inside the interlocutor's turn, its refinement across partial hypotheses, and the ranking and gating discipline that keeps speculation presentable.

A further question the paper flags without resolving is interactional rather than technical. A listener who is visibly ready with a fluent reply at every turn end is, from the interlocutor's side, simply a fluent conversationalist; the assistance is designed to be invisible. Whether and how such assistance should be disclosed in professional settings is a norm that deployment contexts will have to settle, and it is not settled by the mechanism's design.

9. Conclusion

Smart reply waits for complete text; live dialogue does not wait. The mechanism described here moves response generation to the only place it can fit — inside the interlocutor's turn — by generating candidate replies speculatively from streaming partial speech-recognition hypotheses, refining and re-ranking them as each new partial arrives, and releasing the converged set, through a confidence gate, before the listener's turn begins. Ranking by contextual relevance, user-profile match, and predicted conversational utility makes the candidates the listener's own in voice and register; gating keeps the speculation silent until it has earned display. What is claimed is a scheduling insight made mechanical: in conversation, latency is not a property of the model alone but of where in the turn structure the model is allowed to run — and the interlocutor's turn, hitherto dead time for the productive half of dialogue, is long enough to hide it.

References

Levelt, W. J. M. (1989). Speaking: From intention to articulation. MIT Press.

Ma, M., Huang, L., Xiong, H., Zheng, R., Liu, K., Zheng, B., Zhang, C., He, Z., Liu, H., Li, X., Wu, H., & Wang, H. (2019). STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3025–3036.

Qu, S. (2026). Anticipatory Multilingual Conversation Assistance System for Wearable Devices. U.S. Provisional Patent Application No. 64/170,246, filed October 6, 2026.

Qu, S. (2026). Real-Time Anticipatory Response Generation for Cross-Language Dialogue. U.S. Provisional Patent Application No. 64/170,247, filed October 6, 2026.

Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 50(4), 696–735.

AI Disclosure: This working paper was prepared with AI-assisted writing and editing tools. The mechanism description, arguments, and conclusions are the author's own, and the technical content is drawn from the author's patent disclosure cited above.