Closing the Productive Half of Cross-Language Conversation: An Anticipatory Multilingual Conversation Assistance System for Wearable Devices

Abstract

Cross-language conversation support technology has advanced almost exclusively on the receptive side: translation devices and subtitle glasses help a user understand an interlocutor, but leave the user unaided at the moment of formulating a reply in the interlocutor's language. This paper presents a wearable conversation assistance system that closes this productive half of the loop. Worn by only one party to the conversation, the system (1) captures the interlocutor's speech and projects real-time translated subtitles onto a near-eye display; (2) anticipatorily generates, in parallel with the interlocutor's ongoing turn and drawing on streaming partial speech-recognition hypotheses, dialogue history, and a persistent user profile, a small ranked set of candidate responses in the interlocutor's language, presented with spoken pronunciation guidance delivered by in-ear text-to-speech; and (3) provides a free-expression fallback in which the wearer speaks once in their native language and reads aloud the projected translation. The central mechanism is latency concealment through anticipation: because candidate generation overlaps the interlocutor's turn, large-language-model inference time is absorbed within the natural duration of speech and the turn-taking gap, so candidates are ready before the wearer's turn begins. The paper describes the three-tier architecture (smart glasses, paired mobile device, cloud services), the three-step conversation loop, the functional design of candidate ranking and selection — including silent gesture-based selection that keeps the interaction invisible to the interlocutor — and the user-profile and privacy model. The system is described as disclosed in U.S. Provisional Patent Application No. 64/170,246; performance figures cited are the design targets stated in that disclosure, not production measurements.

Keywords: cross-language communication; wearable computing; smart glasses; anticipatory computing; simultaneous translation; human–computer interaction; conversation assistance

1. Introduction

Conversation is a two-sided act, but assistive language technology is one-sided. A business traveller in a foreign-language meeting faces two distinct problems in alternation: understanding what the interlocutor has just said, and producing a reply in the interlocutor's language within the short window that ordinary turn-taking allows. The first problem — the receptive half — has attracted decades of engineering, from handheld translators to translation earbuds and smartphone applications, and more recently to smart glasses that project streaming translated subtitles onto a near-eye display. The second problem — the productive half — remains largely unassisted. At precisely the moment when the cognitive demands peak, the user is left alone: they must compose, in a language they command imperfectly, under time pressure, while the interlocutor waits.

This asymmetry is not a minor gap. Research on conversational turn-taking shows that the gaps between turns in natural dialogue are typically a fraction of a second to a few seconds, and that listeners begin planning their responses while the current speaker is still talking (Sacks, Schegloff, & Jefferson, 1974; Levelt, 1989). A tool that only begins to help once the interlocutor has finished speaking — and that then requires the user to compose, type, or dictate — inserts a delay that the social fabric of conversation tolerates poorly. Turn-based translation devices institutionalise exactly this delay: one party speaks, the device translates, the other party responds, each step serialised and each participant aware of the mediation.

This paper presents a system designed around the productive half of cross-language conversation. The system, disclosed in U.S. Provisional Patent Application No. 64/170,246 (filed October 6, 2026), is a conversation copilot for smart glasses with three defining properties. First, it is single-sided: only the assisted party wears any device, and the interlocutor is never required to wear, operate, or interact with equipment. Second, it is anticipatory: candidate responses are generated during the interlocutor's turn, not after it, using streaming partial recognition results, the dialogue history of the session, and a persistent profile of the wearer. Third, it is production-oriented: its outputs are not only subtitles to be read but candidate utterances to be spoken, accompanied by audio pronunciation guidance, selectable by a silent gesture, with a free-expression fallback when no candidate matches the wearer's intent.

The contribution of the paper is the articulation of this closed loop as a system design: the architecture that supports it (Section 3), the three-step conversation loop it executes (Section 4), the anticipatory generation mechanism and its latency argument (Section 5), the interaction design for candidate selection and pronunciation support (Section 6), and the user-profile and privacy model (Section 7). Section 2 frames the problem against existing approaches, Section 8 summarises alternative embodiments, and Sections 9–10 discuss limitations and open questions. Throughout, the paper confines itself to the system as disclosed in the patent filing; where numerical values appear, they are the design targets stated in that disclosure.

2. Background and Problem Framing

Existing cross-language assistance falls into two categories, both incomplete in the same way.

Translation devices. Handheld translators, translation earbuds, and smartphone translation applications convert one party's speech into another language. They are typically turn-based and symmetric in burden: the device must be passed, shared, or at least attended to by both parties, and each conversational turn is decomposed into speak–translate–listen steps. These systems solve "understanding the other side" and, in dictation mode, a slow form of production — the user speaks their own language and the device speaks the translation — but the mediated, serial rhythm marks the interaction as machine-assisted throughout, and the user's own voice and presence recede.

Subtitle glasses. A second category projects translated captions onto the lenses of smart glasses via a paired smartphone, providing continuous streaming translation display; U.S. Patent Application Publication No. US20200265839A1, for example, describes glasses with subtitles linked to a phone application. This category removes the turn-taking friction of the first for the receptive direction: the wearer reads the interlocutor's words as they are spoken. But the wearer still receives no assistance in formulating a response in the interlocutor's language. The productive half is untouched.

The missing capability is therefore specific: assistance, at the moment of response formulation, that (i) is ready when the wearer's turn begins, (ii) is personalised enough to propose what this wearer, in this role and domain, is likely to want to say, (iii) supports the wearer in actually pronouncing the chosen utterance, and (iv) degrades gracefully to free expression when the proposals miss. Requirement (i) is the binding constraint, and it is a timing constraint: any assistance that starts computing at the end of the interlocutor's turn must fit its entire latency inside the socially acceptable response gap. Contemporary large language models (LLMs) cannot reliably do this for non-trivial generation. The system's answer is to move the computation earlier — into the interlocutor's turn itself — which is the subject of Section 5.

The asymmetry has a cognitive root. Comprehension is forgiving: a listener can reconstruct meaning from partial input, context, and gesture, and can do so at the speaker's pace. Production is not. Formulating an utterance in a second language requires lexical retrieval, grammatical encoding, and phonetic planning to run in sequence under a deadline set by someone else — the moment the interlocutor stops speaking and looks up. In Levelt's (1989) model of speech production, conceptualisation, formulation, and articulation are incremental but effortful for the non-native speaker, and the effort peaks exactly at turn onset. Assistance that arrives after turn onset therefore competes with the very processes it is meant to support; assistance that is ready at turn onset does not. This is why the design treats readiness — not raw generation quality — as the primary requirement, and why it accepts the risk of preparing candidates that a turn's actual ending may make irrelevant.

Two further observations shape the design. First, pronunciation is a production bottleneck distinct from composition: a wearer may know what they want to say yet be unable to say it intelligibly, and romanised text is a poor scaffold for speakers whose native orthography is non-Latin. The system therefore delivers pronunciation guidance as audio, exploiting the well-established technique of shadowing — immediate vocal imitation of a heard model — rather than as transliterated text (Cherry, 1953; Lambert, 1992). Second, the social acceptability of assistance depends on its invisibility to the interlocutor: selection actions must be silent and minimal, and the interlocutor must encounter an apparently ordinary conversation partner, not a device operator.

3. System Architecture

The system comprises three cooperating subsystems: smart glasses worn by the assisted user, a paired mobile device, and cloud services. The division of labour follows the physical constraints of each tier — sensing and display must sit on the glasses; radio management and edge preprocessing suit the phone the user already carries; and the compute-intensive language pipeline suits the cloud, with on-device alternatives described in Section 8.

Smart glasses. The glasses carry the user-facing sensors and effectors:

• Voice pickup. A microphone array combined with a bone-conduction sensor captures speech. The bone-conduction channel serves a discriminative function: it registers the wearer's own speech through skull vibration, distinguishing it from ambient and interlocutor speech arriving through the air, so that the two voices can be routed to separate processing paths — the interlocutor's to the assistance pipeline, the wearer's to fallback transcription or utterance confirmation.

• Near-eye display. A near-eye display — binocular in the preferred embodiment, with optical transparency preferably not lower than 95% — projects subtitles and candidate responses into the wearer's field of view without occluding the interlocutor.

• Open-ear speaker. An open-ear speaker delivers in-ear text-to-speech (TTS) pronunciation guidance without isolating the wearer from environmental sound; the interlocutor's voice remains directly audible at all times.

• Hands-free selection input. Selection among candidates is performed without touching a phone: preferably by electromyography (EMG) gesture sensing — for instance a wristband detecting pinch gestures — or by temple touch input on the glasses frame.

Paired mobile device. A smartphone paired over Bluetooth performs edge preprocessing of the captured audio (noise suppression and voice activity detection), manages the network uplink, and handles power coordination for the glasses.

Cloud services. The cloud tier performs streaming automatic speech recognition (ASR), neural machine translation (MT), dialogue understanding, user-profile retrieval, anticipatory candidate generation via LLM inference, and TTS synthesis. The architecture does not hard-assign these functions to the cloud in principle: the disclosure contemplates that some or all cloud functions may execute on-device as compute allows, and one alternative embodiment (Section 8) shifts candidate generation to a distilled on-device model when connectivity is unavailable.

This tiering embodies a deliberate asymmetry. Everything the interlocutor experiences — speech in, speech out — passes through the unaided human channel; everything the system contributes is confined to the wearer's senses: text on a lens only the wearer can see, audio in an ear only the wearer can hear, selection by gestures only the wearer performs. The interlocutor needs no device, no application, and no instruction.

4. The Three-Step Conversation Loop

Operation proceeds as a repeating three-step loop (steps S10–S60 in the disclosure's flow diagram), driven by the alternation of turns.

Step 1 — Real-time subtitle projection (S10, S20). The interlocutor's speech is captured and streamed through the ASR and MT engines; bilingual subtitles are projected onto the near-eye display as incremental streaming text. The disclosure states a target end-to-end latency not exceeding approximately 1.5 seconds for this step. A presentation-stabilisation mode may hold completed sentences steady on the display — suited to formal presentations — as against word-by-word streaming for rapid dialogue.

Step 2 — Anticipatory candidate generation and presentation (S30, S40). While the interlocutor is still speaking, the system generates a small set of candidate responses and presents them on the display, each accompanied by pronunciation guidance heard once through the open-ear speaker as TTS audio. The wearer may select a candidate by silent gesture; the generation mechanism is detailed in Section 5 and the interaction design in Section 6.

Step 3 — Selection or free-expression fallback (S50, S60). If the wearer selects a candidate, they read it aloud, completing the assisted turn in their own voice. If no candidate matches the wearer's intent, the wearer speaks once in their native language; the system transcribes and translates the utterance and projects the target-language rendering, which the wearer reads aloud. The loop then returns to Step 1 for the next turn.

Two properties of the loop deserve emphasis. First, the fallback is not an error path but a first-class branch: candidate assistance and free expression are two modes of the same productive support, and the wearer moves between them turn by turn without mode-switching overhead. Second, at no point in any branch is the interlocutor required to wear, operate, or interact with any device — the single-sidedness constraint holds across the whole loop, and is stated as a claim-level feature of the disclosed method.

5. Anticipatory Candidate Generation

The central mechanism of the system is the generation of candidate responses during the interlocutor's turn. Its rationale is a latency argument, and its inputs are threefold.

Inputs. The generator consumes (i) streaming partial ASR hypotheses — the recognition system's evolving, incomplete transcription of the interlocutor's utterance while it is still being spoken; (ii) the accumulated dialogue history of the session; and (iii) the persistent user profile of the wearer (Section 7). The partial hypotheses are the enabling input: they let generation begin from an incomplete but semantically rich prefix of the interlocutor's turn, rather than from the completed utterance. Dialogue in context is highly projectable — questions project answers, proposals project acceptances or qualifications — and the partial stream carries much of that projectable content early.

Generation and ranking. Conditioned on these inputs, an LLM-based generator produces a ranked set of two to three candidate responses in the interlocutor's language — the responses the wearer is most likely to want to give next. Candidates are ranked by a scoring function combining three functional criteria stated in the disclosure: contextual relevance to the unfolding exchange, match to the user profile, and predicted conversational utility. The disclosure specifies the ranking at this functional level — the dimensions along which candidates are compared — rather than as a fixed formula, keeping the mechanism independent of any particular scoring implementation.

The latency argument. The timing structure is the point. Because generation runs in parallel with the interlocutor's ongoing speech (the disclosure's timing diagram aligns the generation window with the interlocutor's turn), the LLM inference latency — typically under three seconds in the disclosed design — is hidden within the natural duration of the interlocutor's speech plus the turn-taking gap that follows it. Candidates are therefore ready for display before the wearer's turn begins. The contrast with post-hoc assistance is stark: a system that waits for the interlocutor to finish, then generates, then presents, spends the entire response gap computing, and either delays the wearer's reply beyond social tolerance or truncates generation quality to fit. Anticipation converts the interlocutor's speaking time — otherwise dead time for the assistance pipeline — into the compute budget for the wearer's reply. This is the same asymmetry that human listeners exploit when they plan responses during a partner's turn (Levelt, 1989); the system mechanises it.

Reading the timing diagram. The disclosure's timing diagram (its FIG. 3) makes the argument concrete. Along a single timeline, the interlocutor's turn occupies an interval of speech; inside that same interval, three machine processes run concurrently: streaming recognition and translation feed the subtitles the wearer is reading; the partial hypotheses accumulate into the generator's context; and candidate generation executes. The wearer's turn begins only after the interlocutor's ends — and by then the candidates have been generated, ranked, displayed, and spoken once into the wearer's ear. The generation window is therefore not the seconds after the interlocutor finishes but the whole duration of the interlocutor's speech, which in substantive business dialogue is typically far longer than any acceptable response gap. The longer and more substantive the interlocutor's turn, the more compute the wearer's reply receives — a scaling property that aligns the system's resources with the conversational moments that need them most, since long turns are precisely those that demand considered answers.

Presentation. Each candidate is presented on the near-eye display together with pronunciation guidance delivered as in-ear TTS audio through the open-ear speaker: the wearer hears each candidate spoken once. Delivering pronunciation as audio rather than as romanised text is advantageous for wearers whose native orthography differs from the target language's — a Chinese-speaking wearer answering in Japanese, for instance, gains little from romanisation but can shadow a heard model directly. The wearer may then select a candidate via a silent gesture (for example, an EMG-detected pinch) and read the selected candidate aloud to the interlocutor.

It is worth being precise about what anticipation does and does not claim. The mechanism does not require the partial hypotheses to be correct in full; it requires the candidate set to be useful when the turn ends, and the fallback branch (Section 4, Step 3) absorbs the cases where the interlocutor's turn develops in an unprojected direction. Anticipation is thus a bet placed with the conversation's own predictability, hedged by design.

6. Interaction Design: Selection, Pronunciation, and Invisibility

The productive loop succeeds socially only if its operation is invisible to the interlocutor and effortless for the wearer. Three design decisions carry this burden.

Silent selection. Candidates are few (two to three) precisely so that selection can be a minimal, silent act: a pinch gesture detected by an EMG wristband, a temple touch, or — in an alternative embodiment — a quiet voice command ("number two"). No screen is touched, no phone is raised, and nothing in the wearer's visible behaviour announces that a selection occurred. The small candidate set is thus not merely a generation convenience; it is what keeps the necessary gesture vocabulary to a small set of distinguishable silent gestures, performable under a table, mid-conversation, without breaking eye contact for more than the glance the display already requires.

Audio pronunciation guidance. A selected candidate still has to be spoken, and for many wearers the binding difficulty is phonetic rather than lexical. The system therefore speaks each candidate once through the open-ear speaker as it is presented. The wearer hears a native-form model of the exact string they are about to say and shadows it — repeating with the model's pronunciation, rhythm, and intonation — rather than decoding a romanised transcription. This matters most where romanisation is least informative: for a wearer whose native orthography is a character script, romanised renderings of the target language add a decoding step without adding phonetic truth, whereas a heard model can be imitated directly. The disclosure identifies precisely this advantage for wearers whose native orthography differs from the target language's.

Composure of the whole. The remaining elements serve the same end. The open-ear speaker leaves the wearer's hearing unoccluded, so the interlocutor's voice is never mediated or attenuated. The high-transparency display leaves the interlocutor's face unobstructed. The bone-conduction channel quietly separates the wearer's voice from the interlocutor's, so the system can tell a spoken selection confirmation, a fallback utterance, and ambient speech apart without any explicit mode switching by the wearer. Each decision removes a small social friction; together they aim at a single effect — the interlocutor experiences a person, not a process.

7. The User Profile and Its Privacy Model

Anticipation is only useful if the candidates sound like the wearer. The system's third input, the persistent user profile, supplies this personalisation. As disclosed, the profile includes the wearer's identity, role, business domain, preferred formality level, and frequently used expressions, built over time from the wearer's explicit inputs and — with consent — from prior conversation sessions. The profile conditions both generation and ranking: it shapes what the candidates say (a payments executive receives domain-appropriate phrasing) and how they are ordered (a candidate matching the wearer's role and habitual register ranks above a generic equivalent).

The formality dimension deserves particular note, because it is where generic machine translation most often betrays a professional user. The same propositional content — agreeing to a deadline, declining a price, proposing a meeting — must be realised at very different levels of formality depending on the wearer's role and the relationship at stake, and an error of register is a social error, not a stylistic one. By carrying the wearer's preferred formality level and frequently used expressions as standing inputs to both generation and ranking, the profile lets the candidate set arrive pre-adjusted to the register the wearer would have chosen, rather than requiring the wearer to re-edit machine output under time pressure — an edit the turn-taking window would rarely permit anyway.

Because the profile is, in effect, a compressed model of a person's professional voice, its handling is specified with care in the disclosure. The profile is stored encrypted, and may be stored on-device. Its session-derived component is built only with the wearer's consent. These are architectural commitments, not policy afterthoughts: encryption and optional on-device storage bound the exposure of the profile data, and the consent condition bounds its growth. The design also implies a graceful degradation: a new wearer with a sparse profile receives less personalised candidates but the same loop — profile depth improves ranking quality, it does not gate functionality.

8. Alternative Embodiments

The disclosure states several alternative embodiments, which map the design space around the preferred configuration:

• On-device generation. When network connectivity is unavailable, candidate responses are generated on the paired mobile device using a distilled on-device language model, with a reduced candidate count.

• Monocular display. The near-eye display may be monocular rather than binocular.

• Voice selection. Selection may be performed by voice command (for example, "number two") rather than by gesture.

• Listen-only mode. The system may operate in a subtitle-only mode (Step 1 of the loop alone) for passive scenarios such as lectures and performances, where the wearer needs comprehension without production support.

These variants share the loop's invariants — single-sided wear, interlocutor non-involvement, and the receptive substrate of streaming subtitles — while trading generation capacity, display immersion, and selection modality against connectivity, hardware, and context.

9. Discussion and Limitations

The system's wager is that the productive half of conversation is assistable within the timing structure of conversation itself, provided computation is moved into the interlocutor's turn. Three implications follow for the design of language-assistance technology generally. First, latency budgets should be accounted against turn structure, not against user patience after a turn ends: the relevant resource is the partner's speaking time, and systems that ignore it will always arrive late. Second, personalisation is not an enhancement to candidate generation but a precondition — a generic candidate set would shift the wearer's burden from composing to evaluating, which under turn pressure may be no lighter. Third, the fallback branch deserves the same design care as the headline mechanism: in any anticipatory system, some turns will develop unpredictably, and the quality of the whole is set by how gracefully those turns are absorbed.

The limitations are equally clear, and this paper states them plainly. The system is presented as disclosed in the patent filing; no user studies, accuracy measurements, or deployment results are claimed, and the numerical values cited (subtitle latency target of approximately 1.5 seconds; language-model inference typically under three seconds; candidate sets of two to three; display transparency preferably not lower than 95%) are the design targets and preferred values stated in that disclosure, not production measurements. Anticipatory generation inherits the failure mode of all prediction: when an interlocutor's turn ends somewhere its prefix did not project, the prepared candidates may be useless, and the wearer falls back to free expression with no net gain for that turn — though, by the loop's design, no net loss either. Candidate quality is bounded by the underlying speech recognition, translation, and language-model components, whose selection the disclosure deliberately leaves open. Finally, a device that listens throughout professional conversations raises consent and confidentiality questions for interlocutors that are legal and social rather than technical; the single-sided design answers the operational question (the interlocutor needs no device) but deployment contexts will need their own norms, a question this paper flags rather than resolves.

10. Conclusion

Existing cross-language tools help a user understand, and then abandon the user at the moment of reply. The system presented here closes that productive half of the conversation loop: streaming translated subtitles for comprehension; anticipatory, profile-conditioned candidate responses — generated inside the interlocutor's turn so that model latency is hidden by conversation's own timing — for production; audio pronunciation guidance and silent gesture selection so the reply can actually be delivered, invisibly; and a free-expression fallback so the wearer is never trapped by the machine's guesses. Worn by one party only, and requiring nothing of the other, it reframes machine interpretation from a mediated exchange into a private competence — assistance that aims, ultimately, to be indistinguishable from ability.

References

Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015).

Cherry, E. C. (1953). Some experiments on the recognition of speech, with one and with two ears. Journal of the Acoustical Society of America, 25(5), 975–979.

Lambert, S. (1992). Shadowing. Meta: Journal des traducteurs, 37(2), 263–273.

Levelt, W. J. M. (1989). Speaking: From intention to articulation. MIT Press.

Ma, M., Huang, L., Xiong, H., Zheng, R., Liu, K., Zheng, B., Zhang, C., He, Z., Liu, H., Li, X., Wu, H., & Wang, H. (2019). STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3025–3036.

Qu, S. (2026). Anticipatory Multilingual Conversation Assistance System for Wearable Devices. U.S. Provisional Patent Application No. 64/170,246, filed October 6, 2026.

Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 50(4), 696–735.

U.S. Patent Application Publication No. US20200265839A1. Smart glasses with subtitle display linked to a mobile application.

AI Disclosure: This working paper was prepared with AI-assisted writing and editing tools. The system description, arguments, and conclusions are the author's own, and the technical content is drawn from the author's patent disclosure cited above.