SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks
BibTeX
@misc{sottovoce,
title = {SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks},
author = {Naoki Kimura and Michinari Kono and Jun Rekimoto},
year = {2019},
note = {CHI '19},
doi = {10.1145/3290605.3300376},
url = {https://doi.org/10.1145/3290605.3300376},
} A solid proof of concept that reconstructs speech audio from ultrasound for controlling unmodified smart speakers, showcasing important system design insight despite prototype limitations in latency, hardware bulk, and speaker dependency.
Expert review (33-view rubric)
Conclusion
SottoVoce is a strong proof of concept for controlling unchanged smart speakers from ultrasound-based silent articulation, but it is not a deployable general SSI system.
- Why it matters
- Its important move is to regenerate speech audio from ultrasound and reuse existing voice assistants, rather than directly classifying silent commands.
- What to trust
- Trust the Alexa command success, Google speech-to-text WER, processing time, and the two-participant, four-command evaluation scope.
- Main weakness
- It is speaker-dependent, small-scale, slow, hardware-heavy, and has no patient evaluation.
Global and route ranking are not yet settled.
Final Procedure 1
Final reading procedure
What route is this work on, what stop condition does it remove, what cost is paid, and what remains weak?
The system captures ultrasound images under the jaw, maps them to Mel spectrograms with Network 1, refines them with Network 2, reconstructs audio with Griffin-Lim, and feeds that audio to a smart speaker.
- The pipeline converts ultrasound image sequences into regenerated speech audio.
- The output is used with unchanged Amazon Echo and Echo Show devices.
- The strongest contribution is system decomposition, not raw recognition performance.
- Long-term use, remounting robustness, and patient use are not established.
- Do not treat this as a real-time or open-vocabulary deployed interface.
- ev_sottovoce_author_claim — An ultrasound-only silent-voice system that regenerates audio and controls an unchanged smart speaker with deep neural networks.
- ev_sottovoce_novelty_audio_reconstruction — The paper's main novelty lies in system design to reconstruct speech audio from ultrasound images and use that to control unmodified smart speakers, rather than direct command recognition from ultrasound features.
- ev_sottovoce_latency — The system is not real-time; with 3.68 s of input, neural networks processing consumes 2.36 s and total processing time is 2.61 s, unsuitable for real-time interaction.
Risk: Do not turn a system proof of concept into a deployment claim.
confidence high (0.95)
Route 2
Route
Which communication route does the paper belong to?
This is a silent reconstruction route: silent articulation is converted into speech audio rather than text or direct command labels.
- Input is ultrasound under the jaw.
- Output is regenerated speech audio.
- It should be compared mainly with speech reconstruction systems, not spelling or low-volume systems.
- Do not rank it directly against SilentSpeller without naming the route difference.
- ev_sottovoce_author_claim — An ultrasound-only silent-voice system that regenerates audio and controls an unchanged smart speaker with deep neural networks.
- ev_sottovoce_novelty_audio_reconstruction — The paper's main novelty lies in system design to reconstruct speech audio from ultrasound images and use that to control unmodified smart speakers, rather than direct command recognition from ultrasound features.
Risk: Do not compare different routes as one metric leaderboard.
confidence high (0.95)
Patient evidence
Is there evidence from target users?
There is no target-patient evidence. Voice-loss use is a possible direction, not a tested result.
- The evaluation uses two collaborators, not target patients.
- Whether the system works for laryngectomy or voice-loss users.
- Do not present this as patient-restoration evidence.
- ev_sottovoce_validation_scope — Approximately 500 speech commands were collected from each of two collaborators using an ultrasound probe under the jaw with paired audio, speaker-dependent training only, and end-to-end testing used four Alexa commands repeated five times each.
Risk: Healthy-subject evidence is not patient evidence.
confidence high (0.95)
Performance Ranking 1
Performance ranking
Which numbers matter, and what do they not prove?
It can enter provisional metric comparison, but global practical rank is not set. The numbers are useful but narrow.
- Network 1 plus Network 2 reached 65.0% smart-speaker command success.
- Network 2 output had 33.56% Google speech-to-text WER.
- The paper is conceptually important but practically weaker than live text-entry systems.
- Global route-aware rank has not been recomputed.
- Two participants, four commands, and speaker dependence make the result narrow.
- ev_sottovoce_metric_results — Network 1 plus Network 2 reached 65.0% smart-speaker command success versus 42.5% for Network 1 alone and 90.0% ground-truth audio; Google speech-to-text WER for Network 2 was 33.56%; total processing time was 2.61 s for a roughly 3.68 s command clip.
- ev_sottovoce_validation_scope — Approximately 500 speech commands were collected from each of two collaborators using an ultrasound probe under the jaw with paired audio, speaker-dependent training only, and end-to-end testing used four Alexa commands repeated five times each.
- ev_sottovoce_limitations — The system is speaker dependent, slow (2.61 seconds per command), limited to four end-to-end commands, uses bulky 3.5 MHz probe hardware under the jaw, and users must adapt their silent articulation for best results.
Risk: Do not treat narrow command success as broad communication success.
confidence high (0.90)
Loop Derived 4
Burden
Who pays for the result?
The main burdens are on the user, the device, and speaker-specific data collection.
- The system uses a 3.5 MHz probe under the jaw.
- Training uses roughly 500 commands per collaborator.
- Hardware and calibration cost dominate deployment difficulty.
- Comfort and repeated-use burden are not quantified.
- user (high): The user must adapt silent articulation and tolerate under-jaw probe placement.
- device (high): A 3.5 MHz probe and supporting capture hardware are required.
- data (high): Speaker-specific data collection is required.
- ev_sottovoce_validation_scope — Approximately 500 speech commands were collected from each of two collaborators using an ultrasound probe under the jaw with paired audio, speaker-dependent training only, and end-to-end testing used four Alexa commands repeated five times each.
- ev_sottovoce_limitations — The system is speaker dependent, slow (2.61 seconds per command), limited to four end-to-end commands, uses bulky 3.5 MHz probe hardware under the jaw, and users must adapt their silent articulation for best results.
Risk: High performance can hide hardware, user, and data cost.
confidence high (0.90)
Stop conditions
What still stops communication?
Many stop conditions remain, especially real time, unseen speakers, wearability, and patient evidence.
- The reported processing time is 2.61 s for a roughly 3.68 s input clip.
- The evaluation uses two collaborators and four Alexa commands.
- The system does not yet clear the practical communication threshold.
- Cross-session remount robustness is not established.
- ev_sottovoce_validation_scope — Approximately 500 speech commands were collected from each of two collaborators using an ultrasound probe under the jaw with paired audio, speaker-dependent training only, and end-to-end testing used four Alexa commands repeated five times each.
- ev_sottovoce_limitations — The system is speaker dependent, slow (2.61 seconds per command), limited to four end-to-end commands, uses bulky 3.5 MHz probe hardware under the jaw, and users must adapt their silent articulation for best results.
- ev_sottovoce_latency — The system is not real-time; with 3.68 s of input, neural networks processing consumes 2.36 s and total processing time is 2.61 s, unsuitable for real-time interaction.
Risk: A paper can remove one stop condition while many remain.
confidence high (0.90)
Recovery
What happens after errors?
There is no formal correction mechanism. The only recovery-like behavior is user adaptation after hearing generated output.
- No edit or reject interface is reported.
- Error recovery is weak for interactive use.
- Whether a correction interface would improve real use is untested.
- user (medium): The user must adjust articulation rather than use a designed repair mechanism.
- ev_sottovoce_limitations — The system is speaker dependent, slow (2.61 seconds per command), limited to four end-to-end commands, uses bulky 3.5 MHz probe hardware under the jaw, and users must adapt their silent articulation for best results.
Risk: A system without repair can fail despite decent top-line numbers.
confidence medium (0.80)
Durability
What breaks after time, remounting, or repeated use?
Long-term use is untested. Probe placement, session drift, speaker-dependent training, and continuous ultrasound use remain risks.
- The study does not report long-term repeated use.
- Durability is likely weak until remounting and long-term safety are tested.
- How stable the system is across days or after probe remounting.
- ev_sottovoce_limitations — The system is speaker dependent, slow (2.61 seconds per command), limited to four end-to-end commands, uses bulky 3.5 MHz probe hardware under the jaw, and users must adapt their silent articulation for best results.
- ev_sottovoce_latency — The system is not real-time; with 3.68 s of input, neural networks processing consumes 2.36 s and total processing time is 2.61 s, unsuitable for real-time interaction.
Risk: Same-session success does not prove durable use.
confidence medium (0.85)
Rankings 1
provisional · route · silent_reconstruction
- Axis
- route_level
- Condition
- Provisional route comparison only
- Cost
- Speaker-specific data, under-jaw ultrasound probe, latency, and small command set.
- Caveat
- Route-aware ranking has not been recomputed for all papers.
Performance numbers 7
- success_rate · lab_live
- 65 % — Network 1 + Network 2, Alexa commands Caveat: Two users and four commands; silent-only end-to-end generality is not established.
- success_rate · lab_live
- 42.5 % — Network 1 only, Alexa commands Caveat: Small command set.
- success_rate · lab_live
- 90 % — Ground-truth encoded and decoded audio Caveat: Upper reference, not generated SSI output.
- WER · lab_offline
- 33.56 % — Network 1 + Network 2, Google speech-to-text Caveat: Small two-user command evaluation.
- latency · lab_offline
- 2.61 s — 3.68 s input command clip Caveat: Not real time.
- other · lab_offline
- 2 participants — Training and testing Caveat: Two collaborators only.
- other · lab_offline
- 500 commands per collaborator, approximate — Speaker-dependent training Caveat: Requires per-speaker data.
Limits 5
- technical (high): The system is speaker-dependent and lacks evidence for unseen speakers.
- technical (high): The system processes for 2.61 s after a 3.68 s input clip, so it is not real time.
- evaluation (high): The end-to-end evaluation is small: two participants, four Alexa commands, five repetitions.
- deployment (high): A large 3.5 MHz probe under the jaw is required, so the current prototype is not a wearable deployment.
- scope (high): There is no patient evaluation; voice-loss use remains untested.
Reading guidance
- Verdict
- full-text draft · priority high · confidence high
- Why it matters
- This paper's key value lies in reframing silent speech interaction as a speech regeneration and ecosystem reuse problem, deploying a two-stage DNN approach to produce audio from ultrasonic tongue and jaw imaging. It demonstrates practical integration with existing voice agents, highlighting a promising architectural direction distinct from direct command recognition methods.
- What to trust
- Basis: full text + existing expert seed. Coverage: high. 7 evidence records back the review.
- What is weak
- Speaker-dependent training; latency unsuitable for real-time use (2.61 s per utterance); differences in silent versus voiced articulation require user adaptation; bulky hardware; potential unknown safety issues with continuous ultrasound emission; small vocabulary size. Only two participants were used for training and testing; the command vocabulary is small (four Alexa commands) in end-to-end testing, repeated five times each; and no speaker-independent or open vocabulary evaluations were performed. The device requires a bulky 3.5 MHz convex probe attached under the jaw and digitized display capture; continuous ultrasonic emission safety is not evaluated; the system is not wearable or miniaturized; and its latency (~2.61 s per command) is too slow for real-time use. Prototype supports only a fixed small command vocabulary in speaker-dependent training; no demonstration of open vocabulary or continuous real-time interaction. Overclaim risk: medium.
- Read before
- SilentSpeller: Towards mobile, hands-free, silent speech text entry using electropalatography
- Read next
- NasoVoce: A Nose-Mounted Low-Audibility Speech Interface for Always-Available Speech Interaction
Axes
- Task
- speech-reconstruction
- Modality
- ultrasound
- Hardware
- 3.5 MHz convex ultrasound probe attached under the jaw, with ultrasound images captured to display monitor and digitized video stored
- Body site
- jaw; oral-cavity
- Output
- speech-audio
- Vocabulary
- Command-level
- Metrics
- Network 1 alone achieved an average 42.5% smart speaker command recognition success; Network 1 plus Network 2 achieved 65.0%; ground-truth audio reached 90.0%. Google speech-to-text word error rates were 41.03% for Network 1 outputs and 33.56% for Network 2 outputs, versus 20.61% for ground truth audio.
- Evaluation mode
- Quantitative smart speaker success rates, word error rates with Google speech-to-text, and qualitative user adaptation observations.
- Review confidence
- high
- Overclaim risk
- medium
Expert take
Kimura et al. present a well-engineered proof of concept for silent speech interaction using ultrasonic imaging and deep neural networks. Their primary contribution is a two-stage neural pipeline converting ultrasonic images captured below the jaw into Mel-spectrogram features and refining those into audio signals, which can then control unmodified commercial smart speakers like Amazon Alexa. This architectural decision to reconstruct audio, rather than directly classify commands, is a significant reframing within SSI research. The study provides modest yet concrete quantitative results: a 65% command recognition success rate with the two-network pipeline, 33.56% word error rate on Google STT, and demonstration of system use with limited command sets. They explicitly discuss limitations such as speaker dependence, system latency (~2.61 s), bulky hardware, and user adaptation requirements for silent speech without vocal fold vibration. These aspects clarify that while the prototype is not ready for real-world deployment or continuous real-time interaction, it is a strong conceptual and technical foundation for future research on SSI architectures that reuse existing voice ecosystems via speech regeneration.
True value
This paper's key value lies in reframing silent speech interaction as a speech regeneration and ecosystem reuse problem, deploying a two-stage DNN approach to produce audio from ultrasonic tongue and jaw imaging. It demonstrates practical integration with existing voice agents, highlighting a promising architectural direction distinct from direct command recognition methods.
What changed
Canon before
Most prior silent speech interfaces recognized commands directly or relied on visible cameras or other sensors, lacking integration with unchanged smart speaker ecosystems.
Delta from canon
This work shifts SSI from direct command recognition to speech audio regeneration that can be fed to standard speech recognition engines and smart speakers without modification.
Position in field
An early and influential demonstration of ultrasound-based speech regeneration SSI with system-level insights stronger than its present prototype performance.
Evidence
“ Recent ing a method known as lip reading, images of the mouth researchers have challenged to use deep neural networks of the speaker or the entire face are captured by a camera, with ultrasound imaging for silent speech [7, 51]; however, and the content of the utterance is estimated from those they are not based on convolutional neural networks and are images [52]. ”
author_claim · Abstract · confidence 1.00
“ Approximately 500 speech and Network 2 was unclear, we observed that the sound commands were collected from each collaborator (Table ??). generated by Network 2 was better than that generated by For each command, as well as the voice utterance, a video of Network 1 (Examples of the output audio signals are given the ultrasonic images was recorded. ”
metric · 4 RESULTS · confidence 0.95
“ Approximately 500 speech and Network 2 was unclear, we observed that the sound commands were collected from each collaborator (Table ??). generated by Network 2 was better than that generated by For each command, as well as the voice utterance, a video of Network 1 (Examples of the output audio signals are given the ultrasonic images was recorded. ”
validation_scope · 4 RESULTS · confidence 0.98
“ For this test, the As our model is speaker dependent, both Network 1 and participants spoke the following four commands, five times Network 2 are trained for each speaker. ”
limitation · 5 END-TO-END EVALUATION · confidence 0.97
“ To speaker (Amazon Echo and Amazon Echo Show), and this increase the number of test sets, data augmentation by apply- test confirmed that the generated sounds can control smart ing Gaussian noise to the input Mel-scale spectrum vectors speakers. ”
deployment_claim · 4 RESULTS · confidence 0.95
“ Approximately 500 speech and Network 2 was unclear, we observed that the sound commands were collected from each collaborator (Table ??). generated by Network 2 was better than that generated by For each command, as well as the voice utterance, a video of Network 1 (Examples of the output audio signals are given the ultrasonic images was recorded. ”
actual_novelty · 3 SYSTEM ARCHITECTURE OF SOTTOVOCE · confidence 0.90
“ The total processing time (including video processing, neural networks processing, and conversion of the Mel-scale spectrum to an audio wave) was 2.61 s. ”
deployment_claim · 3 SYSTEM ARCHITECTURE OF SOTTOVOCE · confidence 0.90
Limits
Technical limits
Speaker-dependent training; latency unsuitable for real-time use (2.61 s per utterance); differences in silent versus voiced articulation require user adaptation; bulky hardware; potential unknown safety issues with continuous ultrasound emission; small vocabulary size.
Evaluation limits
Only two participants were used for training and testing; the command vocabulary is small (four Alexa commands) in end-to-end testing, repeated five times each; and no speaker-independent or open vocabulary evaluations were performed.
Deployment limits
The device requires a bulky 3.5 MHz convex probe attached under the jaw and digitized display capture; continuous ultrasonic emission safety is not evaluated; the system is not wearable or miniaturized; and its latency (~2.61 s per command) is too slow for real-time use.
Scope limits
Prototype supports only a fixed small command vocabulary in speaker-dependent training; no demonstration of open vocabulary or continuous real-time interaction.