End-to-End Speech Translation Progress

Tutorial

• EACL 2021 tutorial: Speech Translation
• Blog: Getting Started with End-to-End Speech Translation
• ACL 2020 Theme paper: Speech Translation and the End-to-End Promise: Taking Stock of Where We Are
• INTERSPEECH 2019 survey talk: Spoken Language Translation

Data

Toolkit

• ESPNet-ST
• Fairseq S2T
• NeurST

Paper

2023

• [arXiv] Tuning Large language model for End-to-end Speech Translation
• [arXiv] Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning
• [arXiv] Multilingual Speech-to-Speech Translation into Multiple Target Languages
• [ICCV] MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition
• [INTERSPEECH] MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation
• [INTERSPEECH] Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer
• [INTERSPEECH] Joint Speech Translation and Named Entity Recognition
• [INTERSPEECH] StyleS2ST: Zero-shot Style Transfer for Direct Speech-to-speech Translation
• [INTERSPEECH] Knowledge Distillation on Joint Task End-to-End Speech Translation
• [INTERSPEECH] GigaST: A 10,000-hour Pseudo Speech Translation Corpus
• [INTERSPEECH] Inter-connection: Effective Connection between Pre-trained Encoder and Decoder for Speech Translation
• [INTERSPEECH] AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation
• [INTERSPEECH] Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoff
• [INTERSPEECH] HK-LegiCoST: Leveraging Non-Verbatim Transcripts for Speech Translation
• [INTERSPEECH] Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Models
• [ICML] Pre-training for Speech Translation: CTC Meets Optimal Transport
• [ACL] UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units
• [ACL] Simple and effective unsupervised speech translation
• [ACL] BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric
• [ACL] SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations
• [ACL] Understanding and Bridging the Modality Gap for Speech Translation
• [ACL] Back Translation for Speech-to-text Translation Without Transcripts
• [ACL] AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation
• [ACL] WACO: Word-Aligned Contrastive Learning for Speech Translation
• [ACL] Attention as a guide for Simultaneous Speech Translation
• [ACL Findings] Speech-to-Speech Translation for a Real-world Unwritten Language
• [ACL Findings] CKDST: Comprehensively and Effectively Distill Knowledge from Machine Translation to End-to-End Speech Translation
• [ACL Findings] Duplex Diffusion Models Improve Speech-to-Speech Translation
• [ACL Findings] DUB: Discrete Unit Back-translation for Speech Translation
• [ACL Findings] End-to-End Simultaneous Speech Translation with Differentiable Segmentation
• [ACL Findings] Implicit Memory Transformer for Computationally Efficient Simultaneous Speech Translation
• [ACL Findings] Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data
• [ICASSP] Textless Direct Speech-to-Speech Translation with Discrete Speech Representation
• [ICASSP] M3ST: Mix at Three Levels for Speech Translation
• [EACL Findings] Generating Synthetic Speech from SpokenVocab for Speech Translation
• [AAAI] Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Data

2022

• [arXiv] AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation
• [arXiv] Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Features
• [arXiv] ArzEn-ST: A Three-way Speech Translation Corpus for Code-Switched Egyptian Arabic - English
• [arXiv] Multilingual Simultaneous Speech Translation
• [arXiv] Prabhupadavani: A Code-mixed Speech Translation Data for 25 Languages
• [EMNLP Findings] Does Simultaneous Speech Translation need Simultaneous Models?
• [EMNLP Findings] RedApt: An Adaptor for WAV2VEC 2 Encoding Faster and Smaller Speech Translation without Quality Compromise
• [INTERSPEECH] Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation
• [INTERSPEECH] Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation
• [INTERSPEECH] Large-Scale Streaming End-to-End Speech Translation with Neural Transducers
• [INTERSPEECH] Speech Segmentation Optimization using Segmented Bilingual Speech Corpus for End-to-end Speech Translation
• [INTERSPEECH] Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation
• [INTERSPEECH] SHAS: Approaching optimal Segmentation for End-to-End Speech Translation
• [INTERSPEECH] Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation
• [INTERSPEECH] M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation
• [NAACL] Textless Speech-to-Speech Translation on Real Data
• [ICML] Revisiting End-to-End Speech-to-Text Translation From Scratch
• [ICML] Translatotron 2: Robust direct speech-to-speech translation
• [ACL] Learning When to Translate for Streaming Speech
• [ACL] Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation
• [ACL] UniST: Unified End-to-end Model for Streaming and Non-streaming Speech Translation
• [ACL] Direct speech-to-speech translation with discrete units
• [ACL] STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation
• [ACL Findings] End-to-End Speech Translation for Code Switched Speech
• [LREC] CVSS Corpus and Massively Multilingual Speech-to-Speech Translation
• [LREC] LibriS2S: A German-English Speech-to-Speech Translation Corpus
• [ICASSP] Tackling data scarcity in speech translation using zero-shot multilingual machine translation techniques
• [NN] Improving data augmentation for low resource speech-to-text translation with diverse paraphrasing
• [AAAI] Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement

2021

• [arXiv] Efficient Transformer for Direct Speech Translation
• [arXiv] Zero-shot Speech Translation
• [arXiv] Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention
• [ASRU] Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates
• [ASRU] Assessing Evaluation Metrics for Speech-to-Speech Translation
• [ASRU] Enabling Zero-shot Multilingual Spoken Language Translation with Language-Specific Encoders and Decoders
• [ICNLSP] Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translation
• [INTERSPEECH] Impact of Encoding and Segmentation Strategies on End-to-End Simultaneous Speech Translation
• [EMNLP] Speechformer: Reducing Information Loss in Direct Speech Translation
• [EMNLP] Is "moby dick" a Whale or a Bird? Named Entities and Terminology in Speech Translation
• [EMNLP] Mutual-Learning Improves End-to-End Speech Translation
• [INTERSPEECH] End-to-end Speech Translation via Cross-modal Progressive Training
• [INTERSPEECH] CoVoST 2 and Massively Multilingual Speech-to-Text Translation
• [INTERSPEECH] The Multilingual TEDx Corpus for Speech Recognition and Translation
• [INTERSPEECH] Large-Scale Self-and Semi-Supervised Learning for Speech Translation
• [INTERSPEECH] Kosp2e: Korean Speech to English Translation Corpus
• [INTERSPEECH] AlloST: Low-resource Speech Translation without Source Transcription
• [INTERSPEECH] SpecRec: An Alternative Solution for Improving End-to-End Speech-to-Text Translation via Spectrogram Reconstruction
• [INTERSPEECH] Optimally Encoding Inductive Biases into the Transformer Improves End-to-End Speech Translation
• [INTERSPEECH] ASR Posterior-based Loss for Multi-task End-to-end Speech Translation
• [AMTA] Simultaneous Speech Translation for Live Subtitling: from Delay to Display
• [ACL] Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders
• [ACL] Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• [ACL] Lightweight Adapter Tuning for Multilingual Speech Translation
• [ACL] Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?
• [ACL] Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task
• [ACL] Beyond Sentence-Level End-to-End Speech Translation: Context Helps
• [ACL Findings] Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR
• [ACL Findings] AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation
• [ACL Findings] RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer
• [ACL Findings] Learning Shared Semantic Space for Speech-to-Text Translation
• [ACL Findings] Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation
• [ACL Findings] How to Split: the Effect of Word Segmentation on Gender Bias in Speech Translation
• [ACL Demo] NeurST: Neural Speech Translation Toolkit
• [ICML] Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation
• [NAACL] Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation
• [NAACL] Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks
• [NAACL AutoSimTrans] BSTC: A Large-Scale Chinese-English Speech Translation Dataset
• [AmericasNLP] Highland Puebla Nahuatl–Spanish Speech Translation Corpus for Endangered Language Documentation
• [ICASSP] Task Aware Multi-Task Learning for Speech to Text Tasks
• [ICASSP] A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks
• [ICASSP] An Empirical Study of End-to-end Simultaneous Speech Translation Decoding Strategies
• [ICASSP] Streaming Simultaneous Speech Translation with Augmented Memory Transformer
• [ICASSP] Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoder
• [ICASSP] Cascaded Models With Cyclic Feedback For Direct Speech Translation
• [ICASSP] Jointly Trained Transformers models for Spoken Language Translation
• [ICASSP] Efficient Use of End-to-end Data in Spoken Language Processing
• [EACL] CTC-based Compression for Direct Speech Translation
• [EACL] Streaming Models for Joint Speech Recognition and Translation
• [IberSPEECH] mintzai-ST: Corpus and Baselines for Basque-Spanish Speech Translation
• [AAAI] Consecutive Decoding for Speech-to-text Translation
• [AAAI] UWSpeech: Speech to Speech Translation for Unwritten Languages
• [AAAI] "Listen, Understand and Translate": Triple Supervision Decouples End-to-end Speech-to-text Translation
• [SLT] Tight Integrated End-to-End Training for Cascaded Speech Translation
• [SLT] Transformer-based Direct Speech-to-speech Translation with Transcoder

2020

• [arXiv] Bridging the Modality Gap for Speech-to-Text Translation
• [arXiv] CSTNet: Contrastive Speech Translation Network for Self-Supervised Speech Representation Learning
• [CLiC-IT] On Knowledge Distillation for Direct Speech Translation
• [COLING] Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation
• [COLING] Breeding Gender-aware Direct Speech Translation Systems
• [AACL] SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation
• [AACL Demo] fairseq S2T: Fast Speech-to-Text Modeling with fairseq
• [EMNLP] Effectively pretraining a speech translation decoder with Machine Translation data
• [EMNLP Findings] Adaptive Feature Selection for End-to-End Speech Translation
• [AMTA] On Target Segmentation for Direct Speech Translation
• [INTERSPEECH] Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection
• [INTERSPEECH] Relative Positional Encoding for Speech Recognition and Direct Translation
• [INTERSPEECH] Contextualized Translation of Automatically Segmented Speech
• [INTERSPEECH] Self-Training for End-to-End Speech Translation
• [INTERSPEECH] Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation
• [INTERSPEECH] Self-Supervised Representations Improve End-to-End Speech Translation
• [INTERSPEECH] Investigating Self-Supervised Pre-Training for End-to-End Speech Translation
• [TACL] Consistent Transcription and Translation of Speech
• [ACL] Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation
• [ACL] Phone Features Improve Speech Translation
• [ACL] Curriculum Pre-training for End-to-End Speech Translation
• [ACL] SimulSpeech: End-to-End Simultaneous Speech to Text Translation
• [ACL] Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus
• [ACL Theme] Speech Translation and the End-to-End Promise: Taking Stock of Where We Are
• [ACL Demo] ESPnet-ST: All-in-One Speech Translation Toolkit
• [LREC] CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus
• [LREC] MuST-Cinema: a Speech-to-Subtitles corpus
• [LREC] MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible
• [LREC] LibriVoxDeEn: A Corpus for German-to-English Speech Translation and Speech Recognition
• [ICASSP] Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates
• [ICASSP] Instance-Based Model Adaptation For Direct Speech Translation
• [ICASSP] Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learning
• [ICASSP] Analyzing ASR pretraining for low-resource speech-to-text translation
• [ICASSP] End-to-End Speech Translation with Self-Contained Vocabulary Manipulation
• [AAAI] Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
• [AAAI] Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding

2019

• [ASRU] One-To-Many Multilingual End-to-end Speech Translation
• [ASRU] Multilingual End-to-End Speech Translation
• [ASRU] Speech-to-speech Translation between Untranscribed Unknown Languages
• [ASRU] A Comparative Study on End-to-end Speech to Text Translation
• [IWSLT] Harnessing Indirect Training Data for End-to-End Automatic Speech Translation: Tricks of the Trade
• [IWSLT] On Using SpecAugment for End-to-End Speech Translation
• [INTERSPEECH] End-to-End Speech Translation with Knowledge Distillation
• [INTERSPEECH] Adapting Transformer to End-to-end Spoken Language Translation
• [INTERSPEECH] Direct speech-to-speech translation with a sequence-to-sequence model
• [ACL] Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation
• [ACL] Attention-Passing Models for Robust and Data-Efficient End-to-End Speech Translation
• [NAACL] Pre-training on High-Resource Speech Recognition Improves Low-Resource Speech-to-Text Translation
• [NAACL] MuST-C: a Multilingual Speech Translation Corpus
• [NAACL] Fluent Translations from Disfluent Speech in End-to-End Speech Translation
• [ICASSP] Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation
• [ICASSP] Towards unsupervised speech-to-text translation
• [ICASSP] Towards End-to-end Speech-to-text Translation with Two-pass Decoding

2018

• [NIPS] How2: A Large-scale Dataset for Multimodal Language Understanding
• [IberSPEECH] End-to-End Speech Translation with the Transformer
• [INTERSPEECH] Low-Resource Speech-to-Text Translation
• [LREC] Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluation
• [NAACL] Tied multitask learning for neural speech translation
• [ICASSP] End-to-End Automatic Speech Translation of Audiobooks

2017

• [INTERSPEECH] Sequence-to-Sequence Models Can Directly Translate Foreign Speech
• [INTERSPEECH] Structured-based Curriculum Learning for End-to-end English-Japanese Speech Translation
• [EACL] Towards speech-to-text translation without speech recognition

2016

• [NIPS Workshop] Listen and translate: A proof of concept for end-to-end speech-to-text translation
• [NAACL] An Attentional Model for Speech Translation Without Transcription

2013

• [IWSLT] Improved Speech-to-Text Translation with the Fisher and Callhome Spanish–English Speech Translation Corpus

Contact

Changhan Wang ([email protected])