Amazon's 36 ICASSP papers touch on everything audio

{"value":"The International Conference on Acoustics, Speech, and Signal Processing ([ICASSP](https://www.amazon.science/conferences-and-events/icassp-2021)) starts next week, and as Alexa principal research scientist Ariya Rastrow [explained last year](https://www.amazon.science/blog/icassp-what-signal-processing-has-come-to-mean), it casts a wide net. The topics of the 36 Amazon research papers at this year’s ICASSP range from the classic signal-processing problems of noise and echo cancellation to such far-flung problems as separating song vocals from instrumental tracks and regulating translation length.\n\nA plurality of the papers, however, concentrate on the core technology of **automatic speech recognition** (ASR), or converting an acoustic speech signal into text:\n\n- [**ASR n-best fusion nets**](https://www.amazon.science/publications/asr-n-best-fusion-nets)\n[Xinyue Liu](https://www.amazon.science/author/xinyue-liu), Mingda Li, [Luoxin Chen](https://www.amazon.science/author/luoxin-chen), [Prashan Wanigasekara](https://www.amazon.science/author/prashan-wanigasekara), [Weitong Ruan](https://www.amazon.science/author/wael-hamza), [Haidar Khan](https://www.amazon.science/author/haidar-khan), [Wael Hamza](https://www.amazon.science/author/wael-hamza), [Chengwei Su](https://www.amazon.science/author/chengwei-su)\n\n- [**Bifocal neural ASR: Exploiting keyword spotting for inference optimization**](https://www.amazon.science/publications/bifocal-neural-asr-exploiting-keyword-spotting-for-inference-optimization)\nJon Macoskey, [Grant P. Strimel](https://www.amazon.science/author/grant-p-strimel), [Ariya Rastrow](https://www.amazon.science/author/ariya-rastrow)\n\n- [**Domain-aware neural language models for speech recognition**](https://www.amazon.science/publications/domain-aware-neural-language-models-for-speech-recognition)\n[Linda Liu](https://www.amazon.science/author/linda-liu), [Yile Gu](https://www.amazon.science/author/yile-gu), [Aditya Gourav](https://www.amazon.science/author/aditya-gourav), [Ankur Gandhe](https://www.amazon.science/author/ankur-gandhe), [Shashank Kalmane](https://www.amazon.science/author/shashank-kalmane), [Denis Filimonov](https://www.amazon.science/author/denis-filiminov), [Ariya Rastrow](https://www.amazon.science/author/ariya-rastrow), [Ivan Bulyko](https://www.amazon.science/author/ivan-bulyko)\n\n- [**End-to-end multi-channel transformer for speech recognition**](https://www.amazon.science/publications/end-to-end-multi-channel-transformer-for-speech-recognition)\n[Feng-Ju Chang](https://www.amazon.science/author/feng-ju-chang), [Martin Radfar](https://www.amazon.science/author/martin-radfar), [Athanasios Mouchtaris](https://www.amazon.science/author/athanasios-mouchtaris), [Brian King](https://www.amazon.science/author/brian-king), [Siegfried Kunzmann](https://www.amazon.science/author/siegfried-kunzmann)\n\n- [**Improved robustness to disfluencies in RNN-transducer-based speech recognition**](https://www.amazon.science/publications/improved-robustness-to-disfluencies-in-rnn-transducer-based-speech-recognition)\n[Valentin Mendelev](https://www.amazon.science/author/valentin-mendelev), Tina Raissi, Guglielmo Camporese, [Manuel Giollo](https://www.amazon.science/author/manuel-giollo)\n\n- [**Personalization strategies for end-to-end speech recognition systems**](https://www.amazon.science/publications/personalization-strategies-for-end-to-end-speech-recognition-systems)\n[Aditya Gourav](https://www.amazon.science/author/aditya-gourav), [Linda Liu](https://www.amazon.science/author/linda-liu), [Ankur Gandhe](https://www.amazon.science/author/ankur-gandhe), [Yile Gu](https://www.amazon.science/author/yile-gu), [Guitang Lan](https://www.amazon.science/author/guitang-lan), [Xiangyang Huang](https://www.amazon.science/author/xiangyang-huang), [Shashank Kalmane](https://www.amazon.science/author/shashank-kalmane), [Gautam Tiwari](https://www.amazon.science/author/gautam-tiwari), [Denis Filimonov](https://www.amazon.science/author/denis-filiminov), [Ariya Rastrow](https://www.amazon.science/author/ariya-rastrow), [Andreas Stolcke](https://www.amazon.science/author/andreas-stolcke), [Ivan Bulyko](https://www.amazon.science/author/ivan-bulyko) \n\n- [**reDAT: Accent-invariant representation for end-to-end ASR by domain adversarial training with relabeling**](https://www.amazon.science/publications/redat-accent-invariant-representation-for-end-to-end-asr-by-domain-adversarial-training-with-relabeling)\nHu Hu, [Xuesong Yang](https://www.amazon.science/author/xuesong-yang), [Zeynab Raeesy](https://www.amazon.science/author/zeynab-raeesy), [Jinxi Guo](https://www.amazon.science/author/jinxi-guo), [Gokce Keskin](https://www.amazon.science/author/gokce-keskin), [Harish Arsikere](https://www.amazon.science/author/harish-arsikere), [Ariya Rastrow](https://www.amazon.science/author/ariya-rastrow), [Andreas Stolcke](https://www.amazon.science/author/andreas-stolcke), [Roland Maas](https://www.amazon.science/author/roland-maas) \n\n- [**Sparsification via compressed sensing for automatic speech recognition**](https://www.amazon.science/publications/sparsification-via-compressed-sensing-for-automatic-speech-recognition)\nKai Zhen, [Hieu Duy Nguyen](https://www.amazon.science/author/hieu-duy-nguyen), [Feng-Ju Chang](https://www.amazon.science/author/feng-ju-chang), [Athanasios Mouchtaris](https://www.amazon.science/author/athanasios-mouchtaris), [Ariya Rastrow](https://www.amazon.science/author/ariya-rastrow)\n\n- [**Streaming multi-speaker ASR with RNN-T**](https://www.amazon.science/publications/streaming-multi-speaker-asr-with-rnn-t)\n[Ilya Sklyar](https://www.amazon.science/author/ilya-sklyar), [Anna Piunova](https://www.amazon.science/author/anna-piunova), [Yulan Liu](https://www.amazon.science/author/yulan-liu) \n\n- [**Using synthetic audio to improve the recognition of out-of-vocabulary words in end-to-end ASR systems**](https://www.amazon.science/publications/using-synthetic-audio-to-improve-the-recognition-of-out-of-vocabulary-words-in-end-to-end-asr-systems)\nXianrui Zheng, [Yulan Liu](https://www.amazon.science/author/yulan-liu), [Deniz Gunceler](https://www.amazon.science/author/deniz-gunceler), [Daniel Willett](https://www.amazon.science/author/daniel-willett)\n\n\n\nTo enable personalization of end-to-end automatic-speech-recognition systems, Linda Liu, Aditya Gourav and their colleagues use a word-level biasing finite state transducer, or FST (left). A subword-level FST preserves the weights of the word-level FST. For instance, the weight between state 0 and 5 of the subword-level FST (representing the word “player”) is (-1.6) +(- 1.6)+(-4.8) = -8.\n\nFROM [\\"PERSONALIZATION STRATEGIES FOR END-TO-END SPEECH RECOGNITION SYSTEMS\\"](https://www.amazon.science/publications/personalization-strategies-for-end-to-end-speech-recognition-systems)\n\nTwo of the papers address **language (or code) switching**, a more complicated version of ASR in which the speech recognizer must also determine which of several possible languages is being spoken: \n\n- [**Joint ASR and language identification using RNN-T: An efficent approach to dynamic language switching**](https://www.amazon.science/publications/joint-asr-and-language-identification-using-rnn-t-an-efficent-approach-to-dynamic-language-switching)\n[Surabhi Punjabi](https://www.amazon.science/author/surabhi-punjabi), [Harish Arsikere](https://www.amazon.science/author/harish-arsikere), [Zeynab Raeesy](https://www.amazon.science/author/zeynab-raeesy), [Chander Chandak](https://www.amazon.science/author/chander-chandak), [Nikhil Bhave](https://www.amazon.science/author/nikhil-bhave), [Markus Mueller](https://www.amazon.science/author/markus-mueller), [Sergio Murillo](https://www.amazon.science/author/sergio-murillo), [Ariya Rastrow](https://www.amazon.science/author/ariya-rastrow), [Andreas Stolcke](https://www.amazon.science/author/andreas-stolcke), [Jasha Droppo](https://www.amazon.science/author/jasha-droppo), [Sri Garimella](https://www.amazon.science/author/sri-garimella), [Roland Maas](https://www.amazon.science/author/roland-maas), [Mat Hans](https://www.amazon.science/author/mat-hans), [Athanasios Mouchtaris](https://www.amazon.science/author/athanasios-mouchtaris), [Siegfried Kunzmann](https://www.amazon.science/author/siegfried-kunzmann)\n\n- [**Transformer-transducers for code-switched speech recognition**](https://www.amazon.science/publications/transformer-transducers-for-code-switched-speech-recognition)\nSiddharth Dalmia, [Yuzong Liu](https://www.amazon.science/author/yuzong-liu), [Srikanth Ronanki](https://www.amazon.science/author/srikanth-ronanki), [Katrin Kirchhoff ](https://www.amazon.science/author/katrin-kirchhoff)\n\nThe acoustic speech signal contains more information than just the speaker’s words; how the words are said can change their meaning. Such **paralinguistic signals** can be useful for a voice agent trying to determine how to interpret the raw text. Two of Amazon’s ICASSP papers focus on such signals:\n\n- [**Contrastive unsupervised learning for speech emotion recognition**](https://www.amazon.science/publications/contrastive-unsupervised-learning-for-speech-emotion-recognition)\nMao Li, [Bo Yang](https://www.amazon.science/author/bo-yang), [Joshua Levy](https://www.amazon.science/author/joshua-levy), [Andreas Stolcke](https://www.amazon.science/author/andreas-stolcke), [Viktor Rozgic](https://www.amazon.science/author/viktor-rozgic), [Spyros Matsoukas](https://www.amazon.science/author/spyros-matsoukas), [Constantinos Papayiannis](https://www.amazon.science/author/constantinos-papayiannis), [Daniel Bone](https://www.amazon.science/author/daniel-bone), [Chao Wang](https://www.amazon.science/author/chao-wang) \n\n- [**Disentanglement for audiovisual emotion recognition using multitask setup**](https://www.amazon.science/publications/disentanglement-for-audiovisual-emotion-recognition-using-multitask-setup)\nRaghuveer Peri, [Srinivas Parthasarathy](https://www.amazon.science/author/srinivas-parthasarathy), [Charles Bradshaw](https://www.amazon.science/author/charles-bradshaw), [Shiva Sundaram](https://www.amazon.science/blog/null) \n\nSeveral papers address other **extensions of ASR**, such as speaker diarization, or tracking which of several speakers issues each utterance; inverse text normalization, or converting the raw ASR output into a format useful to downstream applications; and acoustic event classification, or recognizing sounds other than human voices:\n\n- [**BW-EDA-EEND: Streaming end-to-end neural speaker diarization for a variable number of speakers**](https://www.amazon.science/publications/bw-eda-eend-streaming-end-to-end-neural-speaker-diarization-for-a-variable-number-of-speakers)\n[Eunjung Han](https://www.amazon.science/author/eunjung-han), [Chul Lee](https://www.amazon.science/author/chul-lee), [Andreas Stolcke](https://www.amazon.science/author/andreas-stolcke)\n\n- [**Neural inverse text normalization**](https://www.amazon.science/publications/neural-inverse-text-normalization)\n[Monica Sunkara](https://www.amazon.science/author/monica-sunkara), [Chaitanya Shivade](https://www.amazon.science/author/chaitanya-shivade), [Sravan Bodapati](https://www.amazon.science/author/sravan-bodapati), [Katrin Kirchhoff](https://www.amazon.science/author/katrin-kirchhoff)\n\n- [**Unsupervised and semi-supervised few-shot acoustic event classification**](https://www.amazon.science/publications/unsupervised-and-semi-supervised-few-shot-acoustic-event-classification)\nHsin-Ping Huang, [Krishna C. Puvvada](https://www.amazon.science/author/krishna-c-puvvada), [Ming Sun](https://www.amazon.science/author/ming-sun), [Chao Wang](https://www.amazon.science/author/chao-wang)\n\n**Speech enhancement**, or removing noise and echo from the speech signal, has been a prominent topic at ICASSP since the conference began in 1976. But more recent work on the topic — including Amazon’s two papers this year — uses deep-learning methods:\n\n\n\nThe structure of a joint echo control and noise suppression system from Amazon. A microphone (mic) captures the output of a loudspeaker, along with noise and echo. The echo is partially cancelled by an adaptive filter (ĥf), which uses the signal to the speaker. The microphone signal then passes to a residual-echo-suppression (RES) algorithm.\n\nFROM [\\"LOW-COMPLEXITY, REAL-TIME JOINT NEURAL ECHO CONTROL AND SPEECH ENHANCEMENT BASED ON PERCEPNET\\"](https://www.amazon.science/publications/low-complexity-real-time-joint-neural-echo-control-and-speech-enhancement-based-on-percepnet)\n\n- [**Enhancing into the codec: Noise robust speech coding with vector-quantized autoencoders**](https://www.amazon.science/publications/enhancing-into-the-codec-noise-robust-speech-coding-with-vector-quantized-autoencoders)\nJonah Casebeer, Vinjai Vale, [Umut Isik](https://www.amazon.science/author/umut-isik), [Jean-Marc Valin](https://www.amazon.science/author/jean-marc-valin), [Ritwik Giri](https://www.amazon.science/author/ritwik-giri), [Arvindh Krishnaswamy](https://www.amazon.science/author/arvindh-krishnaswamy)\n\n- [**Low-complexity, real-time joint neural echo control and speech enhancement based on Percepnet**](https://www.amazon.science/publications/low-complexity-real-time-joint-neural-echo-control-and-speech-enhancement-based-on-percepnet)\n[Jean-Marc Valin](https://www.amazon.science/author/jean-marc-valin), [Srikanth V. Tenneti](https://www.amazon.science/author/srikanth-v-tenneti), [Karim Helwani](https://www.amazon.science/author/karim-helwani), [Umut Isik](https://www.amazon.science/author/umut-isik), [Arvindh Krishnaswamy](https://www.amazon.science/author/arvindh-krishnaswamy)\n\nEvery interaction with Alexa begins with a wake word — usually “Alexa”, but sometimes “computer” or “Echo”. So at ICASSP, Amazon usually presents work on wake word detection — or **keyword spotting**, as it’s more generally known:\n\n- [**Exploring the application of synthetic audio in training keyword spotters**](https://www.amazon.science/publications/exploring-the-application-of-synthetic-audio-in-training-keyword-spotters)\n[Andrew Werchniak](https://www.amazon.science/author/andrew-werchniak), [Roberto Barra-Chicote](https://www.amazon.science/author/roberto-barra-chicote), [Yuriy Mishchenko](https://www.amazon.science/author/yuriy-mishchenko), [Jasha Droppo](https://www.amazon.science/author/jasha-droppo), [Jeff Condal](https://www.amazon.science/author/jeff-condal), [Peng Liu](https://www.amazon.science/author/peng-liu), [Anish Shah ](https://www.amazon.science/author/anish-shah)\n\nIn many spoken-language systems, the next step after ASR **is natural-language understanding** (NLU), or making sense of the text output from the ASR system:\n\n-[ **Introducing deep reinforcement learning to NLU ranking tasks**](https://www.amazon.science/publications/introducing-deep-reinforcement-learning-to-nlu-ranking-tasks)\n[Ge Yu](https://www.amazon.science/author/ge-yu), [Chengwei Su](https://www.amazon.science/author/chengwei-su), [Emre Barut](https://www.amazon.science/author/emre-barut) \n- [**Language model is all you need: Natural language understanding as question answering**](https://www.amazon.science/publications/language-model-is-all-you-need-natural-language-understanding-as-question-answering)\n[Mahdi Namazifar](https://www.amazon.science/author/mahdi-namazifar), [Alexandros Papangelis](https://www.amazon.science/author/alexandros-papangelis), [Gokhan Tur](https://www.amazon.science/author/gokhan-tur), [Dilek Hakkani-Tür ](https://www.amazon.science/author/dilek-hakkani-tur)\n\nIn some contexts, however, it’s possible to perform both ASR and NLU with a single model, in a task known as **spoken-language understanding:**\n\n- [**Do as I mean, not as I say: Sequence loss training for spoken language understanding**](https://www.amazon.science/publications/do-as-i-mean-not-as-i-say-sequence-loss-training-for-spoken-language-understanding)\n[Milind Rao](https://www.amazon.science/author/milind-rao), [Pranav Dheram](https://www.amazon.science/author/pranav-dheram), [Gautam Tiwari](https://www.amazon.science/author/gautam-tiwari), [Anirudh Raju](https://www.amazon.science/author/anirudh-raju), [Jasha Droppo](https://www.amazon.science/author/jasha-droppo), [Ariya Rastrow](https://www.amazon.science/author/ariya-rastrow), [Andreas Stolcke ](https://www.amazon.science/author/andreas-stolcke)\n\n- [**Graph enhanced query rewriting for spoken language understanding system**](https://www.amazon.science/publications/graph-enhanced-query-rewriting-for-spoken-language-understanding-system)\nSiyang Yuan, [Saurabh Gupta](https://www.amazon.science/author/saurabh-gupta), [Xing Fan](https://www.amazon.science/author/xing-fan), [Derek Liu](https://www.amazon.science/author/derek-liu), [Yang Liu](https://www.amazon.science/author/yang-liu), C[henlei (Edward) Guo](https://www.amazon.science/author/chenlei-guo) \n\n- [**Top-down attention in end-to-end spoken language understanding**](https://www.amazon.science/publications/top-down-attention-in-end-to-end-spoken-language-understanding)\nYixin Chen, [Weiyi Lu](https://www.amazon.science/author/weiyi-lu), [Alejandro Mottini ](https://www.amazon.science/author/alejandro-mottini), [Erran Li](https://www.amazon.science/author/erran-li), [Jasha Droppo](https://www.amazon.science/author/jasha-droppo), [Zheng Du](https://www.amazon.science/author/zheng-du), [Belinda Zeng](https://www.amazon.science/author/belinda-zeng)\n\n\n\nA spoken-language-understanding system combines automatic speech recognition (ASR) and natural-language understanding (NLU) in a single model.\n\nFROM [\\"DO AS I MEAN, NOT AS I SAY: SEQUENCE LOSS TRAINING FOR SPOKEN LANGUAGE UNDERSTANDING\\"](https://www.amazon.science/publications/do-as-i-mean-not-as-i-say-sequence-loss-training-for-spoken-language-understanding)\n\nAn interaction with a voice service, which begins with keyword spotting, ASR, and NLU, often culminates with the agent’s use of synthesized speech to relay a response. The agent’s **text-to-speech** model converts the textual outputs of various NLU and dialogue systems into speech:\n\n- [**CAMP: A two-stage approach to modelling prosody in context**](https://www.amazon.science/publications/camp-a-two-stage-approach-to-modelling-prosody-in-context)\nZack Hodari, [Alexis Moinet](https://www.amazon.science/author/alexis-moinet), [Sri Karlapati](https://www.amazon.science/author/sri-karlapati), [Jaime Lorenzo-Trueba](https://www.amazon.science/author/jaime-lorenzo-trueba), [Thomas Merritt](https://www.amazon.science/author/thomas-merritt), [Arnaud Joly](https://www.amazon.science/author/arnaud-joly), [Ammar Abbas](https://www.amazon.science/author/ammar-abbas), [Penny Karanasou](https://www.amazon.science/author/penny-karanasou), [Thomas Drugman](https://www.amazon.science/author/thomas-drugman)\n\n- [**Low-resource expressive text-to-speech using data augmentation**](https://www.amazon.science/publications/low-resource-expressive-text-to-speech-using-data-augmentation)\n[Goeric Huybrechts](https://www.amazon.science/author/goeric-huybrechts), [Thomas Merritt](https://www.amazon.science/author/thomas-merritt), [Giulia Comini](https://www.amazon.science/author/giulia-comini), [Bartek Perz](https://www.amazon.science/author/bartek-perz), [Raahil Shah](https://www.amazon.science/author/raahil-shah), [Jaime Lorenzo-Trueba](https://www.amazon.science/author/jaime-lorenzo-trueba) \n\n- [**Prosodic representation learning and contextual sampling for neural text-to-speech**](https://www.amazon.science/publications/prosodic-representation-learning-and-contextual-sampling-for-neural-text-to-speech)\n[Sri Karlapati](https://www.amazon.science/author/sri-karlapati), [Ammar Abbas](), Zack Hodari], [Alexis Moinet](https://www.amazon.science/author/alexis-moinet), [Arnaud Joly](https://www.amazon.science/author/arnaud-joly), [Penny Karanasou](https://www.amazon.science/author/penny-karanasou), [Thomas Drugman](https://www.amazon.science/author/thomas-drugman)\n\n- [**Universal neural vocoding with Parallel WaveNet**](https://www.amazon.science/publications/universal-neural-vocoding-with-parallel-wavenet)\n[Yunlong Jiao](https://www.amazon.science/author/yunlong-jiao), [Adam Gabrys](https://www.amazon.science/author/adam-gabrys), Georgi Tinchev, [Bartosz Putrycz](https://www.amazon.science/author/bartosz-putrycz), [Daniel Korzekwa](https://www.amazon.science/author/daniel-korzekwa), [Viacheslav Klimkov](https://www.amazon.science/author/viacheslav-klimkov)\n\nAll of the preceding research topics have implications for voice services like Alexa, but Amazon has a range of other products and services that rely on audio-signal processing. Three of Amazon’s papers at this year’s ICASSP relate to **audio-video synchronization**: two deal with dubbing audio in one language onto video shot in another, and one describes how to detect synchronization errors in video — as when, for example, the sound of a tennis ball being struck and the shot of the racquet hitting the ball are misaligned:\n\n- [**Detection of audio-video synchronization errors via event detection**](https://www.amazon.science/publications/detection-of-audio-video-synchronization-errors-via-event-detection)\nJoshua P. Ebenezer, [Yongjun Wu](https://www.amazon.science/author/yongjun-wu), [Hai Wei](https://www.amazon.science/author/hai-wei), [Sriram Sethuraman](https://www.amazon.science/author/sriram-sethuraman), [Zongyi Liu ](https://www.amazon.science/author/zongyi-liu)\n\n- [**Improvements to prosodic alignment for automatic dubbing**](https://www.amazon.science/publications/improvements-to-prosodic-alignment-for-automatic-dubbing)\n[Yogesh Virkar](https://www.amazon.science/author/yogesh-virkar), [Marcello Federico](https://www.amazon.science/author/marcello-federico), [Robert Enyedi](https://www.amazon.science/author/robert-enyedi), [Roberto Barra-Chicote](https://www.amazon.science/author/roberto-barra-chicote)\n\n- [**Machine translation verbosity control for automatic dubbing**](https://www.amazon.science/publications/machine-translation-verbosity-control-for-automatic-dubbing)\n[Surafel Melaku Lakew](https://www.amazon.science/https:/www.amazon.science/author/surafel-melaku-lakew), [Marcello Federico](https://www.amazon.science/author/marcello-federico), [Yue Wang](https://www.amazon.science/author/yue-wang), [Cuong Hoang](https://www.amazon.science/author/cuong-hoang), [Yogesh Virkar](https://www.amazon.science/author/yogesh-virkar), [Roberto Barra-Chicote](https://www.amazon.science/author/roberto-barra-chicote), [Robert Enyedi ](https://www.amazon.science/author/robert-enyedi)\n\nAmazon’s Text-to-Speech team has an ICASSP paper on the unusual topic of **computer-assisted pronunciation training**, a feature of some language learning applications. The researchers’ method would enable language learning apps to accept a wider range of word pronunciations, to score pronunciations more accurately, and to provide more reliable feedback:\n\n- [**Mispronunciation detection in non-native (L2) English with uncertainty modeling**](https://www.amazon.science/publications/mispronunciation-detection-in-non-native-l2-english-with-uncertainty-modeling)\n[Daniel Korzekwa](https://www.amazon.science/author/daniel-korzekwa), [Jaime Lorenzo-Trueba](https://www.amazon.science/author/jaime-lorenzo-trueba), Szymon Zaporowski, [Shira Calamaro](https://www.amazon.science/author/shira-calamaro), [Thomas Drugman](https://www.amazon.science/author/thomas-drugman), Bozena Kostek \n\n\n\nThe architecture of a new Amazon model for separating a recording's vocal tracks and instrumental tracks.\n\nFROM [\\"SEMI-SUPERVISED SINGING VOICE SEPARATION WITH NOISE SELF-TRAINING\\"](https://www.amazon.science/publications/semi-supervised-singing-voice-separation-with-noise-self-training)\n\nAnother paper investigates the topic of **singing voice separation**, or separating vocal tracks from instrumental tracks in song recordings:\n\n- [**Semi-supervised singing voice separation with noise self-training**](https://www.amazon.science/publications/semi-supervised-singing-voice-separation-with-noise-self-training)\nZhepei Wang, [Ritwik Giri](https://www.amazon.science/author/ritwik-giri), [Umut Isik](https://www.amazon.science/author/umut-isik), [Jean-Marc Valin](https://www.amazon.science/author/jean-marc-valin), [Arvindh Krishnaswamy ](https://www.amazon.science/author/arvindh-krishnaswamy)\n\nFinally, two of Amazon’s ICASSP papers, although they do evaluate applications in speech recognition and audio classification, present general **machine learning methodologies** that could apply to a range of problems. One paper investigates federated learning, a distributed-learning technique in which multiple servers, each with a different, local store of training data, collectively build a machine learning model without exchanging data. The other presents a new loss function for training classification models on synthetic data created by transforming real data — for instance, training a sound classification model with samples that have noise added to them artificially.\n\n- [**Cross-silo federated training in the cloud with diversity scaling and semi-supervised learning**](https://www.amazon.science/publications/cross-silo-federated-training-in-the-cloud-with-diversity-scaling-and-semi-supervised-learning)\n[Kishore Nandury](https://www.amazon.science/author/kishore-nandury), [Anand Mohan](https://www.amazon.science/author/anand-mohan), [Frederick Weber](https://www.amazon.science/author/frederick-weber) \n- [**Enhancing audio augmentation methods with consistency learning**](https://www.amazon.science/publications/enhancing-audio-augmentation-methods-with-consistency-learning)\nTurab Iqbal, [Karim Helwani](https://www.amazon.science/author/karim-helwani), [Arvindh Krishnaswamy](https://www.amazon.science/author/arvindh-krishnaswamy), Wenwu Wang\n\nAlso at ICASSP, on June 8, seven Amazon scientists will be participating in a half-hour live Q&A. Conference registrants may [submit questions to the panelists](https://amazon.qualtrics.com/jfe/form/SV_0uf9M3PTwoL0Rfw) online.\n\nABOUT THE AUTHOR\n\n#### [Larry Hardesty](https://www.amazon.science/author/larry-hardesty)\n\nLarry Hardesty is the editor of the Amazon Science blog. Previously, he was a senior editor at MIT Technology Review and the computer science writer at the MIT News Office.\n\n\n\n\n","render":"<p>The International Conference on Acoustics, Speech, and Signal Processing (<a href=\\"https://www.amazon.science/conferences-and-events/icassp-2021\\" target=\\"_blank\\">ICASSP</a>) starts next week, and as Alexa principal research scientist Ariya Rastrow <a href=\\"https://www.amazon.science/blog/icassp-what-signal-processing-has-come-to-mean\\" target=\\"_blank\\">explained last year</a>, it casts a wide net. The topics of the 36 Amazon research papers at this year’s ICASSP range from the classic signal-processing problems of noise and echo cancellation to such far-flung problems as separating song vocals from instrumental tracks and regulating translation length.</p>\\n<p>A plurality of the papers, however, concentrate on the core technology of <strong>automatic speech recognition</strong> (ASR), or converting an acoustic speech signal into text:</p>\\n<ul>\\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/asr-n-best-fusion-nets\\" target=\\"_blank\\"><strong>ASR n-best fusion nets</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/xinyue-liu\\" target=\\"_blank\\">Xinyue Liu</a>, Mingda Li, <a href=\\"https://www.amazon.science/author/luoxin-chen\\" target=\\"_blank\\">Luoxin Chen</a>, <a href=\\"https://www.amazon.science/author/prashan-wanigasekara\\" target=\\"_blank\\">Prashan Wanigasekara</a>, <a href=\\"https://www.amazon.science/author/wael-hamza\\" target=\\"_blank\\">Weitong Ruan</a>, <a href=\\"https://www.amazon.science/author/haidar-khan\\" target=\\"_blank\\">Haidar Khan</a>, <a href=\\"https://www.amazon.science/author/wael-hamza\\" target=\\"_blank\\">Wael Hamza</a>, <a href=\\"https://www.amazon.science/author/chengwei-su\\" target=\\"_blank\\">Chengwei Su</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/bifocal-neural-asr-exploiting-keyword-spotting-for-inference-optimization\\" target=\\"_blank\\"><strong>Bifocal neural ASR: Exploiting keyword spotting for inference optimization</strong></a><br />\\nJon Macoskey, <a href=\\"https://www.amazon.science/author/grant-p-strimel\\" target=\\"_blank\\">Grant P. Strimel</a>, <a href=\\"https://www.amazon.science/author/ariya-rastrow\\" target=\\"_blank\\">Ariya Rastrow</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/domain-aware-neural-language-models-for-speech-recognition\\" target=\\"_blank\\"><strong>Domain-aware neural language models for speech recognition</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/linda-liu\\" target=\\"_blank\\">Linda Liu</a>, <a href=\\"https://www.amazon.science/author/yile-gu\\" target=\\"_blank\\">Yile Gu</a>, <a href=\\"https://www.amazon.science/author/aditya-gourav\\" target=\\"_blank\\">Aditya Gourav</a>, <a href=\\"https://www.amazon.science/author/ankur-gandhe\\" target=\\"_blank\\">Ankur Gandhe</a>, <a href=\\"https://www.amazon.science/author/shashank-kalmane\\" target=\\"_blank\\">Shashank Kalmane</a>, <a href=\\"https://www.amazon.science/author/denis-filiminov\\" target=\\"_blank\\">Denis Filimonov</a>, <a href=\\"https://www.amazon.science/author/ariya-rastrow\\" target=\\"_blank\\">Ariya Rastrow</a>, <a href=\\"https://www.amazon.science/author/ivan-bulyko\\" target=\\"_blank\\">Ivan Bulyko</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/end-to-end-multi-channel-transformer-for-speech-recognition\\" target=\\"_blank\\"><strong>End-to-end multi-channel transformer for speech recognition</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/feng-ju-chang\\" target=\\"_blank\\">Feng-Ju Chang</a>, <a href=\\"https://www.amazon.science/author/martin-radfar\\" target=\\"_blank\\">Martin Radfar</a>, <a href=\\"https://www.amazon.science/author/athanasios-mouchtaris\\" target=\\"_blank\\">Athanasios Mouchtaris</a>, <a href=\\"https://www.amazon.science/author/brian-king\\" target=\\"_blank\\">Brian King</a>, <a href=\\"https://www.amazon.science/author/siegfried-kunzmann\\" target=\\"_blank\\">Siegfried Kunzmann</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/improved-robustness-to-disfluencies-in-rnn-transducer-based-speech-recognition\\" target=\\"_blank\\"><strong>Improved robustness to disfluencies in RNN-transducer-based speech recognition</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/valentin-mendelev\\" target=\\"_blank\\">Valentin Mendelev</a>, Tina Raissi, Guglielmo Camporese, <a href=\\"https://www.amazon.science/author/manuel-giollo\\" target=\\"_blank\\">Manuel Giollo</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/personalization-strategies-for-end-to-end-speech-recognition-systems\\" target=\\"_blank\\"><strong>Personalization strategies for end-to-end speech recognition systems</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/aditya-gourav\\" target=\\"_blank\\">Aditya Gourav</a>, <a href=\\"https://www.amazon.science/author/linda-liu\\" target=\\"_blank\\">Linda Liu</a>, <a href=\\"https://www.amazon.science/author/ankur-gandhe\\" target=\\"_blank\\">Ankur Gandhe</a>, <a href=\\"https://www.amazon.science/author/yile-gu\\" target=\\"_blank\\">Yile Gu</a>, <a href=\\"https://www.amazon.science/author/guitang-lan\\" target=\\"_blank\\">Guitang Lan</a>, <a href=\\"https://www.amazon.science/author/xiangyang-huang\\" target=\\"_blank\\">Xiangyang Huang</a>, <a href=\\"https://www.amazon.science/author/shashank-kalmane\\" target=\\"_blank\\">Shashank Kalmane</a>, <a href=\\"https://www.amazon.science/author/gautam-tiwari\\" target=\\"_blank\\">Gautam Tiwari</a>, <a href=\\"https://www.amazon.science/author/denis-filiminov\\" target=\\"_blank\\">Denis Filimonov</a>, <a href=\\"https://www.amazon.science/author/ariya-rastrow\\" target=\\"_blank\\">Ariya Rastrow</a>, <a href=\\"https://www.amazon.science/author/andreas-stolcke\\" target=\\"_blank\\">Andreas Stolcke</a>, <a href=\\"https://www.amazon.science/author/ivan-bulyko\\" target=\\"_blank\\">Ivan Bulyko</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/redat-accent-invariant-representation-for-end-to-end-asr-by-domain-adversarial-training-with-relabeling\\" target=\\"_blank\\"><strong>reDAT: Accent-invariant representation for end-to-end ASR by domain adversarial training with relabeling</strong></a><br />\\nHu Hu, <a href=\\"https://www.amazon.science/author/xuesong-yang\\" target=\\"_blank\\">Xuesong Yang</a>, <a href=\\"https://www.amazon.science/author/zeynab-raeesy\\" target=\\"_blank\\">Zeynab Raeesy</a>, <a href=\\"https://www.amazon.science/author/jinxi-guo\\" target=\\"_blank\\">Jinxi Guo</a>, <a href=\\"https://www.amazon.science/author/gokce-keskin\\" target=\\"_blank\\">Gokce Keskin</a>, <a href=\\"https://www.amazon.science/author/harish-arsikere\\" target=\\"_blank\\">Harish Arsikere</a>, <a href=\\"https://www.amazon.science/author/ariya-rastrow\\" target=\\"_blank\\">Ariya Rastrow</a>, <a href=\\"https://www.amazon.science/author/andreas-stolcke\\" target=\\"_blank\\">Andreas Stolcke</a>, <a href=\\"https://www.amazon.science/author/roland-maas\\" target=\\"_blank\\">Roland Maas</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/sparsification-via-compressed-sensing-for-automatic-speech-recognition\\" target=\\"_blank\\"><strong>Sparsification via compressed sensing for automatic speech recognition</strong></a><br />\\nKai Zhen, <a href=\\"https://www.amazon.science/author/hieu-duy-nguyen\\" target=\\"_blank\\">Hieu Duy Nguyen</a>, <a href=\\"https://www.amazon.science/author/feng-ju-chang\\" target=\\"_blank\\">Feng-Ju Chang</a>, <a href=\\"https://www.amazon.science/author/athanasios-mouchtaris\\" target=\\"_blank\\">Athanasios Mouchtaris</a>, <a href=\\"https://www.amazon.science/author/ariya-rastrow\\" target=\\"_blank\\">Ariya Rastrow</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/streaming-multi-speaker-asr-with-rnn-t\\" target=\\"_blank\\"><strong>Streaming multi-speaker ASR with RNN-T</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/ilya-sklyar\\" target=\\"_blank\\">Ilya Sklyar</a>, <a href=\\"https://www.amazon.science/author/anna-piunova\\" target=\\"_blank\\">Anna Piunova</a>, <a href=\\"https://www.amazon.science/author/yulan-liu\\" target=\\"_blank\\">Yulan Liu</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/using-synthetic-audio-to-improve-the-recognition-of-out-of-vocabulary-words-in-end-to-end-asr-systems\\" target=\\"_blank\\"><strong>Using synthetic audio to improve the recognition of out-of-vocabulary words in end-to-end ASR systems</strong></a><br />\\nXianrui Zheng, <a href=\\"https://www.amazon.science/author/yulan-liu\\" target=\\"_blank\\">Yulan Liu</a>, <a href=\\"https://www.amazon.science/author/deniz-gunceler\\" target=\\"_blank\\">Deniz Gunceler</a>, <a href=\\"https://www.amazon.science/author/daniel-willett\\" target=\\"_blank\\">Daniel Willett</a></p>\\n</li>\n</ul>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/15c4b6efdfa140a28a0c4a3a8e0006e2_image.png\\" alt=\\"image.png\\" /></p>\n<p>To enable personalization of end-to-end automatic-speech-recognition systems, Linda Liu, Aditya Gourav and their colleagues use a word-level biasing finite state transducer, or FST (left). A subword-level FST preserves the weights of the word-level FST. For instance, the weight between state 0 and 5 of the subword-level FST (representing the word “player”) is (-1.6) +(- 1.6)+(-4.8) = -8.</p>\n<p>FROM <a href=\\"https://www.amazon.science/publications/personalization-strategies-for-end-to-end-speech-recognition-systems\\" target=\\"_blank\\">“PERSONALIZATION STRATEGIES FOR END-TO-END SPEECH RECOGNITION SYSTEMS”</a></p>\\n<p>Two of the papers address <strong>language (or code) switching</strong>, a more complicated version of ASR in which the speech recognizer must also determine which of several possible languages is being spoken:</p>\\n<ul>\\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/joint-asr-and-language-identification-using-rnn-t-an-efficent-approach-to-dynamic-language-switching\\" target=\\"_blank\\"><strong>Joint ASR and language identification using RNN-T: An efficent approach to dynamic language switching</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/surabhi-punjabi\\" target=\\"_blank\\">Surabhi Punjabi</a>, <a href=\\"https://www.amazon.science/author/harish-arsikere\\" target=\\"_blank\\">Harish Arsikere</a>, <a href=\\"https://www.amazon.science/author/zeynab-raeesy\\" target=\\"_blank\\">Zeynab Raeesy</a>, <a href=\\"https://www.amazon.science/author/chander-chandak\\" target=\\"_blank\\">Chander Chandak</a>, <a href=\\"https://www.amazon.science/author/nikhil-bhave\\" target=\\"_blank\\">Nikhil Bhave</a>, <a href=\\"https://www.amazon.science/author/markus-mueller\\" target=\\"_blank\\">Markus Mueller</a>, <a href=\\"https://www.amazon.science/author/sergio-murillo\\" target=\\"_blank\\">Sergio Murillo</a>, <a href=\\"https://www.amazon.science/author/ariya-rastrow\\" target=\\"_blank\\">Ariya Rastrow</a>, <a href=\\"https://www.amazon.science/author/andreas-stolcke\\" target=\\"_blank\\">Andreas Stolcke</a>, <a href=\\"https://www.amazon.science/author/jasha-droppo\\" target=\\"_blank\\">Jasha Droppo</a>, <a href=\\"https://www.amazon.science/author/sri-garimella\\" target=\\"_blank\\">Sri Garimella</a>, <a href=\\"https://www.amazon.science/author/roland-maas\\" target=\\"_blank\\">Roland Maas</a>, <a href=\\"https://www.amazon.science/author/mat-hans\\" target=\\"_blank\\">Mat Hans</a>, <a href=\\"https://www.amazon.science/author/athanasios-mouchtaris\\" target=\\"_blank\\">Athanasios Mouchtaris</a>, <a href=\\"https://www.amazon.science/author/siegfried-kunzmann\\" target=\\"_blank\\">Siegfried Kunzmann</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/transformer-transducers-for-code-switched-speech-recognition\\" target=\\"_blank\\"><strong>Transformer-transducers for code-switched speech recognition</strong></a><br />\\nSiddharth Dalmia, <a href=\\"https://www.amazon.science/author/yuzong-liu\\" target=\\"_blank\\">Yuzong Liu</a>, <a href=\\"https://www.amazon.science/author/srikanth-ronanki\\" target=\\"_blank\\">Srikanth Ronanki</a>, <a href=\\"https://www.amazon.science/author/katrin-kirchhoff\\" target=\\"_blank\\">Katrin Kirchhoff </a></p>\\n</li>\n</ul>\\n<p>The acoustic speech signal contains more information than just the speaker’s words; how the words are said can change their meaning. Such <strong>paralinguistic signals</strong> can be useful for a voice agent trying to determine how to interpret the raw text. Two of Amazon’s ICASSP papers focus on such signals:</p>\\n<ul>\\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/contrastive-unsupervised-learning-for-speech-emotion-recognition\\" target=\\"_blank\\"><strong>Contrastive unsupervised learning for speech emotion recognition</strong></a><br />\\nMao Li, <a href=\\"https://www.amazon.science/author/bo-yang\\" target=\\"_blank\\">Bo Yang</a>, <a href=\\"https://www.amazon.science/author/joshua-levy\\" target=\\"_blank\\">Joshua Levy</a>, <a href=\\"https://www.amazon.science/author/andreas-stolcke\\" target=\\"_blank\\">Andreas Stolcke</a>, <a href=\\"https://www.amazon.science/author/viktor-rozgic\\" target=\\"_blank\\">Viktor Rozgic</a>, <a href=\\"https://www.amazon.science/author/spyros-matsoukas\\" target=\\"_blank\\">Spyros Matsoukas</a>, <a href=\\"https://www.amazon.science/author/constantinos-papayiannis\\" target=\\"_blank\\">Constantinos Papayiannis</a>, <a href=\\"https://www.amazon.science/author/daniel-bone\\" target=\\"_blank\\">Daniel Bone</a>, <a href=\\"https://www.amazon.science/author/chao-wang\\" target=\\"_blank\\">Chao Wang</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/disentanglement-for-audiovisual-emotion-recognition-using-multitask-setup\\" target=\\"_blank\\"><strong>Disentanglement for audiovisual emotion recognition using multitask setup</strong></a><br />\\nRaghuveer Peri, <a href=\\"https://www.amazon.science/author/srinivas-parthasarathy\\" target=\\"_blank\\">Srinivas Parthasarathy</a>, <a href=\\"https://www.amazon.science/author/charles-bradshaw\\" target=\\"_blank\\">Charles Bradshaw</a>, <a href=\\"https://www.amazon.science/blog/null\\" target=\\"_blank\\">Shiva Sundaram</a></p>\\n</li>\n</ul>\\n<p>Several papers address other <strong>extensions of ASR</strong>, such as speaker diarization, or tracking which of several speakers issues each utterance; inverse text normalization, or converting the raw ASR output into a format useful to downstream applications; and acoustic event classification, or recognizing sounds other than human voices:</p>\\n<ul>\\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/bw-eda-eend-streaming-end-to-end-neural-speaker-diarization-for-a-variable-number-of-speakers\\" target=\\"_blank\\"><strong>BW-EDA-EEND: Streaming end-to-end neural speaker diarization for a variable number of speakers</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/eunjung-han\\" target=\\"_blank\\">Eunjung Han</a>, <a href=\\"https://www.amazon.science/author/chul-lee\\" target=\\"_blank\\">Chul Lee</a>, <a href=\\"https://www.amazon.science/author/andreas-stolcke\\" target=\\"_blank\\">Andreas Stolcke</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/neural-inverse-text-normalization\\" target=\\"_blank\\"><strong>Neural inverse text normalization</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/monica-sunkara\\" target=\\"_blank\\">Monica Sunkara</a>, <a href=\\"https://www.amazon.science/author/chaitanya-shivade\\" target=\\"_blank\\">Chaitanya Shivade</a>, <a href=\\"https://www.amazon.science/author/sravan-bodapati\\" target=\\"_blank\\">Sravan Bodapati</a>, <a href=\\"https://www.amazon.science/author/katrin-kirchhoff\\" target=\\"_blank\\">Katrin Kirchhoff</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/unsupervised-and-semi-supervised-few-shot-acoustic-event-classification\\" target=\\"_blank\\"><strong>Unsupervised and semi-supervised few-shot acoustic event classification</strong></a><br />\\nHsin-Ping Huang, <a href=\\"https://www.amazon.science/author/krishna-c-puvvada\\" target=\\"_blank\\">Krishna C. Puvvada</a>, <a href=\\"https://www.amazon.science/author/ming-sun\\" target=\\"_blank\\">Ming Sun</a>, <a href=\\"https://www.amazon.science/author/chao-wang\\" target=\\"_blank\\">Chao Wang</a></p>\\n</li>\n</ul>\\n<p><strong>Speech enhancement</strong>, or removing noise and echo from the speech signal, has been a prominent topic at ICASSP since the conference began in 1976. But more recent work on the topic — including Amazon’s two papers this year — uses deep-learning methods:</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/8a67b6ead95941bf9b209f4c3804b6ce_image.png\\" alt=\\"image.png\\" /></p>\n<p>The structure of a joint echo control and noise suppression system from Amazon. A microphone (mic) captures the output of a loudspeaker, along with noise and echo. The echo is partially cancelled by an adaptive filter (ĥf), which uses the signal to the speaker. The microphone signal then passes to a residual-echo-suppression (RES) algorithm.</p>\n<p>FROM <a href=\\"https://www.amazon.science/publications/low-complexity-real-time-joint-neural-echo-control-and-speech-enhancement-based-on-percepnet\\" target=\\"_blank\\">“LOW-COMPLEXITY, REAL-TIME JOINT NEURAL ECHO CONTROL AND SPEECH ENHANCEMENT BASED ON PERCEPNET”</a></p>\\n<ul>\\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/enhancing-into-the-codec-noise-robust-speech-coding-with-vector-quantized-autoencoders\\" target=\\"_blank\\"><strong>Enhancing into the codec: Noise robust speech coding with vector-quantized autoencoders</strong></a><br />\\nJonah Casebeer, Vinjai Vale, <a href=\\"https://www.amazon.science/author/umut-isik\\" target=\\"_blank\\">Umut Isik</a>, <a href=\\"https://www.amazon.science/author/jean-marc-valin\\" target=\\"_blank\\">Jean-Marc Valin</a>, <a href=\\"https://www.amazon.science/author/ritwik-giri\\" target=\\"_blank\\">Ritwik Giri</a>, <a href=\\"https://www.amazon.science/author/arvindh-krishnaswamy\\" target=\\"_blank\\">Arvindh Krishnaswamy</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/low-complexity-real-time-joint-neural-echo-control-and-speech-enhancement-based-on-percepnet\\" target=\\"_blank\\"><strong>Low-complexity, real-time joint neural echo control and speech enhancement based on Percepnet</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/jean-marc-valin\\" target=\\"_blank\\">Jean-Marc Valin</a>, <a href=\\"https://www.amazon.science/author/srikanth-v-tenneti\\" target=\\"_blank\\">Srikanth V. Tenneti</a>, <a href=\\"https://www.amazon.science/author/karim-helwani\\" target=\\"_blank\\">Karim Helwani</a>, <a href=\\"https://www.amazon.science/author/umut-isik\\" target=\\"_blank\\">Umut Isik</a>, <a href=\\"https://www.amazon.science/author/arvindh-krishnaswamy\\" target=\\"_blank\\">Arvindh Krishnaswamy</a></p>\\n</li>\n</ul>\\n<p>Every interaction with Alexa begins with a wake word — usually “Alexa”, but sometimes “computer” or “Echo”. So at ICASSP, Amazon usually presents work on wake word detection — or <strong>keyword spotting</strong>, as it’s more generally known:</p>\\n<ul>\\n<li><a href=\\"https://www.amazon.science/publications/exploring-the-application-of-synthetic-audio-in-training-keyword-spotters\\" target=\\"_blank\\"><strong>Exploring the application of synthetic audio in training keyword spotters</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/andrew-werchniak\\" target=\\"_blank\\">Andrew Werchniak</a>, <a href=\\"https://www.amazon.science/author/roberto-barra-chicote\\" target=\\"_blank\\">Roberto Barra-Chicote</a>, <a href=\\"https://www.amazon.science/author/yuriy-mishchenko\\" target=\\"_blank\\">Yuriy Mishchenko</a>, <a href=\\"https://www.amazon.science/author/jasha-droppo\\" target=\\"_blank\\">Jasha Droppo</a>, <a href=\\"https://www.amazon.science/author/jeff-condal\\" target=\\"_blank\\">Jeff Condal</a>, <a href=\\"https://www.amazon.science/author/peng-liu\\" target=\\"_blank\\">Peng Liu</a>, <a href=\\"https://www.amazon.science/author/anish-shah\\" target=\\"_blank\\">Anish Shah </a></li>\\n</ul>\n<p>In many spoken-language systems, the next step after ASR <strong>is natural-language understanding</strong> (NLU), or making sense of the text output from the ASR system:</p>\\n<p>-<a href=\\"https://www.amazon.science/publications/introducing-deep-reinforcement-learning-to-nlu-ranking-tasks\\" target=\\"_blank\\"> <strong>Introducing deep reinforcement learning to NLU ranking tasks</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/ge-yu\\" target=\\"_blank\\">Ge Yu</a>, <a href=\\"https://www.amazon.science/author/chengwei-su\\" target=\\"_blank\\">Chengwei Su</a>, <a href=\\"https://www.amazon.science/author/emre-barut\\" target=\\"_blank\\">Emre Barut</a></p>\\n<ul>\\n<li><a href=\\"https://www.amazon.science/publications/language-model-is-all-you-need-natural-language-understanding-as-question-answering\\" target=\\"_blank\\"><strong>Language model is all you need: Natural language understanding as question answering</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/mahdi-namazifar\\" target=\\"_blank\\">Mahdi Namazifar</a>, <a href=\\"https://www.amazon.science/author/alexandros-papangelis\\" target=\\"_blank\\">Alexandros Papangelis</a>, <a href=\\"https://www.amazon.science/author/gokhan-tur\\" target=\\"_blank\\">Gokhan Tur</a>, <a href=\\"https://www.amazon.science/author/dilek-hakkani-tur\\" target=\\"_blank\\">Dilek Hakkani-Tür </a></li>\\n</ul>\n<p>In some contexts, however, it’s possible to perform both ASR and NLU with a single model, in a task known as <strong>spoken-language understanding:</strong></p>\\n<ul>\\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/do-as-i-mean-not-as-i-say-sequence-loss-training-for-spoken-language-understanding\\" target=\\"_blank\\"><strong>Do as I mean, not as I say: Sequence loss training for spoken language understanding</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/milind-rao\\" target=\\"_blank\\">Milind Rao</a>, <a href=\\"https://www.amazon.science/author/pranav-dheram\\" target=\\"_blank\\">Pranav Dheram</a>, <a href=\\"https://www.amazon.science/author/gautam-tiwari\\" target=\\"_blank\\">Gautam Tiwari</a>, <a href=\\"https://www.amazon.science/author/anirudh-raju\\" target=\\"_blank\\">Anirudh Raju</a>, <a href=\\"https://www.amazon.science/author/jasha-droppo\\" target=\\"_blank\\">Jasha Droppo</a>, <a href=\\"https://www.amazon.science/author/ariya-rastrow\\" target=\\"_blank\\">Ariya Rastrow</a>, <a href=\\"https://www.amazon.science/author/andreas-stolcke\\" target=\\"_blank\\">Andreas Stolcke </a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/graph-enhanced-query-rewriting-for-spoken-language-understanding-system\\" target=\\"_blank\\"><strong>Graph enhanced query rewriting for spoken language understanding system</strong></a><br />\\nSiyang Yuan, <a href=\\"https://www.amazon.science/author/saurabh-gupta\\" target=\\"_blank\\">Saurabh Gupta</a>, <a href=\\"https://www.amazon.science/author/xing-fan\\" target=\\"_blank\\">Xing Fan</a>, <a href=\\"https://www.amazon.science/author/derek-liu\\" target=\\"_blank\\">Derek Liu</a>, <a href=\\"https://www.amazon.science/author/yang-liu\\" target=\\"_blank\\">Yang Liu</a>, C<a href=\\"https://www.amazon.science/author/chenlei-guo\\" target=\\"_blank\\">henlei (Edward) Guo</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/top-down-attention-in-end-to-end-spoken-language-understanding\\" target=\\"_blank\\"><strong>Top-down attention in end-to-end spoken language understanding</strong></a><br />\\nYixin Chen, <a href=\\"https://www.amazon.science/author/weiyi-lu\\" target=\\"_blank\\">Weiyi Lu</a>, <a href=\\"https://www.amazon.science/author/alejandro-mottini\\" target=\\"_blank\\">Alejandro Mottini </a>, <a href=\\"https://www.amazon.science/author/erran-li\\" target=\\"_blank\\">Erran Li</a>, <a href=\\"https://www.amazon.science/author/jasha-droppo\\" target=\\"_blank\\">Jasha Droppo</a>, <a href=\\"https://www.amazon.science/author/zheng-du\\" target=\\"_blank\\">Zheng Du</a>, <a href=\\"https://www.amazon.science/author/belinda-zeng\\" target=\\"_blank\\">Belinda Zeng</a></p>\\n</li>\n</ul>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/f753621dbe374383bf89768b087db820_image.png\\" alt=\\"image.png\\" /></p>\n<p>A spoken-language-understanding system combines automatic speech recognition (ASR) and natural-language understanding (NLU) in a single model.</p>\n<p>FROM <a href=\\"https://www.amazon.science/publications/do-as-i-mean-not-as-i-say-sequence-loss-training-for-spoken-language-understanding\\" target=\\"_blank\\">“DO AS I MEAN, NOT AS I SAY: SEQUENCE LOSS TRAINING FOR SPOKEN LANGUAGE UNDERSTANDING”</a></p>\\n<p>An interaction with a voice service, which begins with keyword spotting, ASR, and NLU, often culminates with the agent’s use of synthesized speech to relay a response. The agent’s <strong>text-to-speech</strong> model converts the textual outputs of various NLU and dialogue systems into speech:</p>\\n<ul>\\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/camp-a-two-stage-approach-to-modelling-prosody-in-context\\" target=\\"_blank\\"><strong>CAMP: A two-stage approach to modelling prosody in context</strong></a><br />\\nZack Hodari, <a href=\\"https://www.amazon.science/author/alexis-moinet\\" target=\\"_blank\\">Alexis Moinet</a>, <a href=\\"https://www.amazon.science/author/sri-karlapati\\" target=\\"_blank\\">Sri Karlapati</a>, <a href=\\"https://www.amazon.science/author/jaime-lorenzo-trueba\\" target=\\"_blank\\">Jaime Lorenzo-Trueba</a>, <a href=\\"https://www.amazon.science/author/thomas-merritt\\" target=\\"_blank\\">Thomas Merritt</a>, <a href=\\"https://www.amazon.science/author/arnaud-joly\\" target=\\"_blank\\">Arnaud Joly</a>, <a href=\\"https://www.amazon.science/author/ammar-abbas\\" target=\\"_blank\\">Ammar Abbas</a>, <a href=\\"https://www.amazon.science/author/penny-karanasou\\" target=\\"_blank\\">Penny Karanasou</a>, <a href=\\"https://www.amazon.science/author/thomas-drugman\\" target=\\"_blank\\">Thomas Drugman</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/low-resource-expressive-text-to-speech-using-data-augmentation\\" target=\\"_blank\\"><strong>Low-resource expressive text-to-speech using data augmentation</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/goeric-huybrechts\\" target=\\"_blank\\">Goeric Huybrechts</a>, <a href=\\"https://www.amazon.science/author/thomas-merritt\\" target=\\"_blank\\">Thomas Merritt</a>, <a href=\\"https://www.amazon.science/author/giulia-comini\\" target=\\"_blank\\">Giulia Comini</a>, <a href=\\"https://www.amazon.science/author/bartek-perz\\" target=\\"_blank\\">Bartek Perz</a>, <a href=\\"https://www.amazon.science/author/raahil-shah\\" target=\\"_blank\\">Raahil Shah</a>, <a href=\\"https://www.amazon.science/author/jaime-lorenzo-trueba\\" target=\\"_blank\\">Jaime Lorenzo-Trueba</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/prosodic-representation-learning-and-contextual-sampling-for-neural-text-to-speech\\" target=\\"_blank\\"><strong>Prosodic representation learning and contextual sampling for neural text-to-speech</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/sri-karlapati\\" target=\\"_blank\\">Sri Karlapati</a>, <a href=\\"\\" target=\\"_blank\\">Ammar Abbas</a>, Zack Hodari], <a href=\\"https://www.amazon.science/author/alexis-moinet\\" target=\\"_blank\\">Alexis Moinet</a>, <a href=\\"https://www.amazon.science/author/arnaud-joly\\" target=\\"_blank\\">Arnaud Joly</a>, <a href=\\"https://www.amazon.science/author/penny-karanasou\\" target=\\"_blank\\">Penny Karanasou</a>, <a href=\\"https://www.amazon.science/author/thomas-drugman\\" target=\\"_blank\\">Thomas Drugman</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/universal-neural-vocoding-with-parallel-wavenet\\" target=\\"_blank\\"><strong>Universal neural vocoding with Parallel WaveNet</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/yunlong-jiao\\" target=\\"_blank\\">Yunlong Jiao</a>, <a href=\\"https://www.amazon.science/author/adam-gabrys\\" target=\\"_blank\\">Adam Gabrys</a>, Georgi Tinchev, <a href=\\"https://www.amazon.science/author/bartosz-putrycz\\" target=\\"_blank\\">Bartosz Putrycz</a>, <a href=\\"https://www.amazon.science/author/daniel-korzekwa\\" target=\\"_blank\\">Daniel Korzekwa</a>, <a href=\\"https://www.amazon.science/author/viacheslav-klimkov\\" target=\\"_blank\\">Viacheslav Klimkov</a></p>\\n</li>\n</ul>\\n<p>All of the preceding research topics have implications for voice services like Alexa, but Amazon has a range of other products and services that rely on audio-signal processing. Three of Amazon’s papers at this year’s ICASSP relate to <strong>audio-video synchronization</strong>: two deal with dubbing audio in one language onto video shot in another, and one describes how to detect synchronization errors in video — as when, for example, the sound of a tennis ball being struck and the shot of the racquet hitting the ball are misaligned:</p>\\n<ul>\\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/detection-of-audio-video-synchronization-errors-via-event-detection\\" target=\\"_blank\\"><strong>Detection of audio-video synchronization errors via event detection</strong></a><br />\\nJoshua P. Ebenezer, <a href=\\"https://www.amazon.science/author/yongjun-wu\\" target=\\"_blank\\">Yongjun Wu</a>, <a href=\\"https://www.amazon.science/author/hai-wei\\" target=\\"_blank\\">Hai Wei</a>, <a href=\\"https://www.amazon.science/author/sriram-sethuraman\\" target=\\"_blank\\">Sriram Sethuraman</a>, <a href=\\"https://www.amazon.science/author/zongyi-liu\\" target=\\"_blank\\">Zongyi Liu </a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/improvements-to-prosodic-alignment-for-automatic-dubbing\\" target=\\"_blank\\"><strong>Improvements to prosodic alignment for automatic dubbing</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/yogesh-virkar\\" target=\\"_blank\\">Yogesh Virkar</a>, <a href=\\"https://www.amazon.science/author/marcello-federico\\" target=\\"_blank\\">Marcello Federico</a>, <a href=\\"https://www.amazon.science/author/robert-enyedi\\" target=\\"_blank\\">Robert Enyedi</a>, <a href=\\"https://www.amazon.science/author/roberto-barra-chicote\\" target=\\"_blank\\">Roberto Barra-Chicote</a></p>\\n</li>\n<li>\\n<p><a href=\\"https://www.amazon.science/publications/machine-translation-verbosity-control-for-automatic-dubbing\\" target=\\"_blank\\"><strong>Machine translation verbosity control for automatic dubbing</strong></a><br />\\n<a href=\\"https://www.amazon.science/https:/www.amazon.science/author/surafel-melaku-lakew\\" target=\\"_blank\\">Surafel Melaku Lakew</a>, <a href=\\"https://www.amazon.science/author/marcello-federico\\" target=\\"_blank\\">Marcello Federico</a>, <a href=\\"https://www.amazon.science/author/yue-wang\\" target=\\"_blank\\">Yue Wang</a>, <a href=\\"https://www.amazon.science/author/cuong-hoang\\" target=\\"_blank\\">Cuong Hoang</a>, <a href=\\"https://www.amazon.science/author/yogesh-virkar\\" target=\\"_blank\\">Yogesh Virkar</a>, <a href=\\"https://www.amazon.science/author/roberto-barra-chicote\\" target=\\"_blank\\">Roberto Barra-Chicote</a>, <a href=\\"https://www.amazon.science/author/robert-enyedi\\" target=\\"_blank\\">Robert Enyedi </a></p>\\n</li>\n</ul>\\n<p>Amazon’s Text-to-Speech team has an ICASSP paper on the unusual topic of <strong>computer-assisted pronunciation training</strong>, a feature of some language learning applications. The researchers’ method would enable language learning apps to accept a wider range of word pronunciations, to score pronunciations more accurately, and to provide more reliable feedback:</p>\\n<ul>\\n<li><a href=\\"https://www.amazon.science/publications/mispronunciation-detection-in-non-native-l2-english-with-uncertainty-modeling\\" target=\\"_blank\\"><strong>Mispronunciation detection in non-native (L2) English with uncertainty modeling</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/daniel-korzekwa\\" target=\\"_blank\\">Daniel Korzekwa</a>, <a href=\\"https://www.amazon.science/author/jaime-lorenzo-trueba\\" target=\\"_blank\\">Jaime Lorenzo-Trueba</a>, Szymon Zaporowski, <a href=\\"https://www.amazon.science/author/shira-calamaro\\" target=\\"_blank\\">Shira Calamaro</a>, <a href=\\"https://www.amazon.science/author/thomas-drugman\\" target=\\"_blank\\">Thomas Drugman</a>, Bozena Kostek</li>\\n</ul>\n<p><img src=\\"https://dev-media.amazoncloud.cn/8c27775d555241e6a158f61419d36e54_image.png\\" alt=\\"image.png\\" /></p>\n<p>The architecture of a new Amazon model for separating a recording’s vocal tracks and instrumental tracks.</p>\n<p>FROM <a href=\\"https://www.amazon.science/publications/semi-supervised-singing-voice-separation-with-noise-self-training\\" target=\\"_blank\\">“SEMI-SUPERVISED SINGING VOICE SEPARATION WITH NOISE SELF-TRAINING”</a></p>\\n<p>Another paper investigates the topic of <strong>singing voice separation</strong>, or separating vocal tracks from instrumental tracks in song recordings:</p>\\n<ul>\\n<li><a href=\\"https://www.amazon.science/publications/semi-supervised-singing-voice-separation-with-noise-self-training\\" target=\\"_blank\\"><strong>Semi-supervised singing voice separation with noise self-training</strong></a><br />\\nZhepei Wang, <a href=\\"https://www.amazon.science/author/ritwik-giri\\" target=\\"_blank\\">Ritwik Giri</a>, <a href=\\"https://www.amazon.science/author/umut-isik\\" target=\\"_blank\\">Umut Isik</a>, <a href=\\"https://www.amazon.science/author/jean-marc-valin\\" target=\\"_blank\\">Jean-Marc Valin</a>, <a href=\\"https://www.amazon.science/author/arvindh-krishnaswamy\\" target=\\"_blank\\">Arvindh Krishnaswamy </a></li>\\n</ul>\n<p>Finally, two of Amazon’s ICASSP papers, although they do evaluate applications in speech recognition and audio classification, present general <strong>machine learning methodologies</strong> that could apply to a range of problems. One paper investigates federated learning, a distributed-learning technique in which multiple servers, each with a different, local store of training data, collectively build a machine learning model without exchanging data. The other presents a new loss function for training classification models on synthetic data created by transforming real data — for instance, training a sound classification model with samples that have noise added to them artificially.</p>\\n<ul>\\n<li><a href=\\"https://www.amazon.science/publications/cross-silo-federated-training-in-the-cloud-with-diversity-scaling-and-semi-supervised-learning\\" target=\\"_blank\\"><strong>Cross-silo federated training in the cloud with diversity scaling and semi-supervised learning</strong></a><br />\\n<a href=\\"https://www.amazon.science/author/kishore-nandury\\" target=\\"_blank\\">Kishore Nandury</a>, <a href=\\"https://www.amazon.science/author/anand-mohan\\" target=\\"_blank\\">Anand Mohan</a>, <a href=\\"https://www.amazon.science/author/frederick-weber\\" target=\\"_blank\\">Frederick Weber</a></li>\\n<li><a href=\\"https://www.amazon.science/publications/enhancing-audio-augmentation-methods-with-consistency-learning\\" target=\\"_blank\\"><strong>Enhancing audio augmentation methods with consistency learning</strong></a><br />\\nTurab Iqbal, <a href=\\"https://www.amazon.science/author/karim-helwani\\" target=\\"_blank\\">Karim Helwani</a>, <a href=\\"https://www.amazon.science/author/arvindh-krishnaswamy\\" target=\\"_blank\\">Arvindh Krishnaswamy</a>, Wenwu Wang</li>\\n</ul>\n<p>Also at ICASSP, on June 8, seven Amazon scientists will be participating in a half-hour live Q&amp;A. Conference registrants may <a href=\\"https://amazon.qualtrics.com/jfe/form/SV_0uf9M3PTwoL0Rfw\\" target=\\"_blank\\">submit questions to the panelists</a> online.</p>\\n<p>ABOUT THE AUTHOR</p>\n<h4><a id=\\"Larry_Hardestyhttpswwwamazonscienceauthorlarryhardesty_162\\"></a><a href=\\"https://www.amazon.science/author/larry-hardesty\\" target=\\"_blank\\">Larry Hardesty</a></h4>\\n<p>Larry Hardesty is the editor of the Amazon Science blog. Previously, he was a senior editor at MIT Technology Review and the computer science writer at the MIT News Office.</p>\n"}