Alexa AI’s natural-language-understanding papers at ICASSP 2022

{"value":"The International Conference on Acoustics, Speech, and Signal Processing (++[ICASSP](https://www.amazon.science/conferences-and-events/icassp-2022)++), which wrapped up late last month, focuses, as its name suggests, on applications close to the acoustic speech signal, such as automatic speech recognition and text-to-speech.\n\nBut in recent years, the line between speech processing and natural-language understanding (++[NLU](https://www.amazon.science/tag/nlu)++) — which focuses on texts’ semantic content — has grown fuzzier, and Alexa AI scientists had several papers on NLU at ICASSP.\n\nAmong the most common NLU tasks are domain classification, or determining the topic of an utterance, and intent classification, or determining the speaker’s goals. Usually, NLU models are trained on data labeled according to both domain and intent.\n\nBut in “++[ADVIN: Automatically discovering novel domains and intents from user text utterances](https://www.amazon.science/publications/advin-automatically-discovering-novel-domains-and-intents-from-user-text-utterances)++”, Alexa AI researchers present a new method for automatically identifying and categorizing domains and intents that an NLU model has never seen before. In the researchers’ experiments, it significantly outperformed its predecessors.\n\nIn many contexts, NLU can be improved through personalization. If two different customers tell a smart device “Play funny videos”, for instance, they may have very different types of content in mind.\n\nPersonalization based on interaction histories is well studied, but in the real world, interaction histories are constantly being updated, revealing new aspects of a customer’s taste or, indeed, changes of taste. In “++[Incremental user embedding modeling for personalized text classification](https://www.amazon.science/publications/incremental-user-embedding-modeling-for-personalized-text-classification)++”, Alexa AI researchers present a new approach to dynamically updating personalization models to reflect recent transactions. In tests on two different datasets, the approach improved prediction accuracy by 9% and 30%, respectively, versus the state of the art.\n\nA third Alexa AI paper, “++[Contrastive knowledge graph attention network for request-based recipe recommendation](https://www.amazon.science/publications/contrastive-knowledge-graph-attention-network-for-request-based-recipe-recommendation)++”, narrows in on the very particular problem of matching online recipes to customer requests. The problem with conventional machine learning approaches to recipe retrieval is that data on customer interactions with recipes is noisy and sparse.\n\nThe Alexa AI researchers use graphs to add structure to the data and contrastive learning to improve the resulting model’s reliability, improving on the state of the art by 5% to 7% on two different metrics and two different datasets.\n\nChristophe Dupuy, an applied scientist in Alexa AI, described two other ICASSP papers that he coauthored, on ++[privacy-protecting machine learning and federated learning](https://www.amazon.science/blog/advances-in-trustworthy-machine-learning-at-alexa-ai)++, in a blog post we published before the conference.\n\n#### **Intent discovery**\n\nWith voice agents like Alexa, domains are general high-level content categories, like Music or Weather, and intents are specific functions, like PlayMusic or GetTemperature.\n\nIn “ADVIN: Automatically discovering novel domains and intents from user text utterances”, the Alexa AI researchers tackle the problem of classifying previously unseen domains and intents in three stages.\n\nIn the first stage, a model simply recognizes that a dataset contains unfamiliar intents. This model is trained on labeled data for known intents and publicly available, labeled out-of-domain utterances, as a proxy for unlabeled data with unknown intents.\n\n\n\nThe two stages of the ADVIN intent discovery model: identification of unseen intents (left) and intent clustering (right).\n\nIn the second stage, another model clusters both the labeled and unlabeled utterances, based on their semantic content. From the clusters of labeled intents, the researchers derive a threshold distance value that maximizes the model’s ability to distinguish intents. Then they apply that value to the unlabeled data, to identify clusters corresponding to new intents.\n\nFinally, in the third stage, they repeat this process, but at a higher level of generality, clustering intents discovered in the previous stage into domains.\n\n#### **Dynamic personalization**\n\nEvery interaction between a customer and an online service generates new data that could be used to update a profile that encodes the customer’s preferences, but it would be highly impractical to update the profile after each one of those interactions.\n\nIn “Incremental user embedding modeling for personalized text classification”, Alexa researchers instead propose keeping a running record of a customer’s most recent interactions and using that to update the customer’s preference in a dynamic way.\n\nThey present a machine learning model that takes as input the request that the NLU model is currently trying to resolve and representations of the customer’s long-term history and short-term history. An attention mechanism determines which aspects of the short-term history are most informative in light of the long-term history and vice versa.\n\nThe output of the attention mechanism is an ad hoc customer profile that the model can use to process the current request.\n\n\n\nThe architecture of the dynamic-personalization model.\n\n#### **Recipe retrieval**\n\n“Contrastive knowledge graph attention network for request-based recipe recommendation” also addresses the question of personalization, although in the specific context of recipe recommendation — deciding which recipes to return, for instance, when the customer says, “Show me recipes for chicken breasts.”\n\nCustomers interact with recipe recommendation services in many different ways, such as browsing through recipes or checking ingredient lists. The most telling interaction, however — the one that proves that the recommended recipe met the customer’s needs — is a “cook along” service, that steps through the recipe to guide meal preparation.\n\nCook-along interactions are relatively rare, however, and the other types of interactions can be extremely noisy, reflecting stray clicks, misinterpretations of recipe titles, and the like. Building a reliable recipe recommendation service requires maximizing the high-value interaction data and filtering out the noise.\n\nGraph-based models are a good way to do both, since they explicitly encode patterns in the data that would otherwise have to be inferred from training examples. The researchers begin by building a recipe graph, in which each node is a recipe, and recipes share edges if they belong to the same cuisine type, share ingredients, include related keywords, and so on.\n\nNext, they add nodes representing customers to the graph. Edges between customer nodes and recipes indicate that customers have interacted with those recipes, and they also encode the types of interactions.\n\n\n\nThe framework for training the recipe recommendation model. Nodes of the knowledge graph represents users, recipes, and attributes (u, i, and a, respectively). Graph augmentation (GA) produces synthetic positive examples, which are used to train knowledge graph attention (KGAT) networks.\n\nFinally, they train a model to create representations of the graph nodes using contrastive learning, in which the model is trained on pairs of examples, one that belongs to the target class — say, recipes that a particular customer has interacted with — and one that doesn’t. The model learns to produce representations that push contrasting examples far apart from each other in the representation space and pull related examples together.\n\nTo produce related examples, the researchers create synthetic variations on the real examples, in which nodes or edges have been randomly dropped. This trains the model to focus on essential features of the data and ignore inessential features, so it generalizes better.\n\nABOUT THE AUTHOR\n\n#### **[Larry Hardesty](https://www.amazon.science/author/larry-hardesty)**\n\nLarry Hardesty is the editor of the Amazon Science blog. Previously, he was a senior editor at MIT Technology Review and the computer science writer at the MIT News Office.","render":"<p>The International Conference on Acoustics, Speech, and Signal Processing (<ins><a href=\"https://www.amazon.science/conferences-and-events/icassp-2022\" target=\"_blank\">ICASSP</a></ins>), which wrapped up late last month, focuses, as its name suggests, on applications close to the acoustic speech signal, such as automatic speech recognition and text-to-speech.</p>\n<p>But in recent years, the line between speech processing and natural-language understanding (<ins><a href=\"https://www.amazon.science/tag/nlu\" target=\"_blank\">NLU</a></ins>) — which focuses on texts’ semantic content — has grown fuzzier, and Alexa AI scientists had several papers on NLU at ICASSP.</p>\n<p>Among the most common NLU tasks are domain classification, or determining the topic of an utterance, and intent classification, or determining the speaker’s goals. Usually, NLU models are trained on data labeled according to both domain and intent.</p>\n<p>But in “<ins><a href=\"https://www.amazon.science/publications/advin-automatically-discovering-novel-domains-and-intents-from-user-text-utterances\" target=\"_blank\">ADVIN: Automatically discovering novel domains and intents from user text utterances</a></ins>”, Alexa AI researchers present a new method for automatically identifying and categorizing domains and intents that an NLU model has never seen before. In the researchers’ experiments, it significantly outperformed its predecessors.</p>\n<p>In many contexts, NLU can be improved through personalization. If two different customers tell a smart device “Play funny videos”, for instance, they may have very different types of content in mind.</p>\n<p>Personalization based on interaction histories is well studied, but in the real world, interaction histories are constantly being updated, revealing new aspects of a customer’s taste or, indeed, changes of taste. In “<ins><a href=\"https://www.amazon.science/publications/incremental-user-embedding-modeling-for-personalized-text-classification\" target=\"_blank\">Incremental user embedding modeling for personalized text classification</a></ins>”, Alexa AI researchers present a new approach to dynamically updating personalization models to reflect recent transactions. In tests on two different datasets, the approach improved prediction accuracy by 9% and 30%, respectively, versus the state of the art.</p>\n<p>A third Alexa AI paper, “<ins><a href=\"https://www.amazon.science/publications/contrastive-knowledge-graph-attention-network-for-request-based-recipe-recommendation\" target=\"_blank\">Contrastive knowledge graph attention network for request-based recipe recommendation</a></ins>”, narrows in on the very particular problem of matching online recipes to customer requests. The problem with conventional machine learning approaches to recipe retrieval is that data on customer interactions with recipes is noisy and sparse.</p>\n<p>The Alexa AI researchers use graphs to add structure to the data and contrastive learning to improve the resulting model’s reliability, improving on the state of the art by 5% to 7% on two different metrics and two different datasets.</p>\n<p>Christophe Dupuy, an applied scientist in Alexa AI, described two other ICASSP papers that he coauthored, on <ins><a href=\"https://www.amazon.science/blog/advances-in-trustworthy-machine-learning-at-alexa-ai\" target=\"_blank\">privacy-protecting machine learning and federated learning</a></ins>, in a blog post we published before the conference.</p>\n<h4><a id=\"Intent_discovery_18\"></a><strong>Intent discovery</strong></h4>\n<p>With voice agents like Alexa, domains are general high-level content categories, like Music or Weather, and intents are specific functions, like PlayMusic or GetTemperature.</p>\n<p>In “ADVIN: Automatically discovering novel domains and intents from user text utterances”, the Alexa AI researchers tackle the problem of classifying previously unseen domains and intents in three stages.</p>\n<p>In the first stage, a model simply recognizes that a dataset contains unfamiliar intents. This model is trained on labeled data for known intents and publicly available, labeled out-of-domain utterances, as a proxy for unlabeled data with unknown intents.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/6c195dd8e85e4fc4b1eeb73c7f370ca0_%E4%B8%8B%E8%BD%BD.jpg\" alt=\"下载.jpg\" /></p>\n<p>The two stages of the ADVIN intent discovery model: identification of unseen intents (left) and intent clustering (right).</p>\n<p>In the second stage, another model clusters both the labeled and unlabeled utterances, based on their semantic content. From the clusters of labeled intents, the researchers derive a threshold distance value that maximizes the model’s ability to distinguish intents. Then they apply that value to the unlabeled data, to identify clusters corresponding to new intents.</p>\n<p>Finally, in the third stage, they repeat this process, but at a higher level of generality, clustering intents discovered in the previous stage into domains.</p>\n<h4><a id=\"Dynamic_personalization_34\"></a><strong>Dynamic personalization</strong></h4>\n<p>Every interaction between a customer and an online service generates new data that could be used to update a profile that encodes the customer’s preferences, but it would be highly impractical to update the profile after each one of those interactions.</p>\n<p>In “Incremental user embedding modeling for personalized text classification”, Alexa researchers instead propose keeping a running record of a customer’s most recent interactions and using that to update the customer’s preference in a dynamic way.</p>\n<p>They present a machine learning model that takes as input the request that the NLU model is currently trying to resolve and representations of the customer’s long-term history and short-term history. An attention mechanism determines which aspects of the short-term history are most informative in light of the long-term history and vice versa.</p>\n<p>The output of the attention mechanism is an ad hoc customer profile that the model can use to process the current request.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/a73c0164d0ac440f9d5ed3e5c2e4d406_%E4%B8%8B%E8%BD%BD1.jpg\" alt=\"下载1.jpg\" /></p>\n<p>The architecture of the dynamic-personalization model.</p>\n<h4><a id=\"Recipe_retrieval_48\"></a><strong>Recipe retrieval</strong></h4>\n<p>“Contrastive knowledge graph attention network for request-based recipe recommendation” also addresses the question of personalization, although in the specific context of recipe recommendation — deciding which recipes to return, for instance, when the customer says, “Show me recipes for chicken breasts.”</p>\n<p>Customers interact with recipe recommendation services in many different ways, such as browsing through recipes or checking ingredient lists. The most telling interaction, however — the one that proves that the recommended recipe met the customer’s needs — is a “cook along” service, that steps through the recipe to guide meal preparation.</p>\n<p>Cook-along interactions are relatively rare, however, and the other types of interactions can be extremely noisy, reflecting stray clicks, misinterpretations of recipe titles, and the like. Building a reliable recipe recommendation service requires maximizing the high-value interaction data and filtering out the noise.</p>\n<p>Graph-based models are a good way to do both, since they explicitly encode patterns in the data that would otherwise have to be inferred from training examples. The researchers begin by building a recipe graph, in which each node is a recipe, and recipes share edges if they belong to the same cuisine type, share ingredients, include related keywords, and so on.</p>\n<p>Next, they add nodes representing customers to the graph. Edges between customer nodes and recipes indicate that customers have interacted with those recipes, and they also encode the types of interactions.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/ece5f2219e8047c7a3c9c2e7c4d112cc_%E4%B8%8B%E8%BD%BD2.jpg\" alt=\"下载2.jpg\" /></p>\n<p>The framework for training the recipe recommendation model. Nodes of the knowledge graph represents users, recipes, and attributes (u, i, and a, respectively). Graph augmentation (GA) produces synthetic positive examples, which are used to train knowledge graph attention (KGAT) networks.</p>\n<p>Finally, they train a model to create representations of the graph nodes using contrastive learning, in which the model is trained on pairs of examples, one that belongs to the target class — say, recipes that a particular customer has interacted with — and one that doesn’t. The model learns to produce representations that push contrasting examples far apart from each other in the representation space and pull related examples together.</p>\n<p>To produce related examples, the researchers create synthetic variations on the real examples, in which nodes or edges have been randomly dropped. This trains the model to focus on essential features of the data and ignore inessential features, so it generalizes better.</p>\n<p>ABOUT THE AUTHOR</p>\n<h4><a id=\"Larry_Hardestyhttpswwwamazonscienceauthorlarryhardesty_70\"></a><strong><a href=\"https://www.amazon.science/author/larry-hardesty\" target=\"_blank\">Larry Hardesty</a></strong></h4>\n<p>Larry Hardesty is the editor of the Amazon Science blog. Previously, he was a senior editor at MIT Technology Review and the computer science writer at the MIT News Office.</p>\n"}