How silicon innovation became the ‘secret sauce’ behind Amazon’s success

{"value":"[Nafea Bshara](https://www.linkedin.com/in/nafeabshara/), Amazon Web Services vice president and distinguished engineer, and the co-founder of Annapurna Labs, an Israeli-based chipmaker that Amazon acquired in 2015, maintains a low profile, as does his friend and Annapurna co-founder, [Hrvoye (Billy) Bilic](https://www.linkedin.com/in/hrvoye-bilic-a088644/).\n\n\n\nNafea Bshara, AWS vice president and distinguished engineer.\n\nEach executive’s LinkedIn profile is sparse, in fact, Bilic’s is out of date.\n\n“We hardly do any interviews; our philosophy is to let our products do the talking,” explains Bshara.\n\nThose products, and silicon innovations, have done a lot of talking since 2015, as the acquisition has led to, among other advancements, the development of five generations of the [AWS Nitro System](https://aws.amazon.com/ec2/nitro/), three generations ([1](https://aws.amazon.com/blogs/aws/new-ec2-instances-a1-powered-by-arm-based-aws-graviton-processors/), [2](https://aws.amazon.com/blogs/aws/coming-soon-graviton2-powered-general-purpose-compute-optimized-memory-optimized-ec2-instances/), [3](https://aws.amazon.com/blogs/aws/join-the-preview-amazon-ec2-c7g-instances-powered-by-new-aws-graviton3-processors/)) of custom-designed, [Arm](https://www.arm.com/)-based [Graviton](https://aws.amazon.com/ec2/graviton/) processors that support data-intensive workloads, as well as [AWS Trainium](https://aws.amazon.com/machine-learning/trainium/), and [AWS Inferentia](https://aws.amazon.com/machine-learning/inferentia/) chips optimized for machine learning training and inference.\n\nSome observers have described the silicon that emerges from Annapurna Labs in the U.S. and Israel as AWS’s “secret sauce”.\n\nNafea’s silicon journey began at [Technion University](https://www.technion.ac.il/en/technion-vision/) in Israel, where he earned bachelor’s and master’s degrees in computer engineering, and where he first met Hrvoye. The two then went on to work for Israel-based Galileo, a company that made chips for networking switches, and controllers for networking routers. Galileo was acquired by U.S. [semiconductor manufacturer Marvell](https://www.wsj.com/articles/SB971744654175712989) in 2000, where Bshara and Bilic would work for a decade before deciding to venture out on their own.\n\n“We had developed at least 50 different chips together,” Bshara explained, “so we had a track record and a first-hand understanding of customer needs, and the market dynamics. We could see that some market segments were being underserved, and with the support from our spouses, Lana and Liat, and our funding friends Avigdor [Willenz] and Manuel [Alba], we started Annapurna Labs.”\n\nThat was mid-2011, and three and half years later Amazon acquired the company. The two friends have continued their journey at Amazon, where their team’s work has spoken for itself.\n\nLast year, industry analyst David Vellante praised AWS’s “revolution in system architecture.”\n\n“Much in the same way that AWS defined the cloud operating model last decade, we believe it is once again leading in future systems. The secret sauce underpinning these innovations is specialized designs… We believe these moves position AWS to accommodate a diversity of workloads that span cloud, data center as well as the near and far edge.”\n\nAnnapurna’s work was highlighted during the AWS [Silicon Innovation Day virtual event](https://pages.awscloud.com/GLBL-Silicon-Innovation-Day-2022-reg-event.html) on August 3. In fact, Nafea was a featured speaker in the event. The Silicon Innovation Day broadcast, which highlighted AWS silicon innovations, included a keynote from [David Brown](https://www.linkedin.com/in/david-brown-aws/), vice president, Amazon EC2; a talk about the history of AWS silicon innovation from [James Hamilton](https://www.linkedin.com/in/jameshamilton4/), Amazon senior vice president and distinguished engineer who holds more than 200 patents in 22 countries in server and datacenter infrastructure, database, and cloud computing; and a fireside chat on the Nitro System with [Anthony Liguori](https://www.linkedin.com/in/anthony-liguori-11b01b58/), AWS vice president and distinguished engineer, and [Jeff Barr](https://www.linkedin.com/in/jeffbarr/), AWS vice president and chief evangelist.\n\nIn advance of the silicon-innovation event, Amazon Science connected with Bshara to discuss the history of Annapurna, how the company and the industry have evolved in the past decade, and what the future portends.\n\n#### **Q. You co-founded Annapurna Labs just over 11 years ago. Why Annapurna?**\n\nA. I co-founded the company with my longtime partner, Billy, and with an amazing set of engineers and leaders who believed in the mission. We started Annapurna Labs because we looked at the way the chip industry was investing in infrastructure and data centers; it was minuscule at that time because everybody was going after the gold rush of mobile phones, smartphones, and tablets.\nWe believed the industry was over indexing on investment for mobile, and under investing in the data center. The data center market was underserved. That, combined with the fact that there was increasing disappointment with the ineffective and non-productive method of developing chips, especially when compared with software development. The productivity of software developers had improved significantly in the past 25 years, while the productivity of chip developers hadn’t improved much since the ‘90s. In assessing the opportunity, we saw a data-center market that was being underserved, and an opportunity to redefine chip development with greater productivity, and with a better business model. Those factors contributed to us starting Annapurna Labs.\n\n#### **Q. How has the chip industry evolved in the past 11 years?**\n\nA. The chip industry realized, a bit late, but nevertheless realized that productivity and time to market needed to be addressed. While Annapurna has been a pioneer in advancing productivity and time to market, many others are following in our footsteps and transitioning to a building-blocks-centric development mindset, similar to how the software industry moved toward object-oriented, and service-oriented software design.\nChip companies have now transitioned to what we refer to as an intellectual property-oriented, or IP-oriented, correct-by-design approach. Secondly, the chip industry has adopted the cloud. Cloud adoption has led to an explosion of compute power for building chips. Using the cloud, we are able to use compute in a ‘bursty’ way and in parallel. We and our chip-industry colleagues couldn’t deliver the silicon we do today without the cloud. This has led to the creation of a healthy market where chip companies have realized they don’t need to build everything in house, in much the same way software companies have realized they can buy libraries from open source or other library providers. The industry has matured to the point where now there is a healthy business model around buying building blocks, or IPs, from providers like [Arm](http://www.arm.com/), [Synopsys](https://www.synopsys.com/), [Alphawave](https://www.awaveip.com/en/home/), or [Cadence](https://www.cadence.com/en_US/home.html).\n\n#### **Q. Annapurna Labs was named after one of the tallest peaks in the Himalayas that’s regarded as one of the most dangerous mountains to climb. What's been the tallest peak you've had to climb?**\n\nA. I’m up in the cloud, I don’t need to climb anything [laughing]. Yes, Billy and I picked the name Annapurna Labs for a couple of reasons. First, Billy and I originally planned to climb Annapurna before we started the company. But then we got excited about the idea, acquired funding, and suddenly time was of the essence, so we put our climbing plans on hold and started the company. We called it Annapurna because at that time – and it’s true even today – there is a high barrier to entry in starting a chip company. The challenge is steep, and the risk is high, so it’s just like climbing Annapurna. We also believed that we wanted to reach a point above the clouds where you could see things very clearly, and without clutter. That’s always been a mantra for us as a company: Avoid the clutter, and look far into the future to understand what the customer really needs versus getting distracted by the day-to-day noise.\n\n#### **Q. What are the unique challenges you face in designing chips for ML training and inference versus more general CPU designs?**\nA. First, I would want to emphasize what challenge we didn’t have to worry about: with the strong foundation, methodologies, and engineering muscle we built delivering multiple generations of Nitro, we had confidence in our ability to execute on building the chips and manufacturing them at high volume, and high quality. So that was a major thing we didn’t need to worry about. Designing for machine learning is one the most challenging, but also the most rewarding tasks I've had the pleasure to participate in. There is an insatiable demand for machine learning right now, so anyone with a good product won’t have any issues finding customer demand. The demand is there, but there are a couple of challenges.\n\nThe first is that customers want ‘just works’ solutions because they have enough challenges to work on the science side. So they are looking for a frictionless migration from the incumbent, let's say GPU-based machine learning, to [AWS Trainium](https://aws.amazon.com/machine-learning/trainium/) or [AWS Inferentia](https://aws.amazon.com/machine-learning/inferentia/). Our biggest challenge is to hide all the complexity so it’s what we refer to internally as boring to migrate. We don’t want our customers, the scientists and researchers, to have to think about moving from one piece of hardware to another. This is a challenge because the incumbent GPUs, specifically NVIDIA, have done a very good job developing broadly adopted technologies. The customer shouldn’t see or experience any of the hard work we’ve done in developing our chips; what the customer should experience is that it’s transparent and frictionless to transition to Inferentia and Trainium. That’s a hefty task and one of our internal challenges as a team.\n\n\n\n\"The customer shouldn’t see or experience any of the hard work we’ve done in developing our chips; what the customer should experience is that it’s transparent and frictionless to transition to Inferentia and Trainium,\" says Bshara.\n\nThe second challenge is more external; it’s the fact that science and machine learning are moving very fast. As an organization that is building hardware, our job is to predict what customers will need three, four, five years down the road because the development cycle for a chip can be two years, and then it gets deployed for three years. The lifecycle is around five years and trying to predict how the needs of scientists and the machine-learning community will evolve over that time span is difficult. Unlike CPU workloads, which aren’t evolving very quickly, machine learning workloads are, and it’s a bit of an art to keep apace. I would give ourselves a high score, not a perfect score, in being efficient in terms of execution and cost, while still being future proof. It’s the art of predicting what customers will need three years from now, while still executing on time and budget. These things only come with experience, and I’m fortunate to be part of a great team that has the experience to strike the right balance between cost, schedule, and future-proofing the product.\n\n#### **Q. At the recent re:MARS conference Rohit Prasad, Amazon senior vice president and Alexa head scientist, said the voice assistant is interacting with customers billions of times each week. Alexa is powered by EC2 Inf1 instances, which use [AWS Inferentia](https://aws.amazon.com/cn/machine-learning/inferentia/?trk=cndc-detail) chips. Why is it more effective for Alexa workloads to take advantage of this kind of specialized processing versus more general-purpose GPUs?**\n\nA. Alexa is one of those Amazon technologies that we want to bring to as many people as possible. It’s also a great example of the Amazon flywheel; the more people use it, the more value it delivers. One of our goals is to provide this service with as low latency as possible, and at the lowest cost possible, and over time improve the machine-learning algorithms behind Alexa. When people say improving Alexa, it really means handling much more complex machine learning, much more sophisticated models while maintaining the performance, and low latency. Using Inferentia, the chip, and Inf1, the EC2 instances that actually hosts all of these chips, Alexa is able to run much more advanced machine learning algorithms at lower costs and with lower latency than a standard general-purpose chip. It's not that the general-purpose chip couldn't do the job, it's that it would do so at higher costs and higher latency. With Inferentia we deliver lower latency and support much more sophisticated algorithms. This results in customers having a better experience with Alexa, and benefitting from a smarter Alexa.\n\n#### **Q. AI has been called the new electricity. But as ML models become increasingly large and complex as you just discussed, there also are concerns that energy consumption for AI model training and inference is damaging to the environment. At the chip level, what can be done to reduce the environmental impact of ML model training and Inference?**\n\nA. What we can do at the chip level, at the [EC2](https://aws.amazon.com/ec2/) level, is actually work on three vectors, which we’re doing right now. The first is drive to lower power quickly by using more advanced silicon processes. Every time we build a chip in an advanced silicon process we're utilizing new semiconductor processes with smaller transistors that require less power for the same work. Because of our focus on efficient execution, we can deliver to EC2 customers a new chip based on a more modern, power-efficient silicon process every 18 months or so.\nThe second vector is building more technologies, trying to accelerate in hardware and in algorithms, to get training and inference done faster. The faster we can handle training and inference, the less power is consumed. For example, one of the technologies we innovated in the last Trainium chip was something called stochastic rounding which, depending upon which measure you're looking at for some neural workloads, could accelerate neural network training by up to 30%. When you say 30% less time that translates into 30% less power.\n\nAnother thing we're doing at the algorithmic level is offering different data types. For example, historically machine learning used a 32-bit [floating point](https://en.wikipedia.org/wiki/Single-precision_floating-point_format). Now we’re offering multiple versions of 16-bit and a few versions of 8-bit. When these different data types are used, they not only accelerate machine learning training, they significantly reduce the power for the same amount of workload. For example, doing matrix multiplication on a 16-bit float point is less than one-third the total power if we had done it with 32-bit floating point. The ability to add things like stochastic rounding or new data types at the algorithmic level provides a step-function improvement in power consumption for the same amount of workload.\n\nThe third vector is credit to EC2 and the Nitro System, we’re offering more choice for customers. There are different chips optimized for different workloads, and the best way for customers to save energy is to follow the classic Amazon mantra – the everything store. We offer all different types of chips, including multiple generations of Nvidia GPUs, Intel Habana, and Trainium, and share with the customer the power profile and performance of each of the instances hosting these chips, so the customer can choose the right chip for the right workload, and optimize for the lowest possible power consumption at the lowest cost.\n\n#### **Q. I’ve focused primarily on machine learning. But let’s turn our attention to more general-purpose workloads running in the cloud, and your work on Graviton processors for Amazon EC2.** \n\nA. Yes, in a way [Graviton](https://aws.amazon.com/ec2/graviton/) is the opposite of our work on machine learning, in the sense that the focus is on building server processors for general-purpose workloads running in EC2. The market for general-purpose chips has been there for thirty or forty years, and the workloads themselves haven’t evolved as rapidly as machine learning, so when we started designing, the target was clear to us.\n\n\n\nAWS is three generations into its Graviton chip journey, and Bshara says the company has plans for \"many more generations\" to come.\n\nBecause this segment of the industry wasn’t moving that fast, we felt our challenge was to move the industry faster, specifically in offering step function improvement in performance, and reducing costs, and power consumption. There are many times when you build plans, especially for chips, where the original plans are rosy, but as the development progresses you have to make tradeoffs, and the actual product falls short of the original promise. With first-generation Graviton, we experienced the opposite; we were pleasantly surprised that both performance and power efficiency turned out better than our original plan. That’s very rare in our industry.\n\nhe same has been true with Graviton2. Because of this there has been a massive movement inside Amazon for general workloads to move to Graviton2, mainly to save on power, but also on costs. For the same workloads, Graviton2 will on average consume 60% less power than same-generation competitive offerings, and we’re passing on those cost-savings to customers. Outside Amazon, at least 48 of AWS’s top 50 customers have not just tested, but have production workloads running on Graviton2.\n\nIn May, Graviton3 processors became available, so it’s still Day 1 as we’re only three generations into this journey. We have plans for many more generations, but it’s always very satisfying and rewarding to hear how boring it is for customers to migrate to Graviton, and to hear all the customer success stories. It is incredibly satisfying to come to work every day and hear some of the success stories from the tens of thousands of customers using Graviton.\n\n#### **Q. You have more than 100 openings on [your jobs page](https://www.amazon.jobs/en/landing_pages/annapurna%20labs?offset=0&result_limit=10&sort=relevant&distanceType=Mi&radius=24km&latitude=&longitude=&loc_group_id=&loc_query=&base_query=&city=&country=&region=&county=&query_options=&). What kind of talent are you seeking? And what are the characteristics of employees who succeed at Annapurna Labs?**\nA. We are seeking individuals who like to work on cutting-edge technology, and approach challenges from a principles-first approach because most of the challenges we confront haven’t been dealt with before. While actual experience is important, we place greater value on proper thinking and a principles-first mindset, or reasoning from first principles.\nWe also value individuals who enjoy working in a dynamic environment where the solution isn’t always the same hammer after the same nail. Given our principles-first approach, many of our challenges get solved at the chip level, the terminal level, and the system level, so we seek individuals who have systems understanding, and are skilled at working across disciplines. It’s difficult for an individual with a single discipline, or single domain knowledge, who isn’t willing to challenge her or himself by learning across other domains, to succeed at Annapurna. Last but not least, we look for individuals who focus on delivering, within a team environment. We recognize ideas are “cheap”, and what makes the difference is delivering on the idea all the way to production. Ideas are a commodity. Executing on those ideas is not.\n\n#### **Q. I've read that Billy and you share the belief that if you can dream it, you can do it. So what's your dream about future silicon development?**\n\nA. That’s true, and it’s the main reason Billy and I wanted to join AWS, because we had a common vision that there’s so much value we can bring to customers, and AWS leadership and Amazon in general were willing to invest in that vision for the long term. We agreed to be acquired by Amazon not only because of the funding and our common long-term vision, but also because building components for our own data centers would allow us to quickly deliver customer value. We’ve been super happy with the relationship for many reasons, but primarily because of our ability to have customer impact at global scale.\n\nAt Amazon, we operate at such a scale and with such a diversity of customers that we are capable of doing application-specific, or domain-specific acceleration. Machine learning is one example of that. What we’ve done with [Aqua](https://aws.amazon.com/redshift/features/aqua/) (advanced query accelerator) for [Amazon Redshift](https://aws.amazon.com/cn/redshift/?trk=cndc-detail) is another example where we’ve delivered hardware-based acceleration for analytics. Our biggest challenge these days is deciding what project to prioritize. There’s no shortage of opportunities to deliver value. The only way we’re able to take this approach is because of AWS. Developing silicon requires significant investment, and the only way to gain a good return on that investment is by having a lot of volume and cost-effective development, and we’ve been able to develop a large, and successful customer base with AWS.\n\nI should also add that before joining Amazon we thought we really took a long-term perspective. But once you sit in Amazon meetings, you realize what long-term strategic thinking really means. I continue to learn every day about how to master that. Suffice to say, we have a product roadmap, and a technology and investment strategy that extends to 2032. As much uncertainty as there is in the future, there are a few things we’re highly convicted in, and we’re investing in them, even though they may be ten years out. I obviously can’t disclose future product plans, but we continue to dream big on behalf of our customers.\n\nABOUT THE AUTHOR\n\n#### **Staff writer**\n","render":"<p><a href=\\"https://www.linkedin.com/in/nafeabshara/\\" target=\\"_blank\\">Nafea Bshara</a>, Amazon Web Services vice president and distinguished engineer, and the co-founder of Annapurna Labs, an Israeli-based chipmaker that Amazon acquired in 2015, maintains a low profile, as does his friend and Annapurna co-founder, <a href=\\"https://www.linkedin.com/in/hrvoye-bilic-a088644/\\" target=\\"_blank\\">Hrvoye (Billy) Bilic</a>.</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/a32abf59115f4b7daa35c22b562a70b0_image.png\\" alt=\\"image.png\\" /></p>\n<p>Nafea Bshara, AWS vice president and distinguished engineer.</p>\n<p>Each executive’s LinkedIn profile is sparse, in fact, Bilic’s is out of date.</p>\n<p>“We hardly do any interviews; our philosophy is to let our products do the talking,” explains Bshara.</p>\n<p>Those products, and silicon innovations, have done a lot of talking since 2015, as the acquisition has led to, among other advancements, the development of five generations of the <a href=\\"https://aws.amazon.com/ec2/nitro/\\" target=\\"_blank\\">AWS Nitro System</a>, three generations (<a href=\\"https://aws.amazon.com/blogs/aws/new-ec2-instances-a1-powered-by-arm-based-aws-graviton-processors/\\" target=\\"_blank\\">1</a>, <a href=\\"https://aws.amazon.com/blogs/aws/coming-soon-graviton2-powered-general-purpose-compute-optimized-memory-optimized-ec2-instances/\\" target=\\"_blank\\">2</a>, <a href=\\"https://aws.amazon.com/blogs/aws/join-the-preview-amazon-ec2-c7g-instances-powered-by-new-aws-graviton3-processors/\\" target=\\"_blank\\">3</a>) of custom-designed, <a href=\\"https://www.arm.com/\\" target=\\"_blank\\">Arm</a>-based <a href=\\"https://aws.amazon.com/ec2/graviton/\\" target=\\"_blank\\">Graviton</a> processors that support data-intensive workloads, as well as <a href=\\"https://aws.amazon.com/machine-learning/trainium/\\" target=\\"_blank\\">AWS Trainium</a>, and <a href=\\"https://aws.amazon.com/machine-learning/inferentia/\\" target=\\"_blank\\">AWS Inferentia</a> chips optimized for machine learning training and inference.</p>\\n<p>Some observers have described the silicon that emerges from Annapurna Labs in the U.S. and Israel as AWS’s “secret sauce”.</p>\n<p>Nafea’s silicon journey began at <a href=\\"https://www.technion.ac.il/en/technion-vision/\\" target=\\"_blank\\">Technion University</a> in Israel, where he earned bachelor’s and master’s degrees in computer engineering, and where he first met Hrvoye. The two then went on to work for Israel-based Galileo, a company that made chips for networking switches, and controllers for networking routers. Galileo was acquired by U.S. <a href=\\"https://www.wsj.com/articles/SB971744654175712989\\" target=\\"_blank\\">semiconductor manufacturer Marvell</a> in 2000, where Bshara and Bilic would work for a decade before deciding to venture out on their own.</p>\\n<p>“We had developed at least 50 different chips together,” Bshara explained, “so we had a track record and a first-hand understanding of customer needs, and the market dynamics. We could see that some market segments were being underserved, and with the support from our spouses, Lana and Liat, and our funding friends Avigdor [Willenz] and Manuel [Alba], we started Annapurna Labs.”</p>\n<p>That was mid-2011, and three and half years later Amazon acquired the company. The two friends have continued their journey at Amazon, where their team’s work has spoken for itself.</p>\n<p>Last year, industry analyst David Vellante praised AWS’s “revolution in system architecture.”</p>\n<p>“Much in the same way that AWS defined the cloud operating model last decade, we believe it is once again leading in future systems. The secret sauce underpinning these innovations is specialized designs… We believe these moves position AWS to accommodate a diversity of workloads that span cloud, data center as well as the near and far edge.”</p>\n<p>Annapurna’s work was highlighted during the AWS <a href=\\"https://pages.awscloud.com/GLBL-Silicon-Innovation-Day-2022-reg-event.html\\" target=\\"_blank\\">Silicon Innovation Day virtual event</a> on August 3. In fact, Nafea was a featured speaker in the event. The Silicon Innovation Day broadcast, which highlighted AWS silicon innovations, included a keynote from <a href=\\"https://www.linkedin.com/in/david-brown-aws/\\" target=\\"_blank\\">David Brown</a>, vice president, Amazon EC2; a talk about the history of AWS silicon innovation from <a href=\\"https://www.linkedin.com/in/jameshamilton4/\\" target=\\"_blank\\">James Hamilton</a>, Amazon senior vice president and distinguished engineer who holds more than 200 patents in 22 countries in server and datacenter infrastructure, database, and cloud computing; and a fireside chat on the Nitro System with <a href=\\"https://www.linkedin.com/in/anthony-liguori-11b01b58/\\" target=\\"_blank\\">Anthony Liguori</a>, AWS vice president and distinguished engineer, and <a href=\\"https://www.linkedin.com/in/jeffbarr/\\" target=\\"_blank\\">Jeff Barr</a>, AWS vice president and chief evangelist.</p>\\n<p>In advance of the silicon-innovation event, Amazon Science connected with Bshara to discuss the history of Annapurna, how the company and the industry have evolved in the past decade, and what the future portends.</p>\n<h4><a id=\\"Q_You_cofounded_Annapurna_Labs_just_over_11_years_ago_Why_Annapurna_28\\"></a><strong>Q. You co-founded Annapurna Labs just over 11 years ago. Why Annapurna?</strong></h4>\\n<p>A. I co-founded the company with my longtime partner, Billy, and with an amazing set of engineers and leaders who believed in the mission. We started Annapurna Labs because we looked at the way the chip industry was investing in infrastructure and data centers; it was minuscule at that time because everybody was going after the gold rush of mobile phones, smartphones, and tablets.<br />\\nWe believed the industry was over indexing on investment for mobile, and under investing in the data center. The data center market was underserved. That, combined with the fact that there was increasing disappointment with the ineffective and non-productive method of developing chips, especially when compared with software development. The productivity of software developers had improved significantly in the past 25 years, while the productivity of chip developers hadn’t improved much since the ‘90s. In assessing the opportunity, we saw a data-center market that was being underserved, and an opportunity to redefine chip development with greater productivity, and with a better business model. Those factors contributed to us starting Annapurna Labs.</p>\n<h4><a id=\\"Q_How_has_the_chip_industry_evolved_in_the_past_11_years_33\\"></a><strong>Q. How has the chip industry evolved in the past 11 years?</strong></h4>\\n<p>A. The chip industry realized, a bit late, but nevertheless realized that productivity and time to market needed to be addressed. While Annapurna has been a pioneer in advancing productivity and time to market, many others are following in our footsteps and transitioning to a building-blocks-centric development mindset, similar to how the software industry moved toward object-oriented, and service-oriented software design.<br />\\nChip companies have now transitioned to what we refer to as an intellectual property-oriented, or IP-oriented, correct-by-design approach. Secondly, the chip industry has adopted the cloud. Cloud adoption has led to an explosion of compute power for building chips. Using the cloud, we are able to use compute in a ‘bursty’ way and in parallel. We and our chip-industry colleagues couldn’t deliver the silicon we do today without the cloud. This has led to the creation of a healthy market where chip companies have realized they don’t need to build everything in house, in much the same way software companies have realized they can buy libraries from open source or other library providers. The industry has matured to the point where now there is a healthy business model around buying building blocks, or IPs, from providers like <a href=\\"http://www.arm.com/\\" target=\\"_blank\\">Arm</a>, <a href=\\"https://www.synopsys.com/\\" target=\\"_blank\\">Synopsys</a>, <a href=\\"https://www.awaveip.com/en/home/\\" target=\\"_blank\\">Alphawave</a>, or <a href=\\"https://www.cadence.com/en_US/home.html\\" target=\\"_blank\\">Cadence</a>.</p>\\n<h4><a id=\\"Q_Annapurna_Labs_was_named_after_one_of_the_tallest_peaks_in_the_Himalayas_thats_regarded_as_one_of_the_most_dangerous_mountains_to_climb_Whats_been_the_tallest_peak_youve_had_to_climb_38\\"></a><strong>Q. Annapurna Labs was named after one of the tallest peaks in the Himalayas that’s regarded as one of the most dangerous mountains to climb. What’s been the tallest peak you’ve had to climb?</strong></h4>\\n<p>A. I’m up in the cloud, I don’t need to climb anything [laughing]. Yes, Billy and I picked the name Annapurna Labs for a couple of reasons. First, Billy and I originally planned to climb Annapurna before we started the company. But then we got excited about the idea, acquired funding, and suddenly time was of the essence, so we put our climbing plans on hold and started the company. We called it Annapurna because at that time – and it’s true even today – there is a high barrier to entry in starting a chip company. The challenge is steep, and the risk is high, so it’s just like climbing Annapurna. We also believed that we wanted to reach a point above the clouds where you could see things very clearly, and without clutter. That’s always been a mantra for us as a company: Avoid the clutter, and look far into the future to understand what the customer really needs versus getting distracted by the day-to-day noise.</p>\n<h4><a id=\\"Q_What_are_the_unique_challenges_you_face_in_designing_chips_for_ML_training_and_inference_versus_more_general_CPU_designs_42\\"></a><strong>Q. What are the unique challenges you face in designing chips for ML training and inference versus more general CPU designs?</strong></h4>\\n<p>A. First, I would want to emphasize what challenge we didn’t have to worry about: with the strong foundation, methodologies, and engineering muscle we built delivering multiple generations of Nitro, we had confidence in our ability to execute on building the chips and manufacturing them at high volume, and high quality. So that was a major thing we didn’t need to worry about. Designing for machine learning is one the most challenging, but also the most rewarding tasks I’ve had the pleasure to participate in. There is an insatiable demand for machine learning right now, so anyone with a good product won’t have any issues finding customer demand. The demand is there, but there are a couple of challenges.</p>\n<p>The first is that customers want ‘just works’ solutions because they have enough challenges to work on the science side. So they are looking for a frictionless migration from the incumbent, let’s say GPU-based machine learning, to <a href=\\"https://aws.amazon.com/machine-learning/trainium/\\" target=\\"_blank\\">AWS Trainium</a> or <a href=\\"https://aws.amazon.com/machine-learning/inferentia/\\" target=\\"_blank\\">AWS Inferentia</a>. Our biggest challenge is to hide all the complexity so it’s what we refer to internally as boring to migrate. We don’t want our customers, the scientists and researchers, to have to think about moving from one piece of hardware to another. This is a challenge because the incumbent GPUs, specifically NVIDIA, have done a very good job developing broadly adopted technologies. The customer shouldn’t see or experience any of the hard work we’ve done in developing our chips; what the customer should experience is that it’s transparent and frictionless to transition to Inferentia and Trainium. That’s a hefty task and one of our internal challenges as a team.</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/72283dc2e48e4ba4b8b2c2b472f917e6_image.png\\" alt=\\"image.png\\" /></p>\n<p>“The customer shouldn’t see or experience any of the hard work we’ve done in developing our chips; what the customer should experience is that it’s transparent and frictionless to transition to Inferentia and Trainium,” says Bshara.</p>\n<p>The second challenge is more external; it’s the fact that science and machine learning are moving very fast. As an organization that is building hardware, our job is to predict what customers will need three, four, five years down the road because the development cycle for a chip can be two years, and then it gets deployed for three years. The lifecycle is around five years and trying to predict how the needs of scientists and the machine-learning community will evolve over that time span is difficult. Unlike CPU workloads, which aren’t evolving very quickly, machine learning workloads are, and it’s a bit of an art to keep apace. I would give ourselves a high score, not a perfect score, in being efficient in terms of execution and cost, while still being future proof. It’s the art of predicting what customers will need three years from now, while still executing on time and budget. These things only come with experience, and I’m fortunate to be part of a great team that has the experience to strike the right balance between cost, schedule, and future-proofing the product.</p>\n<h4><a id=\\"Q_At_the_recent_reMARS_conference_Rohit_Prasad_Amazon_senior_vice_president_and_Alexa_head_scientist_said_the_voice_assistant_is_interacting_with_customers_billions_of_times_each_week_Alexa_is_powered_by_EC2_Inf1_instances_which_use_AWS_Inferentia_chips_Why_is_it_more_effective_for_Alexa_workloads_to_take_advantage_of_this_kind_of_specialized_processing_versus_more_generalpurpose_GPUs_53\\"></a><strong>Q. At the recent re:MARS conference Rohit Prasad, Amazon senior vice president and Alexa head scientist, said the voice assistant is interacting with customers billions of times each week. Alexa is powered by EC2 Inf1 instances, which use AWS Inferentia chips. Why is it more effective for Alexa workloads to take advantage of this kind of specialized processing versus more general-purpose GPUs?</strong></h4>\\n<p>A. Alexa is one of those Amazon technologies that we want to bring to as many people as possible. It’s also a great example of the Amazon flywheel; the more people use it, the more value it delivers. One of our goals is to provide this service with as low latency as possible, and at the lowest cost possible, and over time improve the machine-learning algorithms behind Alexa. When people say improving Alexa, it really means handling much more complex machine learning, much more sophisticated models while maintaining the performance, and low latency. Using Inferentia, the chip, and Inf1, the EC2 instances that actually hosts all of these chips, Alexa is able to run much more advanced machine learning algorithms at lower costs and with lower latency than a standard general-purpose chip. It’s not that the general-purpose chip couldn’t do the job, it’s that it would do so at higher costs and higher latency. With Inferentia we deliver lower latency and support much more sophisticated algorithms. This results in customers having a better experience with Alexa, and benefitting from a smarter Alexa.</p>\n<h4><a id=\\"Q_AI_has_been_called_the_new_electricity_But_as_ML_models_become_increasingly_large_and_complex_as_you_just_discussed_there_also_are_concerns_that_energy_consumption_for_AI_model_training_and_inference_is_damaging_to_the_environment_At_the_chip_level_what_can_be_done_to_reduce_the_environmental_impact_of_ML_model_training_and_Inference_57\\"></a><strong>Q. AI has been called the new electricity. But as ML models become increasingly large and complex as you just discussed, there also are concerns that energy consumption for AI model training and inference is damaging to the environment. At the chip level, what can be done to reduce the environmental impact of ML model training and Inference?</strong></h4>\\n<p>A. What we can do at the chip level, at the <a href=\\"https://aws.amazon.com/ec2/\\" target=\\"_blank\\">EC2</a> level, is actually work on three vectors, which we’re doing right now. The first is drive to lower power quickly by using more advanced silicon processes. Every time we build a chip in an advanced silicon process we’re utilizing new semiconductor processes with smaller transistors that require less power for the same work. Because of our focus on efficient execution, we can deliver to EC2 customers a new chip based on a more modern, power-efficient silicon process every 18 months or so.<br />\\nThe second vector is building more technologies, trying to accelerate in hardware and in algorithms, to get training and inference done faster. The faster we can handle training and inference, the less power is consumed. For example, one of the technologies we innovated in the last Trainium chip was something called stochastic rounding which, depending upon which measure you’re looking at for some neural workloads, could accelerate neural network training by up to 30%. When you say 30% less time that translates into 30% less power.</p>\n<p>Another thing we’re doing at the algorithmic level is offering different data types. For example, historically machine learning used a 32-bit <a href=\\"https://en.wikipedia.org/wiki/Single-precision_floating-point_format\\" target=\\"_blank\\">floating point</a>. Now we’re offering multiple versions of 16-bit and a few versions of 8-bit. When these different data types are used, they not only accelerate machine learning training, they significantly reduce the power for the same amount of workload. For example, doing matrix multiplication on a 16-bit float point is less than one-third the total power if we had done it with 32-bit floating point. The ability to add things like stochastic rounding or new data types at the algorithmic level provides a step-function improvement in power consumption for the same amount of workload.</p>\\n<p>The third vector is credit to EC2 and the Nitro System, we’re offering more choice for customers. There are different chips optimized for different workloads, and the best way for customers to save energy is to follow the classic Amazon mantra – the everything store. We offer all different types of chips, including multiple generations of Nvidia GPUs, Intel Habana, and Trainium, and share with the customer the power profile and performance of each of the instances hosting these chips, so the customer can choose the right chip for the right workload, and optimize for the lowest possible power consumption at the lowest cost.</p>\n<h4><a id=\\"Q_Ive_focused_primarily_on_machine_learning_But_lets_turn_our_attention_to_more_generalpurpose_workloads_running_in_the_cloud_and_your_work_on_Graviton_processors_for_Amazon_EC2_66\\"></a><strong>Q. I’ve focused primarily on machine learning. But let’s turn our attention to more general-purpose workloads running in the cloud, and your work on Graviton processors for Amazon EC2.</strong></h4>\\n<p>A. Yes, in a way <a href=\\"https://aws.amazon.com/ec2/graviton/\\" target=\\"_blank\\">Graviton</a> is the opposite of our work on machine learning, in the sense that the focus is on building server processors for general-purpose workloads running in EC2. The market for general-purpose chips has been there for thirty or forty years, and the workloads themselves haven’t evolved as rapidly as machine learning, so when we started designing, the target was clear to us.</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/55c12e8e274e465ebab2362716117025_image.png\\" alt=\\"image.png\\" /></p>\n<p>AWS is three generations into its Graviton chip journey, and Bshara says the company has plans for “many more generations” to come.</p>\n<p>Because this segment of the industry wasn’t moving that fast, we felt our challenge was to move the industry faster, specifically in offering step function improvement in performance, and reducing costs, and power consumption. There are many times when you build plans, especially for chips, where the original plans are rosy, but as the development progresses you have to make tradeoffs, and the actual product falls short of the original promise. With first-generation Graviton, we experienced the opposite; we were pleasantly surprised that both performance and power efficiency turned out better than our original plan. That’s very rare in our industry.</p>\n<p>he same has been true with Graviton2. Because of this there has been a massive movement inside Amazon for general workloads to move to Graviton2, mainly to save on power, but also on costs. For the same workloads, Graviton2 will on average consume 60% less power than same-generation competitive offerings, and we’re passing on those cost-savings to customers. Outside Amazon, at least 48 of AWS’s top 50 customers have not just tested, but have production workloads running on Graviton2.</p>\n<p>In May, Graviton3 processors became available, so it’s still Day 1 as we’re only three generations into this journey. We have plans for many more generations, but it’s always very satisfying and rewarding to hear how boring it is for customers to migrate to Graviton, and to hear all the customer success stories. It is incredibly satisfying to come to work every day and hear some of the success stories from the tens of thousands of customers using Graviton.</p>\n<h4><a id=\\"Q_You_have_more_than_100_openings_on_your_jobs_pagehttpswwwamazonjobsenlanding_pagesannapurna20labsoffset0result_limit10sortrelevantdistanceTypeMiradius24kmlatitudelongitudeloc_group_idloc_querybase_querycitycountryregioncountyquery_options_What_kind_of_talent_are_you_seeking_And_what_are_the_characteristics_of_employees_who_succeed_at_Annapurna_Labs_80\\"></a><strong>Q. You have more than 100 openings on <a href=\\"https://www.amazon.jobs/en/landing_pages/annapurna%20labs?offset=0&amp;result_limit=10&amp;sort=relevant&amp;distanceType=Mi&amp;radius=24km&amp;latitude=&amp;longitude=&amp;loc_group_id=&amp;loc_query=&amp;base_query=&amp;city=&amp;country=&amp;region=&amp;county=&amp;query_options=&amp;\\" target=\\"_blank\\">your jobs page</a>. What kind of talent are you seeking? And what are the characteristics of employees who succeed at Annapurna Labs?</strong></h4>\n<p>A. We are seeking individuals who like to work on cutting-edge technology, and approach challenges from a principles-first approach because most of the challenges we confront haven’t been dealt with before. While actual experience is important, we place greater value on proper thinking and a principles-first mindset, or reasoning from first principles.<br />\\nWe also value individuals who enjoy working in a dynamic environment where the solution isn’t always the same hammer after the same nail. Given our principles-first approach, many of our challenges get solved at the chip level, the terminal level, and the system level, so we seek individuals who have systems understanding, and are skilled at working across disciplines. It’s difficult for an individual with a single discipline, or single domain knowledge, who isn’t willing to challenge her or himself by learning across other domains, to succeed at Annapurna. Last but not least, we look for individuals who focus on delivering, within a team environment. We recognize ideas are “cheap”, and what makes the difference is delivering on the idea all the way to production. Ideas are a commodity. Executing on those ideas is not.</p>\n<h4><a id=\\"Q_Ive_read_that_Billy_and_you_share_the_belief_that_if_you_can_dream_it_you_can_do_it_So_whats_your_dream_about_future_silicon_development_84\\"></a><strong>Q. I’ve read that Billy and you share the belief that if you can dream it, you can do it. So what’s your dream about future silicon development?</strong></h4>\\n<p>A. That’s true, and it’s the main reason Billy and I wanted to join AWS, because we had a common vision that there’s so much value we can bring to customers, and AWS leadership and Amazon in general were willing to invest in that vision for the long term. We agreed to be acquired by Amazon not only because of the funding and our common long-term vision, but also because building components for our own data centers would allow us to quickly deliver customer value. We’ve been super happy with the relationship for many reasons, but primarily because of our ability to have customer impact at global scale.</p>\n<p>At Amazon, we operate at such a scale and with such a diversity of customers that we are capable of doing application-specific, or domain-specific acceleration. Machine learning is one example of that. What we’ve done with <a href=\\"https://aws.amazon.com/redshift/features/aqua/\\" target=\\"_blank\\">Aqua</a> (advanced query accelerator) for [Amazon Redshift](https://aws.amazon.com/cn/redshift/?trk=cndc-detail) is another example where we’ve delivered hardware-based acceleration for analytics. Our biggest challenge these days is deciding what project to prioritize. There’s no shortage of opportunities to deliver value. The only way we’re able to take this approach is because of AWS. Developing silicon requires significant investment, and the only way to gain a good return on that investment is by having a lot of volume and cost-effective development, and we’ve been able to develop a large, and successful customer base with AWS.</p>\\n<p>I should also add that before joining Amazon we thought we really took a long-term perspective. But once you sit in Amazon meetings, you realize what long-term strategic thinking really means. I continue to learn every day about how to master that. Suffice to say, we have a product roadmap, and a technology and investment strategy that extends to 2032. As much uncertainty as there is in the future, there are a few things we’re highly convicted in, and we’re investing in them, even though they may be ten years out. I obviously can’t disclose future product plans, but we continue to dream big on behalf of our customers.</p>\n<p>ABOUT THE AUTHOR</p>\n<h4><a id=\\"Staff_writer_94\\"></a><strong>Staff writer</strong></h4>\n"}