Wednesday, November 15th 2023
Microsoft Introduces 128-Core Arm CPU for Cloud and Custom AI Accelerator
During its Ignite conference, Microsoft introduced a duo of custom-designed silicon made to accelerate AI and excel in cloud workloads. First of the two is Microsoft's Azure Cobalt 100 CPU, a 128-core design that features a 64-bit Armv9 instruction set, implemented in a cloud-native design that is set to become a part of Microsoft's offerings. While there aren't many details regarding the configuration, the company claims that the performance target is up to 40% when compared to the current generation of Arm servers running on Azure cloud. The SoC has used Arm's Neoverse CSS platform customized for Microsoft, with presumably Arm Neoverse N2 cores.
The next and hottest topic in the server space is AI acceleration, which is needed for running today's large language models. Microsoft hosts OpenAI's ChatGPT, Microsoft's Copilot, and many other AI services. To help make them run as fast as possible, Microsoft's project Athena now has the name of Maia 100 AI accelerator, which is manufactured on TSMC's 5 nm process. It features 105 billion transistors and supports various MX data formats, even those smaller than 8-bit bit, for maximum performance. Currently tested on GPT 3.5 Turbo, we have yet to see performance figures and comparisons with competing hardware from NVIDIA, like H100/H200 and AMD, with MI300X. The Maia 100 has an aggregate bandwidth of 4.8 Terabits per accelerator, which uses a custom Ethernet-based networking protocol for scaling. These chips are expected to appear in Microsoft data centers early next year, and we hope to get some performance numbers soon.
Sources:
The Verge, Tom's Hardware, Microsoft Blog
The next and hottest topic in the server space is AI acceleration, which is needed for running today's large language models. Microsoft hosts OpenAI's ChatGPT, Microsoft's Copilot, and many other AI services. To help make them run as fast as possible, Microsoft's project Athena now has the name of Maia 100 AI accelerator, which is manufactured on TSMC's 5 nm process. It features 105 billion transistors and supports various MX data formats, even those smaller than 8-bit bit, for maximum performance. Currently tested on GPT 3.5 Turbo, we have yet to see performance figures and comparisons with competing hardware from NVIDIA, like H100/H200 and AMD, with MI300X. The Maia 100 has an aggregate bandwidth of 4.8 Terabits per accelerator, which uses a custom Ethernet-based networking protocol for scaling. These chips are expected to appear in Microsoft data centers early next year, and we hope to get some performance numbers soon.
10 Comments on Microsoft Introduces 128-Core Arm CPU for Cloud and Custom AI Accelerator
Yea, you go, Microsloth :(
Before ARM Neoverse, if you wanted to custom create your own high performance cores, you'd need like $10 Billion and like 10,000 employees designing just the core, creating the compiler / OS / etc. etc.. Dropping the costs below the $1 Billion mark opened up the chips business to many more companies (ie: from like 5 companies to maybe 20 companies). Its still requires behemoth-sized Microsoft-like companies to design a high performance core (Amazon, Microsoft, etc. etc.), but its way easier today than ever before.
Total costs for R&D of this Cobalt is probably just $500 Million, maybe far less than that. And Microsoft is absolutely the kind of company where I can imagine ~1-million computers being deployed to Azure alone. All Microsoft needs to do is save $500 per computer/server and this entire venture is worthwhile. Just napkin math. I don't know how much Microsoft really spent on R&D or their servers. But you can see how quickly this works out for them.
Rerun the math for a decade or two ago with a $15 Billion R&D project, and something like that would never pay for itself.
Intel fabs are needed to make all those different architectures at the volume growth required. No serious players will ever fab using Intel due to competitive secrecy.
I'm pretty sure this isn't necessarily as much about ARM being superior as it simply is Microsoft wanting other options in the event that neither Intel or AMD can overcome the current (no pun intended) x64 power and thermal challenges. It's really expensive to power and cool a datacenter full of 400 watt Xeons, so if you can only get 80% of the performance for 50% of the power it's a net win.
the question I have is: what will consumers choose and why? There is one future where both x86 and arm coexist. But there’s also other potential futures where one dominates due to perhaps superior power efficiency and the other languishes. Of course, one major thing that the incumbent x86 has going for it is software compatibility…
Ever since Apple divorced Intel, more and more companies are considering arm. Microsoft. Amazon. Qualcomm Oryon is coming, and many of the major laptop vendors have announced a partnership...dell hp ASUS etc. it could become the case where there are multiple arm vendors that enter, further eroding intel’s sales.
I think Intel may be in trouble. It has to get power consumption in check quickly. Because if nvidia enters the arena, with a strong cpu + GPU offering, I could see many gamers going for the nvidia solution. Right now nvidia is focused on ai and is dominating there. It’s also dominating in GPU sales as well. Intel may just end up being relegated to not much more than the TSMC of the USA. Microsoft had a major AI keynote yesterday and mentioned nvidia and AMD but barely spoke of Intel…