Wednesday, March 27th 2024
Microsoft Copilot to Run Locally on AI PCs with at Least 40 TOPS of NPU Performance
Microsoft, Intel, and AMD are attempting to jumpstart demand in the PC industry again, under the aegis of the AI PC—devices with native acceleration for AI workloads. Both Intel and AMD have mobile processors with on-silicon NPUs (neural processing units), which are designed to accelerate the first wave of AI-enhanced client experiences on Windows 11 23H2. Microsoft's bulwark with democratizing AI has been Copilot, as a licensee of Open AI GPT-4, GPT-4 Turbo, Dali, and other generative AI tools from the Open AI stable. Copilot is currently Microsoft's most heavily invested application, with its most capital and best minds mobilized to making it the most popular AI assistant. Microsoft even pushed for the AI PC designator to PC OEMs, which requires them to have a dedicated Copilot key akin to the Start key (we'll see how anti-competition regulators deal with that).
The problem with Microsoft's tango with Intel and AMD to push AI PCs, is that Copilot doesn't really use an NPU, not even at the edge—you input a query or a prompt, and Copilot hands it over to a cloud-based AI service. This is about to change, with Microsoft announcing that Copilot will be able to run locally on AI PCs. Microsoft identified several kinds of Copilot use-cases that an NPU can handle on-device, which should speed up response times to Copilot queries, but this requires the NPU to have at least 40 TOPS of performance. This is a problem for the current crop of processors with NPUs. Intel's Core Ultra "Meteor Lake" has an AI Boost NPU with 10 TOPS on tap, while the Ryzen 8040 "Hawk Point" is only slightly faster, with a 16 TOPS Ryzen AI NPU. AMD has already revealed that the XDNA 2-based 2nd Generation Ryzen AI NPU in its upcoming "Strix Point" processors will come with over 40 TOPS of performance, and it stands to reason that the NPUs in Intel's "Arrow Lake" or "Lunar Lake" processors are comparable in performance; which should enable on-device Copilot.
Source:
Tom's Hardware
The problem with Microsoft's tango with Intel and AMD to push AI PCs, is that Copilot doesn't really use an NPU, not even at the edge—you input a query or a prompt, and Copilot hands it over to a cloud-based AI service. This is about to change, with Microsoft announcing that Copilot will be able to run locally on AI PCs. Microsoft identified several kinds of Copilot use-cases that an NPU can handle on-device, which should speed up response times to Copilot queries, but this requires the NPU to have at least 40 TOPS of performance. This is a problem for the current crop of processors with NPUs. Intel's Core Ultra "Meteor Lake" has an AI Boost NPU with 10 TOPS on tap, while the Ryzen 8040 "Hawk Point" is only slightly faster, with a 16 TOPS Ryzen AI NPU. AMD has already revealed that the XDNA 2-based 2nd Generation Ryzen AI NPU in its upcoming "Strix Point" processors will come with over 40 TOPS of performance, and it stands to reason that the NPUs in Intel's "Arrow Lake" or "Lunar Lake" processors are comparable in performance; which should enable on-device Copilot.
25 Comments on Microsoft Copilot to Run Locally on AI PCs with at Least 40 TOPS of NPU Performance
Having spent so much money on AI hardware, they now need to try and monetize it. So I won’t be surprise when big firms start to charge for use, others will eventually follow along with some sort of subscription model.
Just another name for telemetry and narrative training hehe
Glad I'm done with upgrading !
When the PC can run the full size GPT inference then the sides can turn ... but this way is just some good sounding stickers in the laptop.
Yeah who would of thunk AI would bring security holes hehe
www.techpowerup.com/forums/threads/nvidia-issues-patches-for-chatrtx-ai-chatbot-suspect-to-improper-privilege-management.320955/
To myself anyway the term run locally seems more like a massive local AI search database installed hehe
With most things now days buying is not an option only yearly subscriptions allowed.
For people naive enough to think AI isn't web search results hehe
In that context, I find the comparison to be misplaced. The NPU is not meant to be a big ML/ AI work horse eating watts 24/24 like no tomorrow, but a low power silicon used for stuff that really doesn't require that much processing power (Or to boost the GPU in some cases). A GPU is faster, but also use more power.
Hell I thought most were against mining saying it's a waste of power blah...
AI is mining to and using your resources to do it and wants you to pay a subscription for the pleasure
AI mines user activities and web data so where is the outrage hehe :slap:
The quality of the data that the A.I. use to be trained on is too valuable, from what I read, controlled, and curated content is favoured. A.I. training on broad user data has been done before, it didn't end well :D(Microsoft trained an A.I. with Twitter users' data, and it became a real piece of shit).
They have better ways to exploit us: when you solve a captcha, you contribute to train an A.I. it's digital labour and that's the currency being used for the "free" stuff (which isn't really free).
ML/A.I don't gather as much complains as mining because on some application the end-user does benefits from it. Stuff like real time translation, subject detection, better autofocus in cameras etc...
People that mine actually make money although not much daily it does add up enough to pay for equipment.. eventually
As far as captcha goes there's an app to get around that nonsense did AI make it lol :laugh:
I was just pointing out the irony that AI does mine and frankly lots of people will reject it just as you point out the manipulated shit show already attempted :cool: