Tuesday, February 18th 2020

UK Prepares $1.6 Billion for the Most Powerful Weather Forecasting Supercomputer

The UK government has set aside a budget of 1.2 billion GBP, which is roughly around 1.56 billion US Dollars. With this budget, the UK government plans to install the world's most powerful supercomputer used for weather forecasting in the year 2022. Previously, the UK government used three Cray XC40 supercomputers that are capable of achieving a maximum of 14 PetaFLOPs at its peak performance. The future system plans to take that number and make it look tiny. With plans to make it 20 times more powerful than the current machine, we can estimate that the future supercomputer will have above 200 PetaFLOPs of computing performance.

The supercomputer deployment will follow a series of cycles, where one is happening in 2022 and that supercomputer will be six times more powerful than the current solution. To get to that 20 times improvement, the supercomputer will get an upgrade over the next five years' time. While we do not know what will power the new machine, it will almost definitely be a CPU plus multi-GPU node configuration, as GPUs have gained a lot of traction in weather prediction models lately.
Add your own comment

43 Comments on UK Prepares $1.6 Billion for the Most Powerful Weather Forecasting Supercomputer

#26
phill
notbIt's a cluster. You need 1000 cores, you buy 1000 cores.
AMD CPUs offer better value at the moment, i.e. you need less sockets to deliver the required cluster performance.

And of course there's a good chance this system can benefit from AVX-512. It really is a big selling point for Intel.

First of all: speeds you've mentioned are for home use, not for datacenters...
Second: this supercomputer will likely be kept offline, i.e. no direct Internet access (it isn't needed).
If they need that many cores, they will have less physical kit with AMD due to the core count, so I'd hope that would help them out in that way to be honest. But either way, I'd love to find out the specs of the hardware they are looking at.. Ignoring the same old question about Crysis, but seeing how it's put together would be amazing :)
I thought that AMD had that instruction set in the newer models of CPUs now? I thought that was the case anyways but I think there might be other versions of that set I guess?
Chrispy_Uh, the cost of internet that fast in the UK is negligible. 1Gb/s internet is typically £65/month for consumers, our business fibre (Daul 4Gb/s symmetric with failover to a different trunk) is £2700/quarter or £900/month. There is healthy fibre competition in Exeter where the Met office is based.

Assume that the power and cooling costs of a supercomputer cluster are 10X higher than the internet connectivity, and you'll be in the right ballpark at least.
You're very lucky if you can get 1Gb for £65, I'm paying about half that for 30Mb!! :cry: The line we have at work is 1Gb up/down, I think that was something like £27k for so many years I think (dual lines different exchanges etc etc..
Posted on Reply
#27
Basard
DeathtoGnomesBudget build... so definitely EPYC....

10 Xeons = 15 EPYC and 100+ more cores = budget build. :p
I agree, Epyc would be the right choice.... but government loves to waste taxpayer dollars, so probably Xeons. :P
Posted on Reply
#28
ARF
notbWill it be less amazing with Intel inside?
What's suddenly wrong with Xeons?
They are corrupt and less fair, and among other things - several times more expensive, slower and more power hungry.

It needs to be EPYC + AMD GPUs.
kurosagi01Definitely going to cost more than 1.6 billion with the way how the UK approach things, also they can just continue reporting its going to rain everyday in the UK which is what we are known for having, don't need super computer for that.
But its purpose will be climate simulation, not weather forecast.
Posted on Reply
#29
Cheeseball
Not a Potato
phillI thought that AMD had that instruction set in the newer models of CPUs now? I thought that was the case anyways but I think there might be other versions of that set I guess?
Zen 2 currently has 256-bit-wide registers (single-cycle!!!) for SIMD instructions, so if they had implemented AVX-512 it would not turn out well for them as it would push computational load dramatically just like how AVX/AVX2 does for current Intel CPUs.

The alternative to this approach for ML (which is one few use cases for AVX-512) is pairing the server CPUs with GPGPUs, preferably one with dedicated tensor cores (mostly NVIDIA at the moment) for higher efficiency.
Posted on Reply
#30
notb
phillIf they need that many cores, they will have less physical kit with AMD due to the core count, so I'd hope that would help them out in that way to be honest.
That's only when you assume they'll go for the highest core count models, which may not be the case.
Their current cluster uses E5-2695 v4, which is 18C. That generation offered SKUs with higher core count and higher frequencies (also both).

HPC systems are carefully selected and fine-tuned.
You can't assume everyone gets the 64-core EPYC right now so there's no reason to go with Intel.

Check the latest Top500 (54th). Out of 147 systems started in 2019, 99 don't use any accelerator - i.e. they're focused on CPU performance.
95 of them use Intel CPUs (from few generations).
Histogram of "Cores per Socket":
10: 1
12: 6
14: 6
16: 15
18: 5
20: 53
24: 5
28: 2
48: 2

Also, people somehow focused on AMD vs Intel, wheareas they may also go for something GPU-based or even ARM. Both approaches are being tested in weather modeling with good results.
Posted on Reply
#31
phill
CheeseballZen 2 currently has 256-bit-wide registers (single-cycle!!!) for SIMD instructions, so if they had implemented AVX-512 it would not turn out well for them as it would push computational load dramatically just like how AVX/AVX2 does for current Intel CPUs.

The alternative to this approach for ML (which is one few use cases for AVX-512) is pairing the server CPUs with GPGPUs, preferably one with dedicated tensor cores (mostly NVIDIA at the moment) for higher efficiency.
Thanks for clearing that up :D
notbThat's only when you assume they'll go for the highest core count models, which may not be the case.
Their current cluster uses E5-2695 v4, which is 18C. That generation offered SKUs with higher core count and higher frequencies (also both).

HPC systems are carefully selected and fine-tuned.
You can't assume everyone gets the 64-core EPYC right now so there's no reason to go with Intel.

Check the latest Top500 (54th). Out of 147 systems started in 2019, 99 don't use any accelerator - i.e. they're focused on CPU performance.
95 of them use Intel CPUs (from few generations).
Histogram of "Cores per Socket":
10: 1
12: 6
14: 6
16: 15
18: 5
20: 53
24: 5
28: 2
48: 2

Also, people somehow focused on AMD vs Intel, wheareas they may also go for something GPU-based or even ARM. Both approaches are being tested in weather modeling with good results.
For things in general going at any price point, you do get more for your money with AMD over Intel at the moment but again comes back to the if there's a better tool for the job, then they can use it. I just mentioned I hope they go with AMD :) More demand for AMD is going to be a good thing for AMD :)
Posted on Reply
#32
HammerOn1024
For weather forecasting?! In the imortal words of Monty Python "Pull the other one!"
Posted on Reply
#33
tomo82
Tatty_OneI think some of the reasoning behind this whether justified or not is a wish to get faster more detailed and localised predictions, for example, in the last 2 weeks or so we have had 2 huge storms coming in on the Jetstream, one a hurricane with months worth of rain in a day or two, huge parts of the country are currently flooded, most rivers throughout the country have burst their banks, many by up to 8ft of water, thousands homeless. Most of these places have flood defences, significant in some places but even they can't handle an extra 8 feet of water depth running in torrents, current forecasting says that areas of the country will get x amount of rain but we can't/don't break those areas down into small pieces and define within an area which towns/cities will get it the worst where some may not get it at all, not sure if this will work but it's in part the Government reacting to an outcry of home owners across the country asking what Government is going to do to make them safe as many of these homeowners cannot even get buildings or contents insurance because of where they live in "flood zones" and they cannot move because nobody wants to buy a house these days on a flood plain.
(I'm a brit) - TBH they could run it off a Pi1 - it just needs to display the word "SHIT". There - I saved you 1.2 billion quid. Maybe that will offset the cost of leaving the EU. Ha! who am I kidding.
Posted on Reply
#34
notb
phillFor things in general going at any price point, you do get more for your money with AMD over Intel at the moment
That's only if you look at CPU price floating in vacuum.

The fact that your needs can be covered with 50% less sockets (when going 64-core EPYC) means indisputable savings if the 64-core EPYC is a product for you.

Down the ladder it's not that obvious, because you end up with the same number of sockets (nodes, blades) either way, so possible savings are limited.
And it's really not obvious that a EPYC one is cheaper just because EPYC CPU is cheaper. Other aspects may mean that a company like Cray is able to offer Intel variant for less - even if the CPU alone is more expensive.
I just mentioned I hope they go with AMD :) More demand for AMD is going to be a good thing for AMD :)
You're a shareholder or what? :o
Posted on Reply
#35
ARF
notbYou're a shareholder or what? :eek:
It looks like the market reacts really positively on the news and the AMD stock is on the rise while Intel's stock is on the decline.


finance.yahoo.com/quote/Amd?ltr=1
Posted on Reply
#36
notb
ARFIt looks like the market reacts really positively on the news and the AMD stock is on the rise while Intel's stock is on the decline.
At this point it's really hard to say why AMD is going up or down. Or why they're priced where they are.

It's certainly not based on this Met Office contract - even if we assume AMD is sure to get it.
Posted on Reply
#37
Chrispy_
phillYou're very lucky if you can get 1Gb for £65, I'm paying about half that for 30Mb!! :cry: The line we have at work is 1Gb up/down, I think that was something like £27k for so many years I think (dual lines different exchanges etc etc..
Well if you're stuck on Openreach's copper last mile then yes, you're completely stuffed :(
~30Mb/s is all you can hope for with fibre to the cabinet in the UK and BT/Openreach's decision to roll out FTTC as an upgrade to ADSL has probably set back the UK by a decade in the internet bandwidth race, IMO. I'm stuck on FTTC myself in london and at least 6 months away from a 1Gb/s HyperOptic FTTP installation.

We are, however, a minority. BT have installed FTTP to 4.1 million UK homes as of October 2019 and In addition to that, here are hybrid fibre/coax FTN networks. Take Virgin Media - who are already offering at least 108Mb/s to 75% of the UK, 53% of which is 360Mb/s capable network. Pricing is similar to Openreach FTTC, so although not quite FTTP it's good enough for most people at the moment. If you're not served by Virgin or Openreach's FTTP then you are probably one of the 15% who are either living out in the wilderness (and completely screwed until 5G can cover your internet bandwidth needs), or you're likely to be in the queue for an FTTP provider to cover your area.

As for business connections to the Met Office in Exeter, they are located are within half a kilometre of at least one Fibre ISP, and there are datacenters in the same radius for numerous bandwidth-hungry businesses including Amazon. I assume the Met office location was picked because it's such a well-connected, datacenter-heavy region, and the physical proximity will keep their internet costs down, not that it's remotely an issue given the running costs of the hardware itself!

In case you're wondering how I know this, my former employer (15 years ago) ran a datacenter in that area (Sowton Industrial Estate) and I spent years driving past all this every day.
Posted on Reply
#38
phill
notbThat's only if you look at CPU price floating in vacuum.

The fact that your needs can be covered with 50% less sockets (when going 64-core EPYC) means indisputable savings if the 64-core EPYC is a product for you.

Down the ladder it's not that obvious, because you end up with the same number of sockets (nodes, blades) either way, so possible savings are limited.
And it's really not obvious that a EPYC one is cheaper just because EPYC CPU is cheaper. Other aspects may mean that a company like Cray is able to offer Intel variant for less - even if the CPU alone is more expensive.

You're a shareholder or what? :eek:
I wish I was considering where they were 3 years ago or so... They have done amazing things and wonders with their hardware :)
I'm interested in finding out how they'd even put together a system like that, I just hope that they are going to use AMD hardware that's all :) It'll be a lovely push forward for them if they do I think :)
Chrispy_Well if you're stuck on Openreach's copper last mile then yes, you're completely stuffed :(
~30Mb/s is all you can hope for with fibre to the cabinet in the UK and BT/Openreach's decision to roll out FTTC as an upgrade to ADSL has probably set back the UK by a decade in the internet bandwidth race, IMO. I'm stuck on FTTC myself in london and at least 6 months away from a 1Gb/s HyperOptic FTTP installation.

We are, however, a minority. BT have installed FTTP to 4.1 million UK homes as of October 2019 and In addition to that, here are hybrid fibre/coax FTN networks. Take Virgin Media - who are already offering at least 108Mb/s to 75% of the UK, 53% of which is 360Mb/s capable network. Pricing is similar to Openreach FTTC, so although not quite FTTP it's good enough for most people at the moment. If you're not served by Virgin or Openreach's FTTP then you are probably one of the 15% who are either living out in the wilderness (and completely screwed until 5G can cover your internet bandwidth needs), or you're likely to be in the queue for an FTTP provider to cover your area.

As for business connections to the Met Office in Exeter, they are located are within half a kilometre of at least one Fibre ISP, and there are datacenters in the same radius for numerous bandwidth-hungry businesses including Amazon. I assume the Met office location was picked because it's such a well-connected, datacenter-heavy region, and the physical proximity will keep their internet costs down, not that it's remotely an issue given the running costs of the hardware itself!

In case you're wondering how I know this, my former employer (15 years ago) ran a datacenter in that area (Sowton Industrial Estate) and I spent years driving past all this every day.
It's even worse for me, aluminum cabling from the DSlam to my home.. I'm at about 700m I believe, I used to be getting upwards of 44Mb, now lucky if I can get 30Mb.. My sister who's up the road manages 65Mb and she and her husband are not even close to using it anywhere near to what I try and do with mine here :laugh: :cry: :laugh: :cry: If I don't laugh I'll be crying!! :( :(

Exeter isn't too far from me, but I'm not so fond about living in a busy city... But still, I digress :)

If only we had some more information about the build of the system etc.... I'd love to see what even goes in a system like this, let alone how it all works together :) Ah, being a hardware w^&%$ has it's fun spots I guess :)
Posted on Reply
#39
DeathtoGnomes
BasardI agree, Epyc would be the right choice.... but government loves to waste taxpayer dollars, so probably Xeons. :p
when you put it like that....I cannot disagree.
Posted on Reply
#40
Prima.Vera
Everybody is talking about hardware, but nobody talks about the software.
I would like some good quality info like this:
- O.S. of the machine;
- The Software which is going to use;
- Who is going to program it, and how many parallel tasks can it run ?
Posted on Reply
#41
notb
Prima.VeraEverybody is talking about hardware, but nobody talks about the software.
I would like some good quality info like this:
- O.S. of the machine;
- The Software which is going to use;
- Who is going to program it, and how many parallel tasks can it run ?
It has been mentioned multiple times?

Met Office is running an atmospheric model (currently one called Unified Model) and delivering weather forecasts. That's the primary purpose of their current infrastructure (Cray XC40) which this is going to replace.

OS - whatever the supplier provides for the cluster (Linux-based).
It will be used by Met staff, obviously...
I don't understand the "how many parallel tasks" part. 64-bit Linux is limited to 4194304 (2^22).
Posted on Reply
#42
TheUn4seen
laszloi really can't imagine why this super computer is needed for forecasting; the satellites give us already the movement of air masses and the path also; sometimes is impossible to forecast in long term as a sudden change of air movement change all and no matter how powerful computer you use is not-predictable ; i'm sure the new one will forecast the same as the old one and both will be incorrect at long term forecast as you can't foresee how nature behave...
Satellite data is the input (or one of many inputs to be precise), which only gives current conditions. And weather forecasting has gotten fairly good over the years, but it's ridiculously demanding, hence the need for dedicated supercomputers. This is a problem which can be solved with throwing more computational power at it.
You can definitely forsee how nature behaves, it's called science, just as as physics can predict how galaxies behave, meteorology can predict how weather behaves. We don't live in fairy tale land, we can actually use our brains and tools to solve even the most complex problems. It's just another tool.

But, it's UK, so it's never going to happen.
Posted on Reply
#43
Imsochobo
notbI can't believe you don't know why forecasting weather makes sense... Maybe that's not what you meant.

As for forecast quality:
Local forecasts (i.e. what you check to know if it's going to rain tomorrow) are very precise for 2-3 days ahead. This is not a computational limit - weather is just too random.
Large area forecasts (i.e. what will happen in your province/country as a whole - large pressure zones, hurricanes etc) can be modeled even few weeks ahead.

There is nothing better. It's a cluster. You make it as big as you need.
What you actually mean is: cheaper. Based on EPYC it may be cheaper.

What you forget is that - even if EPYC's offer better value, they also offer less supply.
So not every supercomputer can be built with AMD interiors. This is one of the fundamental issues behind current Intel pricing (in all segments). 7nm is still relatively limited.
no, AMD can offer chips, intel cannot.
if we want to buy 200 epyc cpu's we can get them in a week.
If we order 200 intel chips we wait 3 months minimum
Posted on Reply
Add your own comment
Nov 22nd, 2026 17:29 CST change timezone

New Forum Posts

Popular Reviews

TPU on YouTube

Controversial News Posts