Imagine a world where we can't predict the weather. Your mom tells you to wear your scarf all the time. Aircrafts can accidentally get into the storm, which would be a disaster — passengers will spill their Prosecco. Awful!

For people, to have knowledge about the weather is essential. We are making observations since the Aristotle times. Back then, The first patterns we found were simple. And I mean it. There were things like "a wind from the south has rain in its mouth." Huh? But what did you expect? Have you tried to live two thousand years ago?

Then — the Middle Ages. The empires wanted to conquer new colonies more and die less (quite unpleasant thing, avoid it). Thanks to medieval sailors, weather prediction techniques improved dramatically, but still it was more of an art, than a science. Those medieval guys still had no idea about the atmospheric pressure, the main weather component nowadays.

The modern meteorology started with physicists' discovery of thermodynamics and hydrodynamics.

In 1873 Cleveland Abbe, an American meteorologist published his work, "The physical basis of long-range weather forecasting." He made the researchers ponder about mathematical methods of weather prediction. For the first time in history.

Though, as it usually happens, people of his time laughed at the idea. No surprise. Imagine if I suggested you to think about the mathematical prediction of lifespan today. You'd die laughing right here.

Though, there was a man who did not laugh. In 1904, Vilhelm Bjerknes, a Norwegian physicist and meteorologist wrote an article named "The problem of weather forecasting as a problem in mechanics and physics." And his work set a basis for the whole modern meteorology.

For the first time in history, he highlighted five main components of weather forecasting:

We still use them. But now we know there are a lot more powers that affect the atmosphere. Modern models are considering even factors such as solar radiation, urban architecture, and air pollution. We'll talk about it later.

Besides those five components, Bjerknes compiled seven equations for atmosphere processes prognosis. Of course, they were differential.

Differential equations are great for building theories and torture the students, but they are completely useless in practice. Computers simply can't solve them analytically. People poorly can. It will take a week or so and include serious simplifications.

That means I can predict the weather for tomorrow. But only a week later. Not very handy, right?

In 1922, Lewis Fry Richardson made use of those equations.

He combined them into the system and searched for the answer using numerical methods. Ironically, his main achievement became his main failure. The result he got was absurd. He predicted the 14.5 hPa (109 mm Hg) change in pressure next day. Old ladies would not be delighted if that happened in reality.

Of course, no one appreciated Richardson's conclusions. Еxperts blamed him for poking his nose in their expertise. Aww, just like in comments to my posts!

Only years after, math proved Richardson’s solution was absolutely correct, and the absurd result was caused by mistake in the initial conditions.

Thirty years later, with the explosive development of computer engineering, first computers allowed us to create mathematical models of numerical weather prediction. They all were based on Richardson's equations.

That's how numerical meteorology, as we know it, started.

When I started to write this post, I thought the whole modern weather prediction was based on something like a huge neural network. Humans feed it with the data, and it tells them they might need an umbrella tomorrow.

Yes, this is my level of stupidity. So I went to a real meteorologist, and he explained me everything.

The problem is, the Earth’s atmosphere is a perfect example of mathematical Chaos theory — tiny changes result in enormous outcomes. Neural networks work contrarily. They find substantial patterns in data completely ignoring small differences. In other words, they look in the wrong direction.

To predict the weather, we need complex hydrodynamic modeling.

Imagine, you fill a tub with water and pour a glass of boiling water in it. Liquids start mixing, forming vortexes and transferring the heat. Heated masses of water go up. Waves form, reflect from tub's edges, and cross, forming new splashes.

Now, calculate what will happen to the dust particle at the other side of the tub.

Same goes for atmosphere, a bit more complicated though. We can't stop the atmosphere and make all the calculations of its initial state.

So we need data. A lot of data.

Nowadays we use three main data sources — weather stations, balloons, and satellites.

Weather stations could be anything. Land facilities, small roof-mounted stations, floating buoys, ball-shaped radar buildings (like in airports), etc. The most widespread type (at least in Europe) is the bee-garden-like fields, where meteorologists work instead of bees.

You probably saw one of these near your city.

Those are our three primary data sources. But there are also some special ones, such as testbed aircrafts and special probes. They are used for specific research if needed.

Weather balloons are large rubber balloons on a rope, filled with helium, carrying some equipment. Twice a day all meteorological stations in every city in the world launch them to collect data. They transmit the information about pressure, humidity, and wind, which greatly help us understand what is happening at the atmosphere.

Balloons can lift up to 25 km and fly 50-100 km from the starting point. Then they just blow up. So boom, much fun. You can get your own for ten bucks on eBay.

Weather satellites are pricey guys stuffed with cameras and antennas. They hang out at 38500 km at the orbit. They are not very big. European Meteosat, for example, can easily fit in your room. It's about 3.5 m in height and weighs like a car — about 1.5 ton.

They take pictures and post them on Instagram #space #weather #cozy. Well, no. Once a day they send them to Earth.

If you want to observe the whole planet by yourself, you only need 5-6 of those. If you're a Flat Earth theory adept, two will be enough.

Those are our three primary data sources. But there also special ones, such are testbed aircrafts or special probes. They are used for specific research if needed.

One more type, which appeared not so long ago but is becoming quite popular now is home weather stations. You can buy one too, connect it to the internet and send data to OpenWeatherMap or Weather Underground. They collect it and make predictions.

Professional institutions do not use the home stations data, despite the fact they collect a lot nowadays. The reason is, all the meteorological instruments in the world have the same calibrations. 50ºF is 50ºF. Everywhere around the globe.

But no one calibrates the home stations. Nobody controls if they installed correctly. 5-10% error may not be noticeable for home use, but definitely not acceptable for global institutions.

Every time I hear "meteorologists predicts a..." on TV, I imagine a bunch of magicians in pretty cloaks, sitting by the fire, predicting. Unfortunately, that's not how it works.

A well-trained human expert can accurately predict today's forecast. For tomorrow, a little worse. After that — the end; he needs computer's help.

The "meteorologists" nowadays are the big buzzing boxes made by IBM (generally). They calculate what will happen to the atmosphere, running the numerical weather prediction models.

There are two types of models:

Global models calculate the entire atmosphere of the planet, and are extremely computationally expensive. Besides, most of them are still running on CPU, with almost no use of video cards. Only a couple of institutions in the US and Europe can (and want to) deal with them.

Though, there is a life hack.

The thing is, the Earth is a sphere. And we still have Coriolis force. Whatever happens in one hemisphere barely will occur on the other side of the equator.

So we can only calculate the weather for one hemisphere as if the other one does not exist. And it works. There are 15 such models in the world, mostly in large countries — UKMET in England, EUMETSAT in France, CMC in Canada, JMA in Japan, SLAV in Russia (what an ironic name).

There are also regional models. They run in the specific areas and have no idea what happens in the big big world.

Regional models work best for modeling tricky terrains like mountains, rivers, and a snow cover. They pay attention to the depth of the lakes, they know more about the snow or ice cover, while global models can be extremely stupid there. So always check the local forecast.

Modern models are based on bunch of climate processes. Here's a tiny bit of them:

Some processes are not even researched thoroughly. E.g., we are still poorly modeling physics inside the clouds.

Here is the battle greater than Batman vs. Superman. Meet, the always confronting, American GFS and European ECMWF models.

Both models use the same data to do the same thing — observe the Earth's atmosphere and predict the changes.

ECMWF is considered better than American GFS, but don't rush to conclusions.

The major point in their confrontation happened in 2012. The US coast was expecting the Sandy hurricane. American model predictions indicated it wouldn't reach the shore. Contrary to that, European model alarmed that it will hit in 4-7 days, so the immediate evacuation was needed.

Imagine the struggle of the American meteorologists.

On October 29, Sandy hit the eastern coast. It affected 24 states, from Florida to Maine, dealt severe damage to New Jersey and New York, reached Michigan and Wisconsin. Sandy caused $68.7 billion damage and took 233 lives.

The American model was compromised, raising the issue "Are Europeans Better Than Americans at Forecasting Storms?". It resulted in global changes made to GFS in 2015 and 2017.

Wonder, what is the difference in those two models?

GFS | ECMWF
--- | ---
American
Created Earlier than European, I did not find the exact year though. | European
Created in 1975. First forecasts made in 1979.
All the data is open and free
Anyone can download it from the official website. | Exclusive access
Access to files will cost you about $150K per year.
8,16, and 30+ day forecasts
You can easily find out what model it is by looking at the number of days. | 10, 15, and 30+ day forecasts
Both make 3-day and weekly forecasts. The 15-day forecast is ECMWF's trait.
4 times a day, 3-hour calculations
Launches at 12 am/pm and 6 am/pm UTC, make calculations for 3.5 hours. More launches enable faster fixing of the errors in calculations. | 2 times a day, 6-hour calculations
Launches at 12 am and 12 pm UTC, but because of its complexity, it works on the forecast whole 6 hours. Just in time for the morning news.
Resolution 0.5 degree (~50 km)
Both models divide the Earth in the grid. The smaller the tiles, the higher accuracy of the forecast, but it's harder to calculate it. | Resolution 0.25 degrees (~25 km)
Even that accuracy is not always enough. Especially for the large cities. Here come the regional models.
Better modeling tropical area
Tropical cyclone track forecasting is very important for Americans. | Better for temperate climate
Europeans, obviously, care more about the intracontinental stuff.
Windguru
windguru.cz displays forecasts, based on the American model. Super-handy for all the kite-, wind-, whatever- surfers. | Foreca
Yahoo, MSN Weather app, Cortana and many other popular services use Foreca's data, which works based on the European model.

Most popular forecasting websites on internet display either GFS or ECMWF data. Official institutes, though, make their own forecasts, based on the regional models.

Go play with windy.com to feel the difference between models' predictions.

A guy asked me one day: "if the accuracy of the forecast is just 50%, what weather-people do, exactly? Toss the coin?"

I gave him a coin and asked to tell me the temperature, precipitation and the direction of the wind for the next Thursday.

He ran away with my quarter :(

Over 50 years of development, the accuracy of models has increased from 40% to 95%. Computers became faster, equations — more complicated, and now we have dozens of satellites monitoring our planet every minute.

When you check the current temperature in the morning, with 95% chance, it will stay like this all day. Not bad. In a week though, the accuracy drops to 75%, which is that precise 2-3% multiplied by seven days.

Temperature is not an issue for us anymore. We can predict it with max accuracy. Precipitation is the question. With current models' development level, we only have 80%.

Also interesting, that the accuracy depends on the season. It's 1% lower in Summer than in Winter. Here, check the 5-day seasonal forecast accuracy in GFS and ECMWF:

The GFS is losing to ECMWF again. Only half a percent though. All the other models will differ much more.

Again, it doesn't mean any of the models better. Accuracy is profoundly affected by local factors: terrain, urban architecture, pollution. Literally anything can change the forces alignment.

To resolve those issues for specific areas we use regional models.

Let's take an imaginary small Whateverville town near the lake, surrounded by mountains with a bunch of rivers around. And let's try to predict the weather there using global model with their 25 km accuracy.

No chance. It's almost useless in our case. That's why we need local models.

They tuned for work with local areas — cities, states, countries, etc. Most often their resolution set to 3-5 km, but it can be any, depending on how many computers you have.

Regional models know much more about local processes. For example, a lake covered in ice and a lake without it affect the weather differently. Same goes for snow coverage and building height. Global models won't down to that level.

Regional models though have no idea of the world beyond their borders; they get all the information from global models.

Your favorite TV weather reports are provided by regional models. On the internet, you see results of GFS and ECMWF.

Researchers' favorite local models WRF (Weather Research and Forecasting) originated from GFS. It's very simple and open, so it's frequently used by private companies.

WRF is available on GitHub. Anyone can download and launch it: github.com/NCAR/WRFV3. You can't use it out of the box though. As a typical OpenSource product it needs a proper dose of suffering. Though, if you're lucky (and stubborn), you deal with it for the evening.

For a modern machine, it will take about 10 hours to make a forecast. So if you want to make your own, someday, download WRF, get data and run it for a night.

Regional models are easily configurable and updated if they get the new data.

Imagine, there's a lake in Whateverville. For a global model, it's just a paltry ant — it does not matter. Though, this lake affects the climate of Whateverville more, than all the other atmospheric phenomena on Earth. It's 50F in the forecast, but 70F in reality. Ok, huh?

That's why regional models and human-meteorologists are vital. They will measure, preserve, and correct.

In last ten years, the new idea has gained popularity. It called Еnsemble methods. No, it's not about gypsies.

The point is to produce a set (ensemble) of forecasts with slightly different input data, instead of making a single prediction for the most likely weather.

Our atmosphere is still a perfect illustration of chaos theory. Even if we get unlimited CPU, we won't be able to calculate all the details.

Though, we can add perturbations and see how it affects the forecast. If we have initial 5 m/s wind speed data, we can ask, what if it was 7 m/s? What if the pressure drops couple pascals and forms the clouds? Will it change the whole picture?

Let's calculate a lot of possible outcomes, so we have something to choose from!

ECMWF creates 50 additional "hypothetical" simulations, while GFS makes 20. That is one another factor that affects the calculation time (6h in European, 3h in American). But when it comes to predicting the hurricane patterns, every possible result is essential.

Tech-guys are eager to use neural networks everywhere. Including weather, of course.

As I said earlier — it's a bad idea. But there is a trick. We can mix and correct results of global models and neural networks. Brilliant! Nobel Prize winner here!

Nope. Most progressive meteorological institutions already use this fantastic idea. Some even abandoned it already (insider info here).

What's the point? There are always systematic deviations in any model. For my city, for example, the rain expected in a week, most likely will hit couple days earlier. The temperature here is always two degrees lower than in an estimate. It happens in my city all the time, and I'm sure in your part of the world it happens too.

Such errors are constant so that we can find them with machine learning. All we need is to compare the forecast and the reality, and give that result to our favorite machine learning algorithm. Any will cope, they all identify patterns flawlessly.

Machine don't predict. It corrects the forecast. It finds the repetitive errors, which human won't even think of.

This method has one main problem - no one knows how to evaluate its accuracy. Yes, it improves the forecast in some places, but what if it makes them worse for others?

I can't wait when we have something from the "Back to the future," something that will tell us "the rain ends in 23 seconds". Though, I have to disappoint you. Even if we reach this point during the lifetime we have, we'll be too old to give a damn about the weather outside.

The progress is hard and slow. We know so little about our atmosphere. We did not measure every single corner of the globe so our models could consider that. Any new building affects the weather for the whole block. Too few data, too few physicists, too few meteorologists. We have much to grow and improve our services

Our models are inaccurate, but we are trying. Though, they give us a lot of interesting data that we can use right now. Just try.