Which one is right?

Which one is right? This might be the most common question when looking at multiple forecasts issued by an agency or from model guidance. This is a tough question to answer. It depends in part, in what you’re trying to quantify. For this post I’m only worried about the Isaias’s location. After all, if you can’t get the location right, how good will the intensity forecasts be?

The folks at the National Hurricane Center are among the best at trying to integrate all kinds of model guidance and releasing their own answer every six hours. And, over the longer term, on average they do a great job. The figure to the right shows the skill of the 2019 official forecasts and model guidance for storms in the Atlantic basin during the 2017-2019 seasons. The higher the skill relative to CLIPER5 (a climatology and persistence model) the better the model performance.

This is a significant achievement. There is a lot of information that is used to develop the forecast, and they acknowledge the forecast is not perfect. One means to assess the uncertainty is to look at the “cone of uncertainty”. The cone shows the band based on the performance of forecasts over the past seasons where statistics would suggest that around 2/3’s of the time a storm center will actually be located. The 18z forecast, cone of uncertainty, and ATCF tracks are shown below.

But what about the performance on individual storms? Forecasts often mention that one model or another tends to be performing well. The exact reasons for this no doubt vary and may be related to environmental variables. Kinetic Analysis is starting to develop decision support tools to help people understand the performance of model guidance during a storm’s life. There is not a single way to look at this, and the answer will likely vary as a function of forecast lead time.

A simple way to look at model performance would be to look at the distance between the forecast position of a storm and its verified location. The figure below is an example of this and shows the distance (in km) between the 1-day lead time forecast at 00Z and 12Z and the actual storm centers from the CMC, ECMWF, GFS, HWRF and UKMO models and the NHC forecast for tropical cyclone Isaias.

The problem with this approach is that there is a lot of noise. Essentially, the figure above just gives you a scalar measure of the error. You don’t have any knowledge if the forecast predicted that the storm would move faster or slower than, or track more to the left or right of what, actually happened. The figure below presents a 2-dimensional view of how the 1-day lead forecasts shown above evolved through time relative to the verified storm location. The center of each plot is the storm’s location. The ends and corners in the lines are the forecast position of the storm at 00z and 12z. The lines are colored by the verification day: yellow is the start of the storm and the dark blue the most recent verified time. The dots on the outer rim of the “radar plots” show the storms direction at the time of verification. You can see that the OFCL (the NHC) forecast seems to be zeroing in on the storm’s behavior.

We feel this plot gives a more intuitive sense of model performance and provides important additional information. However, an additional piece of information, the average performance over a storm’s life is also needed. This information for this example is shown in the table. Note that this is not a rigorous comparison as the number of verifications varies by model and forecast. Nonetheless, the table captures the average performance shown above. The value of the previous figure is that it shows how the model and forecast performance has varied over time.

In addition to the model track, people need to be aware of the spatial extent of the hazards (winds, waves, storm surge, rain, inland flooding, etc.) associated with the tropical cyclone. We use the forecasts and model guidance in our ocean and atmosphere models to calculate this information. An example of the sustained winds for Isaias based on the 18z forecast calculated with our standard resolution model is shown below. We generate all of this information whenever a forecast or model guidance is updated, essentially every 6 hours in the Atlantic.