AVERAGE FORECAST ERRORS–ORD

May 23, 2020

By:  Winn Soldani

Congratulations to all of you who participated in the first week of the CYC Weather challenge.  Matt and I really appreciate your enthusiasm and effort!

Results for the week are pending, but I took a moment to look at a few trends I noticed across a few variables.  Hopefully you’ve been keeping track of your own forecasts and errors, but, even so, the averages I’m about to present here are perhaps instructive.

There were a total of 87 forecasts across the week that you made that were valid for the contest (i.e. not too late, etc).  We examined the average forecast error across a few variables and then I totaled up a few interesting stats.

MAX TEMPERATURE

  • Average forecast error: -1.8 Degrees
  • # of forecasts that were correct: 7
  • Too high: 21
  • Too low: 59

Here we see a trend that will continue across the rest of this little analysis.  On average, all forecasts were too low (temperature and wind speed).

Interestingly, though, one day alters our perspective on the analysis above.  On Tuesday, May 19, out of 15 forecasts, 11 of them were too HIGH (one was right on, three were too low).  Take that day out, and we now have an average error of -2.6 degrees, and only 10 out of the remaining 72 forecasts were too high,  We had a definite negative bias on most days, except when we didn’t!

MIN TEMPERATURE

  • Average forecast error: -1.8 degrees
  • # of forecasts that were correct: 13
  • Too high: 19
  • Too low: 55

There was slightly less of a negative bias to the low temperatures on average.  While the average error was the same as the high temperatures, there were more correct forecasts (nice work!) and fewer that were too low.

If a day stands out here, it’s day 4 (Thursday).  Out of 16 forecasts, none were correct, and ALL were too low. The average error on this day was a whopping 4.9 degrees (!).

MAX WIND SPEED

I personally find that max wind speed can be the hardest variable to forecast.  Most model guidance and MOS give us hourly looks at the wind speed however, the highest wind speed can—and really almost always does—fall between model guidance—either when the atmosphere “mixes out” as temps get warm enough or perhaps at the arrival of a front (synoptic or sea-breeze).

That struggle can be seen in the miss in our forecasts this past week!

  • Average error: -4.0 kts
  • # correct: 4
  • # too high: 11
  • # too low: 72

Wow!  Almost every single forecast wind speed was too low!

What’s also interesting here is the magnitude of the miss.  The temperature average miss of less than 2 degrees is really not bad.  However, when we look at winds we can sort of calculate an average miss percentage (temperature is trickier since the scale doesn’t stop at 0).

The average max wind speed for the week was 17.8 knots—straight average across the 5 days.  A -4 knot error is therefore a -22% error in wind speed forecast.   Whether or not the percent error would be the same in other conditions is unclear, but this feels like a significant miss.

CONCLUSION

I hope this little analysis raises some interesting questions for you. For instance, if you followed the typical temperature misses I outlined, maybe take a look at what was different about the day(s) where you missed high vs low.  What did you “get” and what elements of the pattern/local effects did you not?  Were your errors larger or smaller than the average?  Why?

And there are three variables I did not discuss here—rainfall, noon wind speed, and noon wind direction.  Rainfall is harder to describe without getting into histograms and scatter plots (and this a volunteer operation, so…).  Nevertheless, what can you glean from your forecast vs. what verified?  Wind direction and speed at noon I’ll also leave to you but note that, like daily max wind speed, almost every single forecast of noon wind speed was too low!  Did you have a negative “miss” tendency for the noon wind?  How close were you on direction?  Did you have a pattern to your misses on direction?  Why?

Forecasting, and then validating and critiquing your own forecasts is the best way to become a better forecaster.  We hope you’ve learned something this week and look forward to the two weeks to come.  Please reach out in comments here, on Facebook, or via email with anything!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: