Calibration and Sharpness: The Two Independent Aspects of Forecast Quality
What Is a Good Forecast? Forecasts are like friends: Trust is the most important factor (you don’t ever want your friends to lie to you), but among your trustable friends, you prefer meeting those that tell you the most interesting stories. What do I mean…
How To Fix Mean Absolute Error, Part 2
Part 2: How the Ranked Probability Score reconciles statisticians with practitioners In part 1, we challenged the usual way to evaluate Mean Absolute Error by just taking the difference between the predicted mean and the observed outcome. We found that it is necessary to use…
How To Fix Mean Absolute Error, Part 1
The deceiving surprises that Mean Absolute Error hides from us Executive summary Mean Absolute Error is the first choice for practitioners when it comes to evaluate their model, owing to its simple definition and its intuitive business relevance. The evaluation metric Ranked Probability Score, by…
Mean Absolute Percentage Error (MAPE) Has Served Its Duty and Should Now Retire
Executive summary According to Gartner (2018 Gartner Sales & Operations Planning Success Survey), the most popular evaluation metric for forecasts in Sales and Operations Planning is Mean Absolute Percentage Error (MAPE). This needs to change. Modern forecasts concern small quantities on a disaggregated level such…
Forecasting Few Is Different, Part 2
This is a two-part story by Malte Tichy about dealing with sales forecasts that concern fast-moving as well as slow-moving items. Find Part 1 here. How precise can a sales forecast become? The mechanism of fluctuation cancellation ensures that granular low-scale-forecasts are more imprecise, noisy…
Forecasting Few Is Different, Part 1
When dealing with sales forecasts that concern fast-moving as well as slow-moving items, we must account for the non-proportional scaling of relative forecast uncertainty with selling rates, which largely determines the achievable level of precision. Executive summary For the same forecast quality, predictions for slow-moving…
You Should Not Always Have Known Better: Understand and Avoid the Hindsight Selection Bias in Probabilistic Forecast Evaluation
This blog post on forecast evaluation discusses common pitfalls and challenges that arise when evaluating probabilistic forecasts. Our aim is to equip Blue Yonder’s customers and anyone else interested with the knowledge required to setup and interpret evaluation metrics for reliably judging forecast quality. The…