When creating a forecast, one question always comes up: how can I achieve optimal forecast accuracy?
Several factors go into creating an accurate forecast, including the appropriate algorithm selection, parameter choices, data selection, error measurement, and more... unless you are using some black box model!
The system you are using can help you with some of these choices: For example in SAP Integrated Business Planning we have quite a few algorithms that provide automated parameter selection and we provide vast documentation on the strengths and weaknesses of the different algorithms.
However, there is one point that is often overlooked. Once you've decided on a set of algorithms to experiment with and chosen the parameters (or you let the system chose them for you), and you've identified the most fitting error measure for your requirements, the next question you should be asking is: which data will be used to test the quality of the different algorithms?
There are two alternatives: either use the Ex-Post forecast or a Test Phase.
As these terms might be unfamiliar to some, let's begin with the basics.
Ex-Post forecast is a prediction calculated for past periods for which actual demand history is also available. It is used to assess the accuracy and effectiveness of forecasting models by comparing model predictions made in the past with the actual outcomes that we now have data for. Essentially, it provides a measure of model performance. The term "Ex-Post" is Latin for "after the event".
Here is a very simple example of Ex-Post using simple moving average as the forecasting method:
Ex-Post forecast can be calculated differently depending on the algorithm. In particular, each algorithm can have different “warmup” periods (in the example with Moving Average above, the first three periods are such warmup periods).
Following is a simple example of how Ex-Post forecast is computed in case of regression algorithms (e.g. the Machine Learning algorithm Gradient Boosting of Decision Trees). In case you are not familiar at all with regression algorithms, you can skip this section.
In case of regression algorithms, the model learns from past correlation of the sales history and additional data. And then applies the learning to the past periods.
Let’s assume you have trained some kind of decision tree model on your sales history using events, promotions and the month of the year (this one is system-generated in SAP IBP) as additional data. Here is an example of the sales:
The algorithm then might have learned this rule:
This rule is based on observations of the past sales history. Now the decision tree that the algorithm created using the sales history will be used to create the Ex-Post forecast.
The Ex-Post forecast for month 6 will be exactly 32 because the algorithm has learned that if promotions is 10 and there is an event, then the sales are equal to 32. Notice that if you had been in month 5 you could not have had this information at that time! As you can see, the Ex-Post forecast uses past sales to predict past sales, even if at that point of time the actual sales were not known!
The Ex-Post forecast is not the same as an offset forecast, where you pretend to be in the past and look at how the forecast “would have looked like”. It simply gives an indication of how well the algorithm approximates the sales history.
Since the expectation is that the future sales will resemble the past sales, Ex-Post forecast is still a valid tool to measure forecast accuracy. The system calculates the forecast accuracy by comparing the differences between the actual values and the Ex-Post values.
Various forecast algorithms may have varying initialization periods, i.e., they can begin generating Ex-Post forecasts in different time buckets. This variability can affect the error calculation for the Ex-Post forecast. In SAP IBP, there are several strategies to approach this issue. One such method - Error Measure Calculation Horizon - can be found on the Forecast Model's General tab. This function allows the user to specify how many recent historical periods should be factored into error measurement calculations. This ensures that the error is assessed based on the same number of time periods, irrespective of the algorithm used.
In a statistical context, a Test Phase refers to a specific period set aside in the model development process to assess the performance of a statistical model or algorithm. During this phase, the model is applied to a test data set -a subset of the original data that was not used during the model training. The model's predictions are then compared against the actual results in the test data. This provides meaningful insights into how well the model or algorithm is likely to perform on new, unseen data.
The Test Phase always consists of the most recent periods in the historical horizon. For example, if the current month is January and the number of Test Phase periods is 4, then the Test Phase consists of the last 4 months of the previous year, from September to December.
When a Test Phase is applied, each algorithm that you chose to compare calculates a forecast for the Test Phase based on historical data from the periods before the Test Phase. The chart shows this process.
In SAP IBP you can define a Test Phase by maintaining the Test Phase Periods field on the Forecasting Steps tab of the Forecast Model. The system will preserve this number of the latest historical time periods for testing purpose, and let each algorithm calculate forecast for this period from the remainder of the history horizon. The calculated forecasts are compared with the actuals, and the most accurate algorithm is selected.
Now that we explored how the accuracy of different algorithms is measured within a Best Fit with both Ex-Post and Test Phase, let’s see which one you should use.
Generally speaking, it’s best practice to use a Test Phase rather than Ex-Post forecast to determine the best algorithm in a Best Fit model.
This is because Ex-Post forecast can lead to overfitting of the algorithm, meaning that the algorithm is very good at approximating the sales history. As a result, the forecast accuracy of the Ex-Post is very good, but the generated forecast still has poor quality.
Some algorithms have more accurate Ex-Post forecast then others, but it does not necessarily mean that their forecast will also be more accurate. For example, the exponential smoothing type of algorithms, especially in case of high smoothing coefficient, tend to calculate very accurate Ex-Post forecast.
Using a Test Phase ensures that you are checking the accuracy of the algorithm on a completely different set of data, therefore having more realistic view on which algorithm will perform best in the future. Keep in mind that unless you specify a Test Phase, SAP IBP uses Ex-Post forecast by default for the Best Fit.
Are there some cases in which Ex-Post still makes sense? The answer is yes. One very good reason to keep using Ex-Post forecast is the amount of data you have. To make a very simple example, you can’t use a Test Phase of 6 months if the total sales history is 8, as this would not leave enough datapoints for the algorithm to learn (especially if we are talking about more advanced algorithms that need a lot of data). If you are forecasting seasonal time series, then you should leave at least enough data to cover two season cycles.
Also note that if there is a change point (e.g. a significant level shift) in the test phase, the algorithm selection might not be optimal.
I hope this helped you in your journey to optimize your forecast. Do not hesitate to contact me for any questions regarding Ex-Post and Test Phase in SAP IBP for demand.
You can find more information on the topic of choosing the best algorithm in the official help:
Several factors go into creating an accurate forecast, including the appropriate algorithm selection, parameter choices, data selection, error measurement, and more... unless you are using some black box model!
The system you are using can help you with some of these choices: For example in SAP Integrated Business Planning we have quite a few algorithms that provide automated parameter selection and we provide vast documentation on the strengths and weaknesses of the different algorithms.
However, there is one point that is often overlooked. Once you've decided on a set of algorithms to experiment with and chosen the parameters (or you let the system chose them for you), and you've identified the most fitting error measure for your requirements, the next question you should be asking is: which data will be used to test the quality of the different algorithms?
There are two alternatives: either use the Ex-Post forecast or a Test Phase.
As these terms might be unfamiliar to some, let's begin with the basics.
What is Ex-Post forecast?
Ex-Post forecast is a prediction calculated for past periods for which actual demand history is also available. It is used to assess the accuracy and effectiveness of forecasting models by comparing model predictions made in the past with the actual outcomes that we now have data for. Essentially, it provides a measure of model performance. The term "Ex-Post" is Latin for "after the event".
Here is a very simple example of Ex-Post using simple moving average as the forecasting method:
| -10 | -9 | -8 | -7 | -6 | -5 | -4 | -3 | -2 | -1 | Current Week | +1 | +2 | +3 | |
| Sales history | 20 | 30 | 40 | 20 | 21 | 32 | 30 | 40 | 20 | 21 | ||||
| Ex-Post Forecast | 30 =(20+30+40)/3 | 30 = (30+40+20)/3 | 27 = (40+20+21)/3 | 24 = (20+21+32)/3 | 28 = (21+32+30)/3 | 34 = (32+30+40)/3 | 30 =(30+40+20)/3 | |||||||
| Forecast | 27=(40+20+21)/3 | 27 | 27 | 27 |
Ex-Post forecast can be calculated differently depending on the algorithm. In particular, each algorithm can have different “warmup” periods (in the example with Moving Average above, the first three periods are such warmup periods).
A simple example of Ex-Post forecast of regression algorithms
Following is a simple example of how Ex-Post forecast is computed in case of regression algorithms (e.g. the Machine Learning algorithm Gradient Boosting of Decision Trees). In case you are not familiar at all with regression algorithms, you can skip this section.
In case of regression algorithms, the model learns from past correlation of the sales history and additional data. And then applies the learning to the past periods.
Let’s assume you have trained some kind of decision tree model on your sales history using events, promotions and the month of the year (this one is system-generated in SAP IBP) as additional data. Here is an example of the sales:
| Month of the year | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 1 | 2 |
| Sales history | 20 | 30 | 40 | 20 | 21 | 32 | 30 | 40 | 20 | 21 | 25 | 20 | 21 | 30 |
| Event | X | X | ||||||||||||
| Promotions | 10 | 10 | 10 |
The algorithm then might have learned this rule:
This rule is based on observations of the past sales history. Now the decision tree that the algorithm created using the sales history will be used to create the Ex-Post forecast.
The Ex-Post forecast for month 6 will be exactly 32 because the algorithm has learned that if promotions is 10 and there is an event, then the sales are equal to 32. Notice that if you had been in month 5 you could not have had this information at that time! As you can see, the Ex-Post forecast uses past sales to predict past sales, even if at that point of time the actual sales were not known!
The Ex-Post forecast is not the same as an offset forecast, where you pretend to be in the past and look at how the forecast “would have looked like”. It simply gives an indication of how well the algorithm approximates the sales history.
Since the expectation is that the future sales will resemble the past sales, Ex-Post forecast is still a valid tool to measure forecast accuracy. The system calculates the forecast accuracy by comparing the differences between the actual values and the Ex-Post values.
Various forecast algorithms may have varying initialization periods, i.e., they can begin generating Ex-Post forecasts in different time buckets. This variability can affect the error calculation for the Ex-Post forecast. In SAP IBP, there are several strategies to approach this issue. One such method - Error Measure Calculation Horizon - can be found on the Forecast Model's General tab. This function allows the user to specify how many recent historical periods should be factored into error measurement calculations. This ensures that the error is assessed based on the same number of time periods, irrespective of the algorithm used.
What is a Test Phase?
In a statistical context, a Test Phase refers to a specific period set aside in the model development process to assess the performance of a statistical model or algorithm. During this phase, the model is applied to a test data set -a subset of the original data that was not used during the model training. The model's predictions are then compared against the actual results in the test data. This provides meaningful insights into how well the model or algorithm is likely to perform on new, unseen data.
The Test Phase always consists of the most recent periods in the historical horizon. For example, if the current month is January and the number of Test Phase periods is 4, then the Test Phase consists of the last 4 months of the previous year, from September to December.
When a Test Phase is applied, each algorithm that you chose to compare calculates a forecast for the Test Phase based on historical data from the periods before the Test Phase. The chart shows this process.
In SAP IBP you can define a Test Phase by maintaining the Test Phase Periods field on the Forecasting Steps tab of the Forecast Model. The system will preserve this number of the latest historical time periods for testing purpose, and let each algorithm calculate forecast for this period from the remainder of the history horizon. The calculated forecasts are compared with the actuals, and the most accurate algorithm is selected.
Conclusion - How to choose the best algorithm?
Now that we explored how the accuracy of different algorithms is measured within a Best Fit with both Ex-Post and Test Phase, let’s see which one you should use.
Generally speaking, it’s best practice to use a Test Phase rather than Ex-Post forecast to determine the best algorithm in a Best Fit model.
This is because Ex-Post forecast can lead to overfitting of the algorithm, meaning that the algorithm is very good at approximating the sales history. As a result, the forecast accuracy of the Ex-Post is very good, but the generated forecast still has poor quality.
Some algorithms have more accurate Ex-Post forecast then others, but it does not necessarily mean that their forecast will also be more accurate. For example, the exponential smoothing type of algorithms, especially in case of high smoothing coefficient, tend to calculate very accurate Ex-Post forecast.
Using a Test Phase ensures that you are checking the accuracy of the algorithm on a completely different set of data, therefore having more realistic view on which algorithm will perform best in the future. Keep in mind that unless you specify a Test Phase, SAP IBP uses Ex-Post forecast by default for the Best Fit.
Are there some cases in which Ex-Post still makes sense? The answer is yes. One very good reason to keep using Ex-Post forecast is the amount of data you have. To make a very simple example, you can’t use a Test Phase of 6 months if the total sales history is 8, as this would not leave enough datapoints for the algorithm to learn (especially if we are talking about more advanced algorithms that need a lot of data). If you are forecasting seasonal time series, then you should leave at least enough data to cover two season cycles.
Also note that if there is a change point (e.g. a significant level shift) in the test phase, the algorithm selection might not be optimal.
I hope this helped you in your journey to optimize your forecast. Do not hesitate to contact me for any questions regarding Ex-Post and Test Phase in SAP IBP for demand.
You can find more information on the topic of choosing the best algorithm in the official help:
- SAP Managed Tags:
4 Comments
| User | Count |
|---|---|
| 7 | |
| 3 | |
| 2 | |
| 2 | |
| 2 | |
| 2 | |
| 1 | |
| 1 | |
| 1 | |
| 1 |
