Accurate forecasts of product demand play a crucial role in optimizing logistics and production operations in both the short and long term. Therefore, achieving the highest possible level of prediction accuracy is highly desirable. To effectively train predictive models and ensure optimal performance, it is essential to assess the deviation between the forecasted demand and the actual demand using an appropriate metric.
However, the selection of a metric becomes critical as it should accurately represent the prediction error. If an unsuitable metric is utilized, the predictive models will not be adequately optimized, leading to inaccurate predictions. Conventional metrics such as Mean Absolute Percentage Error (MAPE) or Root Mean Square Error (RMSE), although widely used, may not be suitable for evaluating forecasting errors, particularly for demand patterns characterized by sporadic or intermittent occurrences. These conventional metrics often fail to adequately account for factors such as temporal shifts (i.e., prediction occurring before or after actual demand) or cost-related considerations.
To address these limitations, a novel metric that goes beyond statistical considerations and incorporates relevant business aspects. By combining statistical analysis with business-related factors, proposed metric offers a more comprehensive evaluation of forecasting accuracy. To validate the effectiveness of this metric.
Introduction
The choice of forecasting method depends on evaluating which approach best suits the problem at hand. Therefore, it is essential to assess the performance of forecasting methods and evaluate predictions using various accuracy measures that have been proposed and extensively discussed over the past few decades. However, there is no universally optimal performance metric that can be universally applied to all types of forecasting problems.
Depending on the selected metric, forecasts can yield significantly different performances, making accurate evaluation challenging. Furthermore, some metrics are not suitable for certain types of data. For instance, the mean absolute percentage error (MAPE) produces infinite or undefined values when actual values are zero or close to zero, which is often the case in certain applications. Consequently, many existing metrics work well with smooth and linear patterns but become less accurate or even unusable when dealing with intermittent patterns, which is frequently observed in product demand forecasting due to occurrences of zero values.
The evaluation of forecasts for stock-keeping units using traditional accuracy metrics can lead to misleading conclusions, highlighting the need to develop appropriate error measures that incorporate additional factors in industries where intermittent demand is prevalent. Intermittent demand is characterized by non-demand intervals, while lumpy demand exhibits large fluctuations in the size of demand occurrences.
Stock-keeping-oriented Prediction Error Costs (SPEC)
To address the limitations of existing metrics in analyzing demand patterns with frequent zero values, a new metric that considers both the shortcomings mentioned earlier and economic factors.
The purpose of this new metric is to measure the accuracy of predictions by comparing the actual demand with the forecast in terms of the costs incurred over the forecast period. There are costs associated with keeping inventory in the warehouse. The longer the inventory stays and the greater the quantity, the higher the stock-keeping costs. Additionally, there are opportunity costs when customer orders cannot be fulfilled due to unavailability of items in the warehouse. The proposed metric should be zero for a perfect prediction that optimally manages storage, and as the deviation from the perfect prediction increases, the metric should reflect the costs of the misprediction.
By using SPEC, it becomes possible to measure forecasts in two dimensions, considering both the magnitude and the timing. In situations where predictions are based on SKUs, solely focusing on the difference between the actual value and the forecast may not be sufficient for a proper evaluation of the time series. SPEC addresses this by accounting for the deviation in the x-direction (time) as well as the y-direction (magnitude) of the forecast. This approach enables a comprehensive assessment of the prediction quality across the entire time series, even for multi-step-ahead forecasts. SPEC overcomes the limitations of traditional error metrics and provides a more comprehensive evaluation of different forecasting models, considering factors such as stock-keeping costs and opportunity costs.
By adjusting the weights α 1 and α 2, the sensitivity of SPEC can be customized to the specific costs associated with a particular application. Choosing α1 = 1 – α2 ensures that the SPEC values remain consistent despite changes in the cost ratio. The relative positions of the SPEC curves for different predictive models can be used to analyze which model performs better overall or to determine the optimal parameter ratio α1/α2 for a particular model. It is recommended to select α 1 and α 2 based on the characteristics of the application and the related costs.
Equation
Stock-keeping-oriented prediction error cost
N = length of the time series
yt = actual demand at time t
ft = corresponding forecast
α1, α2 = opportunity and stock-keeping costs (the sum of which is always 1)
SAP IBP Data Analysis
To assess the effectiveness of the Stock-keeping-oriented Prediction Error Costs (SPEC) metric, several forecast models were created and evaluated using different performance metrics, including Mean Absolute Percentage Error (MAPE), Root Mean Square Error (RMSE), Mean Squared Error (MSE), and SPEC with varying alpha values.
The objective was to compare the performance of these forecast models and determine the suitability of the SPEC metric in capturing forecasting errors, particularly in the context of demand planning. By incorporating economic aspects such as stock-keeping costs and opportunity costs, the SPEC metric aims to provide a more comprehensive evaluation of forecast accuracy.
Each forecast model was trained using historical time series data and then evaluated using the aforementioned performance metrics. The SPEC metric was calculated using different alpha values, which represent the weights assigned to stock-keeping costs and opportunity costs in the overall cost equivalent.
By conducting this comparative analysis, it was possible to gauge the effectiveness of the SPEC metric in capturing forecasting errors and its ability to outperform traditional metrics such as MAPE, RMSE, and MSE in the context of demand planning. The choice of alpha values allowed for flexibility in adjusting the sensitivity of the SPEC metric to the specific cost implications of the application.
SAP IBP Statistical Forecast
Outcome
SPEC Avoid Overstocking 0.75, Avoid Understocking 0.25
The results of the Statistical Forecast Quantity - Stock-keeping-oriented Prediction Error Costs (SPEC) metric with alpha values of 0.75 and 0.25 have shown promising outcomes in comparison to other evaluation metrics. These findings suggest that SPEC (0.75 & 0.25) provides a more accurate assessment of forecast accuracy when compared to alternative approaches. The incorporation of stock-keeping costs and opportunity costs in the SPEC metric contributes to its effectiveness in evaluating forecast performance, particularly in scenarios related to quantity forecasting.
Caution
SPEC metric particularly suitable for intermittent and lumpy historical data sets, is one that takes into account the duration of the error rather than calculating it solely for individual periods. This approach is especially useful in accurately assessing forecast accuracy for datasets with sporadic and uneven patterns.
As a consequence of the iterative nature of the SPEC metric, forecast errors in periods further in the past from the current period are assigned higher significance. However, if recent forecast errors hold greater importance for your specific needs compared to errors from earlier periods, It is recommend implementing a test phase in which the algorithm determining the optimal forecast calculation is configured accordingly. This approach ensures that the SPEC metric can be customized to prioritize recent forecast accuracy when evaluating and selecting the best forecast algorithm for your purposes.
You should not use SPEC if the time series or the ex-post forecast contains negative values. Such values are replaced with zeros when the errors are calculated, which is likely to skew the results.
References
A New Metric for Lumpy and Intermittent Demand Forecasts: Stock-keeping-oriented Prediction Error Co...
SAP IBP Help Documentation
Best Regards,
Lingaiah Vanam
SAP Community Member
- SAP Managed Tags:
2 Comments
| User | Count |
|---|---|
| 5 | |
| 5 | |
| 4 | |
| 2 | |
| 2 | |
| 1 | |
| 1 | |
| 1 | |
| 1 | |
| 1 |
