Sunday, 19 February 2012

Measuring Forecast Error


Ok so I am going to get a little technical today, so make sure you get yourself a coffee before reading on.....

Regardless of the media or work type that is being forecasted one of the biggest challenges for any contact centres in getting a better forecast, is the way in which forecast accuracy is measured and analyzed. The importance of forecasting accuracy should be obvious to any planner, for all other WFM processes are dependent and without accuracy it becomes impossible to match up the workforce accurately to achieve service goals. This is true whether the objective is to get a good schedule fit or build an accurate staffing plan for the next 12 months.

The value created from measuring the accuracy of your forecast is two fold, on one hand being aware of error prompts learning and improvement, whilst on the other confidence in the forecast itself allows a contact center to drive towards plan adherence. In fact, on the point of adherence your forecast accuracy along with other metrics is a very good guide for real-time analysts (RTAs) to determine how tight to manage and target center plan adherence, as well as build contingency plans to compensate for actual/forecasted variance if it is expected to be high.

Before you set forecasting accuracy targets, or beat yourself up (or anyone else for that matter) for not being accurate, a key consideration is to determine how much dispersion (how spread out numbers are) already exists in your call flow i.e how consistent and stable is it. Why is this important? It is important because the higher the dispersion the harder it is to get an accurate forecast by using historical data alone. This is the reason smaller or medium-sized contact centers are more difficult to manage, because the smaller the numbers you are dealing with the larger the dispersion when something changes, for example 1 call out of a total of 10 has a much bigger deviation than 1 out of a 100.


The best way to measure deviation in call flow is to use 'Standard Deviation' to compare like for like time intervals. For example you could measure the total calls on a Monday for the last 4 weeks or on an intra-day basis - 9am on Monday for the last 4 weeks ect ect. I am not going to get into the specifics of how to calculate SD, this is after all not a statistical maths lesson, however the basic definition of SD is that it is a way to measure the dispersion from its mean for a set of data. The more spread apart the data, the higher the deviation. It has all sorts of uses and is the foundation for Six Sigma methodology.

The graph depicts what is considered to be a normal variance distribution, and each band is considered 1 standard deviation. If your call flow is sitting in SD1 near to the middle (highly unlikely) then the chance of accurately forecasting using historical data alone are very high.




Forecasting Methods
The overall accuracy of any forecasting method, no matter what method you use, is determined by comparing the forecasted values with the actual values. There are several methods of measuring accuracy but the three most popular are mean absolute deviation (MAD), mean squared error (MSE) and mean absolute percent error (MAPE). A problem with both MAD and MSE is that their values depend on the magnitude of the item being forecast. For example if the forecast item is measured in thousands, the MAD and MSE results can be very large.

Therefore it would be my recommendation to avoid these two measures and instead use MAPE. MAPE in its traditional form is computed as the average of the absolute difference between the forecasted and actual values, and is expressed as a percentage of the actual values. MAPE is perhaps also the easiest measure to interpret, and as I early said it remains undistorted by any single large value. There is tons of information available online on how to calculate MAPE, it drawbacks, and its alternatives, so I will leave you to explore.

So MAPE will tell you the size of your forecast error which of course is very important, however what it won't tell you is how efficient you are forecasting, what should be the minimum error rate, or even whether the different methods and models you are using are making the forecast better or worse. To determine this I advise you use a very simple process called forecast value added (FVA). It requires a little extra effort upfront but in the long run can really add accuracy value and reduce forecasting man-hour cost by helping you to avoid pointless forecasting processes. It basically uses the simplest, least labor intensive method of forecasting (namely a "naïve" forecast) to benchmark against the forecasting accuracy of each of the stages in your current process.

This is a diagram of a typical forecasting process, by running FVA
you are able to answer the following questions;

  • are all the stages in your forecasting process actually adding accuracy?
  • is the effort expended at each stage actually worth it?
  • where in the process are your weak points?

If you want to learn more about FVA and how to implement it, then I recommend downloading SAS's excellent white paper on the topic, which can be downloaded from their site. Alternatively drop me a mail (go to contact page on this blog) and I will be happy to send you a copy.


The last stage for any forecasting accuracy measurement activity should be first to understand why there is any inaccuracy and then to build a plan to improve. I am constantly surprised at the number of centers I come across that measure all sorts of metrics and KPIs but when I ask them "so how do you use that?", the number that actually do anything with the data is shocking. The whole point of measuring the accuracy of your forecast is to improve it, and the only way I know of doing this is to try and understand why you have a gap. So ask yourself the questions;

"Can I confidently explain each major deviation from forecast? and what learning from each one can be applied to future forecasts?

For example an influx of calls in the morning throughout the whole month will likely be built into the next forecast if you are using any type of historical method. However knowing why this has happened informs you whether you should be including/excluding/smoothing this data for future forecasts. If the influx of calls was a result of a one-off TV advert being played during the mornings of that month only, would you want to include this in future forecasts?.

Thats it for me for now, please do comment with questions as well as let me know if you find this useful.

Doug