Demand Forecasting
Forecast Accuracy Measurement
What’s your forecast accuracy telling you? Stop and ask a few questions
Forecast accuracy is an important performance metric in any effective S&OP process, but it can be measured in various ways. Comparing your company’s accuracy to an industry standard will be difficult to impossible if you don’t know the details behind the measurement. More importantly, the metric needs to resonate within your organization as a meaningful indicator of forecast relevance. So then…
What details should one consider for forecast accuracy measurement?
Here’s the Steelwedge Top Six:
1. Aggregation level: Are you measuring accuracy at a product SKU or family level? What about other hierarchy levels? Odds are your accuracy will appear to be better at an aggregated level such as family. This happens because variability of forecasts and actuals tend to cancel out one another as data is combined. The result is a smoothing of results and lowering of error calculations. Recommendation: Measure accuracy at the same level as the majority of forecasts are captured.
2. Error Calculation: In its most basic form, accuracy is a measure of the difference between a prediction and what actually happened. How far off were we? Error is equal to the difference between forecast and actual. Often, this is captured as a percentage value called percent error. Mean absolute percent error (MAPE) calculates the average of errors. Since we don’t want positives and negatives to cancel out each other, we use the absolute values of each error. There are other methods, but MAPE is fairly common. Weighted MAPE is a method used to give greater importance (weight) to items with greater activity. Amount of activity may be defined as the proportion a particular item is of the total. Recommendation: Keep it simple. Make sure people understand the measurement and how they can impact it.
3. Unit of Measure: “We forecast in both units and dollars. Which should we use for measuring accuracy?” Weighted accuracy measures, such as weighted MAPE, will give greater influence to items that constitute a greater portion of the sales volume. Higher dollar but low unit volume items will contribute much more to a measurement in dollars. Conversely, high unit volume, low dollar items will factor in more prominently using a unit based forecast. Which is preferable? It really depends on your business. Recommendation: Consider important business decisions made in the Executive S&OP meeting. Are they usually focused on $ or units?
4. Offset period: If we measure accuracy using the most recent forecast for a given period, it will likely be more accurate than a forecast made three months prior to a given period. That’s because we have better information as we get closer to the current period. But, how valuable is a forecast made in the very near term if the organization cannot act upon that forecast? It has virtually no value. The offset period defines the number of periods prior to an actual period for which a forecast will be measured against the given period’s actuals. For example, if our offset is 3 months, we can measure accuracy using actuals from August and the forecast for August that was captured in May. Recommendation: Set the offset period to most closely match the organization’s planning horizon.
5. Time buckets: Should we measure accuracy using weeks, months or quarters? Typically, you will want to measure accuracy in the same period buckets used to forecast. In some cases, where demand patterns follow a “hockey stick” high demand in the last month of a quarter, it may be more appropriate to use quarterly buckets. Recommendation: Measure accuracy using the same buckets you use to forecast unless there’s a compelling reason to move to a bigger bucket.
6. Which Forecast?: In a collaborative S&OP process, there may be several forecasts captured (Sales, Marketing, Demand Planning, Consensus, etc). Which should we use for accuracy measurement? If you’re only going to use one, then go with the forecast used by Operations to build or procure product. A typical example would be the Consensus Plan. Measuring accuracy against multiple forecasts will provide greater insights into potential areas for improvement. Recommendation: Measure accuracy using the forecast provided to Operations. Publish results throughout the organization. Also, measure accuracy across other forecasts to isolate areas for improvement.
Are there other aspects you’d add to this list? Please let us know.
Lora Cecere on the SAP Insider Event: Where is SAP APO headed?
Those following Supply Chain Industry Analyst Lora Cecere’s new Supply Chain Shaman blog (http://www.supplychainshaman.com) have read with keen interest her observations about SAP’s progress in the area of Supply Chain Planning. Lora points out that while SAP has made tremendous progress in many areas it is also struggling with integrating its many components – specifically Lora says that the “integration of business intelligence and performance management is moving [too] slowly.” Her notes on the growing disappointment with SAP APO – from within and outside the SAP organization – are also worth noting (http://www.supplychainshaman.com/2010/04/inside-insider:
“I leave the event with two major disappointments. The first is that the integration of business intelligence and performance management is moving slowly. …too slowly for this curmudgeon analyst. I was hoping to see the results of the Teradata/SAP Business Objects integration and the launch of a new generation of predictive analytics. While there is some progress in Performance Management, it is largely traditional reporting/dashboards.
The second is that SAP APO—SAP’s supply chain planning suite—was largely business as usual. At the event, I saw small, incremental changes, but no major innovation like I saw in MII, PLM and transportation management. I keep crossing my fingers. I would love to see SAP have the courage to blow up APO and start again. Who knows if it works for PLM, maybe there is a chance to bring innovation to a solution — and the larger Supply Chain Planning (SCP) market– that sorely needs to be redefined.”
As SAP friends and partners know, SAP has some truly outstanding employees and the SCM Product Group continues under the brilliant leadership of Lori Mitchell-Keller. Yet, overcoming legacy products and dated, mis-guided inertia is difficult for even the most effective of executives. The great news is that a whole new generation of cloud-based supply chain planning and S&OP applications that integrate tightly into the SAP suite are now available. These applications are changing the game and will ensure that SAP users are well supported well into the next generation or whenever it is that SAP is finally able to overcome its legacy and move forward.
Mix vs Volume Planning: What is your planning time fence telling you?
We’ve all been there. The speaker is providing way too much detail. “Just get to the point!”
What’s the right amount of information?
In forecasting demand and supply, there really is a point of TMI (too much information). Detailed forecasting can be counter-productive. It requires more effort to forecast at a detailed product mix level than at a volume level. Similarly, detailed capacity planning requires more effort than rough cut planning. So where should we draw the line?
The planning horizon dictates the appropriate timeframes for mix versus volume planning. The planning horizon is defined as the period of time needed to purchase and receive raw materials plus the manufacturing time needed to produce finished goods. Within the planning horizon, detailed forecasts and plans are needed by the Operations side of the business to produce the right products in the right quantities at the right times.
Beyond the planning horizon, who needs the details? Nobody. So why are so many companies forecasting in detail so far out? Because they can. Planning tools enable detailed forecasting and easy aggregation well into the future. Yet, this technological capability should not lead to the conclusion that more detail is better. If nobody needs the detail, it is wasteful to dedicate the additional effort to plan across a greater number of items.
The solution is to plan at mix detail inside the time fence and at volume level outside the fence. A good S&OP demand forecasting and planning system should make it easy to plan at appropriate levels across products, customers, geographies AND time periods.
S&OP: What can we learn from Martin Luther King, Jr.?
Last week, we celebrated the birth of Martin Luther King, Jr. MLK was a brilliant man with an amazing talent for delivering the spoken and written word. As I pondered several quotes from MLK, there was one that struck me as having a strong relevance within the business world.
“We must learn to live together as brothers or perish together as fools.”
Now don’t get me wrong. The words of MLK go much deeper than business relationships. Still, I can’t help but think that the teachings of MLK can offer us some helpful insights into how we manage our businesses. Effective S&OP requires individuals from varied backgrounds, functional responsibilities and positions within the organization work together for the benefit of the overall organization.
All too often, companies struggle with functional silos, poor communication and a misalignment of effort. If you are a leader in your organization, collaboration and consensus building should be your goal. Be engaged in the S&OP process and encourage all participants to contribute their insights. A dictatorship is not S&OP. Likewise, a free-for-all democracy where each participant has an equal vote is not S&OP either. An effective process leverages bottom-up inputs from distributed resources as well as top-down market and business insights from company leaders.
MLK on leadership:
“A genuine leader is not a searcher for consensus but a molder of consensus.”
Sales and Operations Planning, Collaborative Demand Planning Depends on Bottom-up, Top-down and Statistical Forecasting
An effective S&OP program depends on solid, accurate demand forecasts. Best practice companies do three things well: statistical, top-down and bottom-up forecasting. Many companies are doing one or two of these, but few are doing all three well. Of course, some companies do none. Let’s just say these companies have a huge upside improvement potential.
A statistically generated forecast should use a “best fit” approach to select the mathematical algorithm that minimizes error (such as mean absolute percentage error (MAPE)). The statistical engine should select the best algorithm for each time-phased data series or set of regression data. The resulting forecast should serve as a starting point for bottom-up and top-down forecasts.
Bottom-up forecasts are accumulated from many contributors. A distributed sales force may have hundreds or thousands of contributors. Each contributor has a specific area of expertise such as a specific customer, product or geographic area. The contributor enters her forecasts for her specific area of responsibility. Forecasts from all contributors are summed to capture an overall bottom-up forecast.
Conversely, a top-down approach applies a more centralized view. A small number of forecasters will look at various inputs and generate forecasts. Influencing factors may include market data, economic indicators, and general product and customer trends. Here, too, the statistically generated forecast is a good number from which to start.
The beauty of top-down and bottom-up forecasts is their ability to look at the world from differing vantage points. The folks in the “ivory tower” know important information, but they don’t know everything. The folks in the field have keen insights into their unique areas, but they only see their small piece. The challenge is to capture the small pieces without tainting the field forecaster’s view. In other words, don’t tell the field forecasters the top-down targets. When field forecasters are told what their forecasts are expected to be, they tend to send back values right in line with the top-down values. Such tainted bottom-up forecasts miss the point of gathering field intelligence.
An effective marriage will capture top-down and bottom-up forecasts separately. A management by exception S&OP tool will make comparisons quickly to enable users to analyze critical differences and refine the ultimate consensus driven forecast.
Excel, S&OP and the Comfortable Old Chair
I have an old recliner chair that’s so comfortable, I refuse to let it go. It’s various shades of green (much lighter in the seat area) and worn in the usual spots after 20 years of service.
Excel is a lot like that old chair. I’m often impressed by the creativity found in home-grown Excel planning spreadsheets. Typically, an individual has spent months or years to model the company’s forecasting and planning intricacies producing a complex set of worksheets with never-ending links and calculations. The net effect is that this individual is the only one who really understands the model. Obviously, if this wise individual were to leave the organization, the planning tool and process would break down or cease to function at all.
Smart organizations recognize this vulnerability and take steps to establish supportable enterprise-wide tools. And thus starts the internal battle over the apparent need to throw out Excel in favor of a scalable S&OP tool.
Newsflash: You don’t have to choose. You can have a collaborative, scalable S&OP solution and keep Excel. Enterprise Enabled Excel was expressly created to address the need to provide an organization-wide collaborative planning S&OP solution with the familiarity and ease-of-use of Excel as a front-end user interface.
Steelwedge Software provides a planning and forecasting environment using Excel coupled with a world-class, highly scalable, server-based enterprise planning infrastructure. You get all the advantages of Excel without the complexity, long learning curve and inflexibility of a traditional enterprise application.
Imagine a single planning application that offers ease of use, familiarity, power, flexibility, aggegration/disaggregation, qualitative notes tracking, waterfall reporting, offline capabilities, internal and external collaboration, security, scalability, drill down, data integrity, reduced data latency, and an archived Plan of Record.
The Steelwedge solution uses Excel templates supplemented by Internet Explorer (web browser-based) “pop-up” windows as the presentation interface to deliver a powerful next-generation planning application that leverages AJAX. Web Services, and XML technologies. E3 represents a radically new approach to managing your Excel-based planning process.
An enterprise S&OP tool with an Excel interface. The best of both worlds! Have your cake and eat it too! And…
Keep the comfortable chair!
Receive Blog Updates Via Email
Categories
Recent Posts
Archives
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009
- July 2009
- June 2009
- May 2009
- April 2009
- March 2009
- February 2009
- January 2009
- December 2008
- November 2008
- October 2008
- September 2008
- July 2008
- June 2008
- May 2008
- April 2008
- March 2008
- February 2008
- January 2008
- December 2007
- November 2006
Tag Cloud
Demand Forecasting (30)
Managing in a Recession (33)
Sales & Operations Planning (95)
Sales Forecasting (40)
Steelwedge User Forum (20)
Steelwedge Webinar (12)
WP Cumulus Flash tag cloud by Roy Tanck and Luke Morton requires Flash Player 9 or better.