Demand Forecasting Methods: Using Machine Learning and Predictive Analytics to See the Future of Sales
What is the top pain point for business executives? Gartner, the world’s largest IT research firm, gives a clear answer: demand volatility. Too many factors — from weather fluctuations to posts by social media influencers — impact buyers, causing them to frequently change their minds.
Worse still, things reshaping customer intentions happen quite unexpectedly. Think, for instance, of the teenage climate activist Greta Thunberg. Her refusal to fly for environmental reasons kick-started the “flight shame” movement, that caused a five-percent decrease in air passenger numbers in Sweden.
There is no magic wand to predict scenarios like the “Thunberg effect”. But there are technologies to improve the accuracy of demand forecasting. Honestly, it will never be 100 percent precise, yet it can be precise enough to help you achieve your business goals.
In this article, we will look at the capabilities of advanced forecasting methods and outline their current limitations.
The place and role of forecasting in demand and supply planning
Demand forecasting is the estimation of a probable future demand for a product or service. The term is often used interchangeably with demand planning, yet the latter is a broader process that commences with forecasting but is not limited to it.
Demand planning, according to the Institute of Business Forecasting and Planning (IBF) applies “forecasts and experience to estimate demand for various items at various points in the supply chain.” In addition to making estimations, demand planners take part in inventory optimization, ensure the availability of products needed, and monitor the difference between forecasts and actual sales.
Demand planning serves as the starting point for many other activities, such as warehousing, shipping, price forecasting, and, especially, supply planning that aims at fulfilling the demand and requires data on the anticipated needs of customers.
Here, again, we return to forecasting. Getting as close to reality as possible is the key to improving efficiency across the entire supply chain. How do you reach the uppermost accuracy possible? The answer depends on business type, available resources, and objectives. Let’s compare the existing options: traditional statistical forecasting, machine learning algorithms, predictive analytics that combine both approaches, and demand sensing as a supporting tool.
Traditional statistical forecasting — good for stable markets, ill-disposed to changes
Traditional statistical methods (TSM) have been here for ages and remain a staple of forecasting processes. The only difference if compared with the previous century is that all calculations are performed automatically, by modern software. For example, you can create time-series forecasts for sales and trends in Excel.
Data sources. To predict the future, statistics utilizes data from the past. That’s why statistical forecasting is often called historical. The common recommendation is collecting data on sales for at least two years.
Why to use it. Traditional forecasting is still the most popular approach to predict sales, and for a reason. As a rule, demand planning solutions based on statistical techniques seamlessly integrate with Excel and existing Enterprise Resource Planning (ERP) systems without requiring additional tech expertise. The most advanced systems can consider seasonality and market trends as well as apply numerous methods to finetune results.
Things to consider. An important prerequisite of statistical forecasting accuracy is stability. We assume that history repeats itself: Situations that occured two or three years ago will reoccur. Which is far from being true. Flawless in an ideal world, statistical methods often fail to foresee illogical alterations in customer preferences or predict when market saturation will occur.
A statistical forecasting software dashboard. Source: GMDH Streamline
Best fit. All in all, automated statistical forecasting offers a satisfying level of accuracy for:
- mid- to long-term planning,
- well-established products, that enjoy stable demand, and
- predicting total demand rather than sales of separate stock-keeping units (SKUs).
Does it make business sense to invest in more sophisticated technologies? We’ll try to clear things up in the next section.
Machine learning for demand planning — advanced accuracy at the price of added complexity
Increased computer power on the one hand and increased demand volatility on the other created prerequisites for wider use of machine learning (ML) to design predictions.
Data sources. Built upon statistical models, machine learning utilizes additional internal and external sources of information to make more accurate, data-driven predictions. ML engines can work with both structured and unstructured data including past financial and sales reports (historical data), marketing polls, macroeconomic indicators, social media signals (retweets, shares, spikes in followers), weather forecasts, and more.
Data sources for demand forecasting with machine learning. Source: IBF (Institute of Business Forecasting and Planning ).
Why to use it. Machine learning applies complex mathematical algorithms to automatically recognize patterns, capture demand signals and spot complicated relationships in large datasets. Apart from analyzing huge volumes of information, smart systems continuously retrain models, adapting them to changing conditions thus addressing volatility. These capabilities enable ML-based software to produce more accurate and reliable forecasts in complex scenarios.
What does more accurate really mean? Companies that added machine learning to their existing systems report an increase of 5 to 15 percent in forecast reliability (up to 85 and even 95 percent). In addition to this, your team gets rid of time-consuming manual adjustments and recalibrations.
Things to consider. To take advantage of the machine learning solution, you need sufficient processing power and really large batches of high-quality data. Otherwise, the system won’t be able to learn and generate valuable predictions.
Also, bear in mind the additional complexity in terms of software maintenance and result interpretation. While ML mechanisms come to conclusions without human intervention, it’s up to a live tech expert to determine what features should be fed to the model, which of them have the largest impact on the output, and why the model generates a certain prediction.
All of the above drives up your spendings on equipment and human resources, so better make sure that revenue from a 5-percent improvement in accuracy will cover accompanying expenses.
Best fit. The list of situations in which machine learning definitely works better than traditional statistics includes:
- short- to mid-term planning,
- volatile demand patterns,
- fast changing environment, and
- new product launches.
Comparison between traditional and machine learning approaches to demand forecasting.
Machine learning solutions for demand forecasting
As you can see, employing machine learning comes with some tradeoffs. Depending on the planning horizon, data availability, and task complexity, you can use different statistical and ML solutions.
Predictive sales analytics: modeling the future
A most common enterprise application of machine learning teamed with statistical methods is predictive analytics. It allows for not only estimating demand but also for understanding what drives sales and how customers are likely to behave under certain conditions.
To help you discover what may happen in the future, predictive analytics software performs the following set of operations:
- aggregating historical and new data from different sources, including ERP and Customer Relationship Management (CRM) systems, points of sales (POSs), sensors, customer demand studies, social media, marketing surveys;
- cleansing data;
- determining which forecasting algorithm fits your product best;
- building predictive models to identify likely outcomes and discover relationships between various factors; and
- monitoring models to measure their business results and improve prediction accuracy.
The predictive analytics tools enable businesses to combine company information with important economic indicators, promotional events, weather changes, and other factors that affect customer preferences and buying decisions. It facilitates spotting new market opportunities and generates more granular insights into future demand.
Major drawbacks. Predictive analytics is not the simplest technique as it involves complex machine learning algorithms. Besides, it is geared to generate forecasts for at least a month out and is ill-suited and not meant to visualize the nearer future.
When it comes to shorter periods and daily granularity, demand sensing tools get in the game.
Demand sensing: managing real-time changes
A relatively new concept in the planning process, demand sensing employs machine learning to capture real-time fluctuations in purchase behavior. Many experts do not view it as a standalone forecasting method, but rather a way to adjust existing predictions. That said, the technology can be of great help for companies, operating in fast-changing markets.
Demand sensing solutions extract daily data from POS systems, warehouses, and external sources to detect an increase or decrease in sales by comparison with historical patterns. The system automatically evaluates the significance of each divergence, analyzes influence factors, and offers adjustments to short-term plans.
A demand sensing software dashboard, capturing a change in demand in the short term, and showing factors that cause the fluctuation. Source: E2Open
Adopting demand sensing reportedly reduces near-time forecast errors by 30 to 40 percent. It empowers companies to rapidly address sudden changes in customer needs and facilitates building a data-driven supply chain. Of course, you can’t make all decisions based on this technique alone, as it doesn’t work for mid- or long-term planning. But it may serve as a valuable complement to traditional forecasting methods.
Major drawbacks. Heavily relying on machine learning algorithms, demand sensing inherits all ML pros and cons. It requires significant computing power, massive volumes of data, and a large library of pre-built models. On top of all, some highly sensitive models may send false signals, so you need human logic to analyze results produced by a demand sensing engine.
When machine learning works best for demand planning: successful use cases
Not every business requires costly machine learning solutions to develop a reliable demand plan. But when you face a highly volatile environment or have no historical data, or must consider a large number of variables, investments in smarter technologies will pay off in spades. Below, we’ll consider typical scenarios when machine learning brings the most value to a forecasting process.
New product introduction (NPI)
Traditional forecasting needs two to five years of sales data to guarantee an acceptable level of accuracy. With new items, you have no sales history. Still, you can’t neglect predicting demand as it drives multiple important processes, from procurement to logistics management to marketing support.
Apart from market research and collecting expert opinions, the most common approach to forecasting launches is to identify clusters of predecessors with similar properties and product life cycle curves. Machine learning algorithms can be employed to extract specific patterns from huge volumes of unstructured data, find similarities and develop predictions, considering other sources of relevant information such as web analytics and social media. This entails a higher degree of accuracy and reduces the time needed to create forecasts from days to hours.
Products with a short life cycle
In some industries, companies refresh their assortments every few months, which makes the forecasting task much more difficult. For example, fashion companies launch new products at least twice a year, and the apparel should be sold quickly to make room for the next collection. In this scenario, a demand estimation process must involve examining fashion trends, seasonality, and other external factors — along with historical data related to previous collections.
Machine learning has proven to be effective in such complicated scenarios, and the experience of the global brand Luxottica illustrates this fact. The world’s largest company in the eyewear industry uses machine learning to predict demand for 2000 new styles added to its collection annually. Thanks to the smart engine analyzing data from past launches and spotting patterns of common demand behaviors, the manufacturer has improved its sales forecast accuracy by 10 percent — a significant enhancement for a large volume of products that go out of fashion quickly.
Weather changes can trigger significant demand fluctuations, especially in the case of seasonal products (from swimwear to umbrellas to fur coats), cosmetics, food, and vehicles. Machine learning algorithms help businesses detect and measure the impacts of meteorological elements on sales, and with predictive analytics, you can build “what-if” models for different scenarios.
This approach allows suppliers and retailers to effectively address weather-associated surges and drops in local demand. Reports from ML forecasting adopters show that considering weather effects (for instance, of unusually high temperatures in winter) improves forecast accuracy by 5 to 15 percent for individual food products and up to 40 percent for product groups.
Companies run thousands of consumer promotions that are supposed to drive sales. Alas, different surveys reveal that 20 to 50 percent of these events fail to generate noticeable uplift in demand. Moreover, a study by Nielsen Holdings, the number one market research firm in the US, argues that 59 percent of trade promotions don’t break even, or, in other words, result in additional expense not profit. Ironically, 59 percent appears in a Gartner report, that refers to the number of companies that still use spreadsheets to plan promotions and forecast their impact.
Obviously, without improved technologies, companies can hardly generate reliable predictions for costly marketing campaigns. The reason? Outcomes of promotions depend on numerous factors with complex relationships, hidden in large batches of raw data. Luckily, machine learning can cope with this challenging task, that was proved by the world’s biggest yogurt manufacturer Danone. Thanks to the use of a machine learning engine, the dairy giant witnessed a 20 percent reduction in promotion forecast errors along with a 30 percent decrease in lost sales.
Due to machine learning, Danone achieved better accuracy in forecasting the impact of commercials and promotions on demand. Source: Transart.
Overall, enhancements in promotion predictability entail two immediate benefits. First, they prevent marketing teams from spending too much money on events that won’t pay off. Secondly, they result in more precise inventory management, eliminating the risk of over- or understocking.
Too many variables to analyze
This is the most common issue impacting forecasting accuracy. Highly variable environment, dozens of factors driving buying behaviors, many types of data involved — all these often make demand planning too complex to be successfully performed with simple tools.
The significant complexity of supply chain, short-term demand spikes, and the high cost of errors (with human lives at stake) prompted the Blood and Transport department of the UK’s National Health System (NHS) to transfer from spreadsheets and manual databases to ML-fueled planning system with enhanced predictive capabilities. It allowed hospitals to reduce waste from blood overstocks by 30 percent without any drop in service quality and enabled rapid responding to potential shortages. “If there’s no yogurt on the supermarket shelf — well, that’s unfortunate. If there’s no blood in the hospital, the consequences are very different,” an NHS executive explained as the reason to invest heavily in the advanced solution.
Human brains still matter
Forecasting demand is a challenging task, and it still has much room for improvement. A recent study shows that less than 30 percent of business deals are accurately predicted.
As mentioned before, the adoption of machine learning tools can somewhat narrow the gap between anticipation and reality. But it doesn’t mean that every company should immediately jump to complex intelligent technology. You can start with small enhancements to your existing system that will address those problems that are difficult to solve by traditional methods. For example, use a machine learning module to make data-driven changes in planning for the short term and leave long-term forecasting to old-school statistics.
However smart your forecasting solution is, the key decisions still rest with human capital. You need industry specialists to define which factors should be considered in your predictive models. Human logic is still required to evaluate the relevance of outcomes produced by digital brains and to make final conclusions, based on common sense and deep domain expertise. That’s why even ML-powered demand planning systems often include a collaborative platform that allows for engaging different specialists in a forecasting process. Only by taking the best of what both artificial and human intelligence offer can you see and plan a better future for your business.