When should you Update Predictive Models?

New clients often have questions about why and how frequently CAN needs to update their predictive models.  Predictive models need to be updated because everyday new data is being created.  For example, your customers are buying more, subscribing or unsubscribing.  The environment is constantly changing.  While predictive models can handle a lot of new new data, overtime environmental changes build up causing predictive models to lose their effectiveness.  After a month, quarter, or a year it is necessary to update predictive models with new data.
As these new patterns emerge its important to periodically take time to investigate your data, update your models, and challenge your assumptions about your business. But how often should you do this?

To answer that question, consider the following:
– How often is my data changing?
– How often do I plan on making decisions with the data?
To understand how often your data is changing, its necessary to understand what type of data you are gathering and how often. The average number of customer transactions per year can be a good place to start. For example, if on average your customers purchase from you once a month or once a quarter, it may be possible to identify new patterns in their behavior on an annual basis, which would only require you to update your models and assumptions annually. Models primarily based on customer demographics are good examples of this because a customer’s demographic profile tends to change little over time, often times not even annually.  Demographics include gender, age, income, geography, etc.
Contrastingly, if your customers have more frequent transactions (i.e. daily, weekly, bi-weekly), your customer data is growing quickly in size and complexity. The patterns created as a result of this new data are also more dynamic.  This means you need to update your models and challenge your assumptions more frequently.  Often this is monthly or quarterly.
Knowing how often you plan to make decisions with your results also bounds some of the uncertainty around updating your predictive models. For example, if it only makes sense to focus marketing efforts or strategic initiatives quarterly, quarterly updates are sufficient.  However, if you or someone in your company uses the outputs daily, weekly or monthly, you need to update your predictive models monthly.  In extreme cases were new data is created by the minute, such as commodities trading, daily updates are required.
Finding a happy medium is what we look for when making suggestions to our clients, but regardless its really up to your discretion. However, I plan on following up with another blog post soon discussing in detail what happens when models become obsolete and start to lead you astray.