The Age of Big Data: “…Big Data has the potential to utterly transform the relationship that individuals have with institutions, customers with companies, patients with the healthcare system, students with universities, and voters with government. And that means once it has fully penetrated society and industry, the Big Data revolution may very well prove a turning point in our economic – and ultimately, cultural – history as great as the electronics revolution. . . perhaps even as great as the first and second Industrial Revolutions.”
–“Why? Because once the relationship of individuals to institutions transforms, the benefits to the individual consumer, citizen, patient and student will be profound.” (Forbes) http://goo.gl/kxAlq
How a Looming Talent Gap Will Crush Enterprise Hopes for Big Data: “’A lot of companies don’t know how to find data scientists, and don’t understand data science,’ … ‘These enterprise companies can’t implement a proper data analytical solution because they have no data talent.'”
— “Part of the problem is an overall lack of big data skills in the United States. In May 2011, the McKinsey Global Institute laid out the numbers: ‘By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions.’” (ReadWriteWeb) http://goo.gl/SQqyS
Big Data Security Is Inevitable: “There’s been a fair amount of discussion about the fact that security analytics is becoming a big data problem. … If you think that enterprises recognize these trends, boning up on Hadoop, Cassandra, and NoSQL, and hiring data scientists to tag along with security analysts, think again. There’s a growing security skills shortage that will preclude these activities before they even start.” (Network Worlds) http://goo.gl/tME2c
Big Data Holds Big Promise for Government: “Big data has the potential to transform the work of government agencies, unlocking advancements in efficiency, the speed and accuracy of decisions and the capability to forecast, according to a separate report from MeriTalk.”
–“…the Centre for Advanced Spatial Analysis (CASA) at University College London is combining data from London’s Oyster cards – used to pay for public transport – and Twitter messages. Tube-travel patterns are regular: people who enter the system at one station tend to leave it at a particular other one. Twitter messages reveal a city’s structure and its activity.” (Smart Data Collective) http://goo.gl/aqokm
Why are companies spending more on analytics despite cutbacks elsewhere? “Analyst Dan Vesset, author of IDC’s “Worldwide Business Analytics Software” report, credits ‘attention-grabbing headlines’ about big data, rather than the data stockpiles themselves, with helping to put business analytics on the agenda of senior executives. Goodnight seems equally dubious, saying big data is the hot new topic ‘because people got tired of talking about the cloud.'” (InformationWeek) http://goo.gl/feQ2o
All posts tagged: Predictive Analytics
Empower the great the build something greater.
Forbes: The Age of Big Data / A Looming Talent Gap For Data Scientists / Why Companies Are Spending More on Analytics
Analytics and the Nimble Corporation
Brian Sommer at ZDNet has an excellent three part blog series about nimble verses the ossified companies and how each will use big data analytics technology. (Mr. Sommer defines ossified companies cease to develop and innovate and becoming stagnant or rigid in their ways.)
Brian says ossified companies could care less about responding to their market, therefore analytics will be a waste of time and money for them. Brian writes, “They are so rigid in their world view, their processes and business practices that they choose to ignore the very suggestions that could save their firms. They’ve not only ossified, they’ve turned deaf, too.”
What are the characteristics of an ossified company? Brian says they lack “leadership, vision, a continuous change capability and a culture that rewards risk-taking and change over risk avoidance and slavish adherence to ever growing obsolete processes.”
On the flip side of the coin, the nimble companies will live and die by analytics technology because responding to market conditions is part of their DNA.
A nimble firm experiments. Thomas Edison tried something like 6000 attempts at creating a long-lasting light bulb. Edison would have never been allowed 1% of those at most companies. A nimble firm can refine their analytics to isolate experimental results from those of other markets. The insights from these experiments will guide the eventual rollout of game-changing new solutions/processes/etc.
A nimble firm has many current hypotheses about the market. They use analytics to test, prove/disprove and refine these.
A nimble firm, and this is most important, can scale fast. When they see a new market opportunity, they test, refine and then use explosive energy to seize the awaiting market opportunity. These firms can exploit a new market opportunity with incredible speed and precision. They are not only capable of change, but, they can change their entire firm almost overnight.
I highly recommend reading all three posts.
The Ossified Organization Won’t ‘Get’ Analytics (part 1 of 3)
How Tough Will Analytics Be in Ossified Firms? (Part 2 of 3)
Analytics and the Nimble Organization (part 3 of 3)
Is Your Data Valuable?
Consider this, the data you collect about your products could become another profitable revenue stream. Ken Oestreich of Gigaom.com:
When thinking about the value of the data a company collects vs. the traditional value of the product it may produce, collecting and analyzing broad categories of customer + product data is becoming equally — if not more — valuable than the product itself.
Some industry examples:
Grocery and/or consumer retailers keep massive records on purchasing habits of customers, overtly for affinity programs and for making targeted offers. However, this data could be federated with other complementary retailers to establish (geographic, temporal, or correlative) purchasing patterns to increase overall sales per customer. For example, Neilson recently signed an agreement with Walmart where Neilson gets to perform point-of-sale analysis with Walmart data, and (presumably) cross-correlate it with other retailers.
Real estate and geographic data such as from Zillow or Factual can provide core accretion of data value for complementary data-based services. Indeed, such data is available to be crossed and mashed-up for use in healthcare, local government, retail marketing/sales, leisure services, and much more. Consider the value for developing assessing detailed demographics, localized services, etc. However, these business models, both with Zillow and Factual, are of pure data services, rather than a derivative from a “legacy” core business.
So how do you know if your data if valuable? Mr. Oestreich says you should consider the following:
- Uniqueness – is your data unique to you, and therefore hard to replicate by others?
- Size/completeness – how physically large is your data? Does it reach back temporally? Does it include deep details and/or trends? To whom would this be valuable, either pre- or post-analysis?
- Desirability – in addition to uniqueness, would your data be useful/desirable by others in adjacent spaces? How marketable is it?
- Complementarity – consider how valuable your data is to complement or complete other forms of data, or to build-upon other data sets in a mash-up fashion?
- Statistical/correlative relevance – can your data be statistically analyzed for patterns? Are there correlations or patters that might be drawn between it, and other external data sets?
- Long tail relevance – does your data contain elements that lie +/-3 sigma outside the norm? could this “fringe” data be valuable to incrementally increasing sales, or for addressing customer needs outside the core set?
- Data gravity – this new term, coined by Dave McCrory, speaks to the physical immobility of data, and the tendency/requirement to co-locate other applications and data sets with it. Is your data potentially so large that it might actually attract actual applications onto the platform? Perhaps within a special purpose compute-cloud?
You can read the entire post here
Analytics Fights Fires in New York City
When we talk about predictive analytics it’s usually in the context of profitability – where it works wonderfully with products like, ahem…CAN’s very own Pulse. (Sorry, couldn’t help myself.)
But as analytics (or “big data” if you will) becomes more mainstream we find it being used to solve problems for everything from crime fighting, to city planning and even recruiting in Major League Baseball. (more…)
How to Apply Predictive Analytics
Predictive Analytics allows people to make better decisions about how to spend their limited time, energy and money. The potential impact of predictive analytics on business will be similar to the personal computer, relational database and Internet. The power of predictive analytics is that it is a scientific business process improvement method that can be used to model complicated and hard to measure actives, such as why people buy something or which employees are likely to leave. Many business executives understand this potential and are excited about applying predictive analytics to their businesses.
CAN has 4 years of experience helping 200+ companies realize the benefits of predictive analytics. We have developed a 6 stage process for applying predictive analytics to our clients’ businesses that maximizes our clients return on investment, increases their chances for success, and makes sure that the results of our research are applied.
The first stage is to define the company’s mission, vision and values. We want to know why the company was started, and why it exists. We want to know what they want to accomplish in the future. Most importantly we want to know how they do business; what values they have that are unique and perminant even when the strategy changes. This understanding set the priorities and filters that guide future discussions. The second stage is to define the company’s goals. Unlike mission, vision and values, goals have clear beginnings and ends and typically can be accomplish in less than a year. Companies typically have one to three goals. Goals should be in alignment with the company’s vision for the future, and should be accomplished in a way that adheres to the company’s values. The third stage is to define the business question to be answered. The business question is about business process improvement, and should not involve technology or research questions. When answered a business questions should have a noticeable impact on at least one of the three parts of a business; sales, operations and administrative support.
The fourth stage is to determine what resources are available. Resources include political approval, availability of necessary data, and determining research methodology. It is important to note that we only determine research methodology once we have defined the business question. If we don’t have the necessary resources to answer the business question, we go back to stage 3 and try to refine the business question to fit the available resources. It is also during this stage that we determine which of CAN’s resources are best for the client. There are two basic options; custom solution or CAN’s products. If possible we try to answer the business question using one of CAN’s 5 products. This allows us to minimize cost while increasing chances for success. Our 5 products are designed to answer 5 key business questions that the majority of business owners have: 1. Tracker: Who is most likely to purchase my product next? 2. Capture: Where and when should I spend my marketing budget? 3. Pulse: How do I attract and retain my best customers? 4. Beacon: Which employees are most likely to leave and why? 5. Terrain: What sales are likely to be next quarter? If a business question can’t be answered using one of CAN’s products, we offer custom solutions. Many of CAN’s clients leverage our custom solutions to develop a competitive advantage in predictive analytics. When developing custom solutions it is essential that we become apart of our clients team and fully understand their business, goals and resources. Before committing to any custom projects, CAN requires that we build a proof of concept. The purpose of the proof of concept is to make sure that we fully understand what we need to build and that we have all the necessary resources. The fifth stage is to determine how the models will be implemented. While our research is complex, we make sure that our work is easy to understand and use, because that is how it gets implemented. We use 4 methods to implement our research, and often combine multiple methods depending on what the client’s goals are.
1. Formal Reports help our clients understand the nuances and details of our research. Formal reports are most useful when the results of our research will influence a company’s strategy, will be used by a small and specialized audience, and frequent updates are not required. 2. Marketing Summaries provide our clients with colorful and easy to understand summaries of our research. Many of our clients use these summaries as marketing pieces to communicate quickly with large and unspecialized audiences about research that impact future strategy. Marketing pieces go beyond executive summaries because they can be used be used to inform executives, employees, customers and the community. 3. Dashboards help our clients quickly get the up to date information that need to run their operations. While formal reports and marketing summaries often include data visualizations, dashboards are unique because they can be quickly updated and display key information on a single screen that can be monitored at a glance. Dashboards are typically used by a small and specialized audience that is trained to understand and use the information on the dashboard. Dashboards can also be very useful for sensitive information, because administrators can control access by user on a need to know basis. 4. Workflow Integration provides our clients with the ability to use our research to impact the activities and operations of large number of people through the systems they are currently using. Workflow integration is useful because users don’t have to learn or get in the habit of using a new system. The predictive models and coefficients that CAN develops act as a filter to the current database and either users are presented with familiar data fields or if need new fields. The method that we choose depends on who the audience will be and how they will use the results. The fewer the people that need access to our research the more important security and control becomes. If the use of our research is for strategy development then we typically publish a formal report. However if the use of our research is to optimize operations then we publish it as a dashboard, marketing piece, or integrate it into the software you are currently using. The sixth step is to evaluate the model. As part of developing models we run tests to make sure that they are statistical robust. However, it is important to further evaluate a model before and after we implement it. The first evaluation criteria is does the model answer the intended business question. The second criteria is does the model produce results that reflect reality. While the model might be statistical robust, it is useless if it produces misleading results that the experts in your business know aren’t true. The third evaluation criteria is once implemented does the model produce the expected results. Predictive analytics is a very new field. While the technology is exciting, it is predictive analytics ability to answer hard to answer or previously impossible to answer business questions that is most exciting. What separates CAN from our competition is our focus on making sure that we answer our clients’ business questions, instead of being enamored by the technology. Our hope is that we can help our clients apply predictive analytics to their businesses, and that our 6 stage process helps them maximize their return on investment, increase changes for success, and makes sure that the results of our research are applied.
CAN Navigators Help You Work Smart
CAN’s goal is to use predictive analytics and data science to help our client’s work smart. While predictive analytics and data science can help a company generate more profit, the technology can be very complicated and the process very cumbersome. CAN is committed to making predictive analytics and data science simple. Simple systems get added, and our systems are only valuable when they are implemented.
CAN’s Navigator program is key to making predictive analytics simple. We developed the Navigator program, because in addition to great software, data visualization and reports, wanted to provide our clients with great customer support. Navigator’s are here to help introduce you to CAN, predictive analytics and data science, understand your business, create a plan so that you can work smart, and then work with you and CAN data scientists to implement the best solution for you. (more…)
Why Jefferson Decided to Join CAN
Jefferson joined CAN before we had this blog, our website, our products, or our office at 1209 Harney St. This video is about how and why he decided to join Contemporary Analysis. He knew we had potential and decided to become apart of CAN’s future and the future of data science. CAN specializes is predictive analytics. Predictive analytics involves collecting data about your business and customers, and then applying theory and math to build simple systems to help you work more effectively and efficiently.
Our systems are tailored to fit your company no matter how big or small or what industry you are in. We have built simple systems for fast-growing technology companies, Fortune 500 companies as well as small companies in a variety of industries including community colleges, insurance companies, software companies and engineering firms
Tadd and Jefferson go Mining for Data in Wyoming
CAN is helping one of our clients improve their asset management strategy, by building predictive models to determine when heavy equipment is most likely to fail.
CAN’s asset management models will allow our client save hundreds of thousands of dollars each year, by converting emergency repairs into scheduled maintenance. Imagine the money and time that can be saved if repairs can be preemptively made in several hours instead of the weeks or months it takes to make repairs in the field.
While we could have developed the model from our offices in the Old Market, we needed to make sure that we understood the conditions on the ground. Jefferson and Tadd decided to take a trip to Wyoming and spend a week learning about the machines and interviewing the experts that use the equipment on a daily basis.
Their goal was to make sure that we had political support from the people that were going to use our models, and that we could build balanced models that combine data, theory and math. The following are some of the photos from their trip. I hope you enjoy.
Mining for Data
We might push paper for a living, but we love to get our hands dirty to build beautiful models and to understand your business! Please contact us to learn how we can help you.
Using Mean Absolute Error for Forecast Accuracy
Using mean absolute error, CAN helps our clients that are interested in determining the accuracy of industry forecasts. They want to know if they can trust these industry forecasts, and get recommendations on how to apply them to improve their strategic planning process. This posts is about how CAN accesses the accuracy of industry forecasts, when we don’t have access to the original model used to produce the forecast.
First, without access to the original model, the only way we can evaluate an industry forecast’s accuracy is by comparing the forecast to the actual economic activity. This is a backwards looking forecast, and unfortunately does not provide insight into the accuracy of the forecast in the future, which there is no way to test. Thus it is important to understand that we have to assume that a forecast will be as accurate as it has been in the past, and that future accuracy of a forecast can be guaranteed.
As consumers of industry forecasts, we can test their accuracy over time by comparing the forecasted value to the actual value by calculating three different measures. The simplest measure of forecast accuracy is called Mean Absolute Error (MAE). MAE is simply, as the name suggests, the mean of the absolute errors. The absolute error is the absolute value of the difference between the forecasted value and the actual value. MAE tells us how big of an error we can expect from the forecast on average.
One problem with the MAE is that the relative size of the error is not always obvious. Sometimes it is hard to tell a big error from a small error. To deal with this problem, we can find the mean absolute error in percentage terms. Mean Absolute Percentage Error (MAPE) allows us to compare forecasts of different series in different scales. For example, we could compare the accuracy of a forecast of the DJIA with a forecast of the S&P 500, even though these indexes are at different levels.
Since both of these methods are based on the mean error, they may understate the impact of big, but infrequent, errors. If we focus too much on the mean, we will be caught off guard by the infrequent big error. To adjust for large rare errors, we calculate the Root Mean Square Error (RMSE). By squaring the errors before we calculate their mean and then taking the square root of the mean, we arrive at a measure of the size of the error that gives more weight to the large but infrequent errors than the mean. We can also compare RMSE and MAE to determine whether the forecast contains large but infrequent errors. The larger the difference between RMSE and MAE the more inconsistent the error size. The following is an example from a CAN report,
While these methods have their limitations, they are simple tools for evaluating forecast accuracy that can be used without knowing anything about the forecast except the past values of a forecast.
Finally, even if you know the accuracy of the forecast you should be mindful of the assumption we discussed at the beginning of the post: just because a forecast has been accurate in the past does not mean it will be accurate in the future. Professional forecasters update their methods to try to correct for past errors. However, these corrections may make the forecast less accurate. Also, there is always the possibility of an event occurring that the model producing the forecast cannot anticipate, a black swan event. When this happens, you don’t know how big the error will be. Errors associated with these events are not typical errors, which is what RMSE, MAPE, and MAE try to measure. So, while forecast accuracy can tell us a lot about the past, remember these limitations when using forecasts to predict the future.
To learn more about forecasting, download our eBook, Predictive Analytics: The Future of Business Intelligence.
Interested in seeing how we can help you with forecasting?
Featured Posts – Click the Brain
