Getting Value From Your Data Series: The Road May Be Rocky But It’s Well Worth the Effort!

Cross-posted from the PatternBuilders blog with permission.
By Marilyn Craig and Mary Ludloff

roadworkUnless you’ve been asleep for the past couple of years, you, like us, have heard this phrase again and again:  Data is the new oil.  It certainly sounds great but what exactly does it mean? Here’s our take: Getting the most value out of your data can make you better at what you do as well as enable you to do more with what you have. In other words, there’s unrealized value in those data silos that all companies have. But make no mistake: the road to realizing data value is paved with good intentions and often times, poor execution and results.

oil-drillsToday, most companies are drowning in data—there’s historical data from operations, data from public sources, data from partners and acquisitions, data you can purchase from data brokers, etc.  These companies have read all the research and want to leverage their data assets to make “better” operational decisions, to offer their existing customer base more insights, to pursue new revenue opportunities. Of course, the real value in that data is derived from the business analytics that deliver the insights that drive better decisions. As we’ve said quite often on this blog: Data, without the proper use of analytics, is meaningless. If data is the new oil, think of analytics as the oil drills—you need both to be successful. Continue reading

Lessons Learned from the Google Flu Tracker—Why We Need More Than Just Data

Cross-posted from the PatternBuilders blog with permission.
By Marilyn Craig and Mary Ludloff

God we trustWe read an interesting paper and post about Google Flu Trends (GFT) and its foibles last week. The paper points out a couple of lessons that those of us living in the big data analytics world have learned the hard way but the dangers are worth revisiting as tools like ours (AnalyticsPBI for Azure) begin to move big data analytics into the mainstream of organizational practices. After all, our tool (and others like it) makes it easy and even fun for analytics junkies to use all those available zettabytes of data and answer questions that they’ve long wondered about. But the paper also reminded us of the dangers of ignoring the natural cycles of an analytics process that we talked about in this recent post. If Google followed the PatternBuilders Analytics Methodology, they might have avoided many of the errors that GFT is now spitting out. In fact, the authors of the paper point out that:

“Although not widely reported until 2013, the new GFT has been persistently overestimating flu prevalence for a much longer time. GFT also missed by a very large margin in the 2011-2012 flu season and has missed high for 100 out of 108 weeks starting with August 2011… This pattern means that GFT overlooks considerable information that could be extracted by traditional statistical methods.”

This overestimation is attributed to two primary factors: data hubris and algorithm dynamics. Continue reading