Author: Andy Hayler, Co-Founder and CEO, The Information Difference
Master data management (MDM) and Big Data are both fashionable topics at present in the world of enterprise software, but how do they relate to one another? Large global organizations struggle to get a single, consistent view of their master data such as customer or product even in the world of conventional data. Big Data refers to volumes of data that relational databases struggle to handle, at least in an economic manner. Such data may be machine generated, such as sensor data, or may be customer related, such as the traffic from web sites or social media. In either case it can be seen that it would be desirable to link such data up to a company’s products and customers rather than dealing with it in isolation. For example, identifying an issue with your customer web traffic is one thing, but you really want to be able to track the issue back to the right customer account or accounts, which is where master data comes in.
In December 2013 The Information Difference carried out a survey to try and understand what was really happening on the ground. 179 corporations (mostly from Europe and the USA) shared their experiences with Big Data, with some interesting results. Over a quarter of respondents claimed to have a live Big Data application, with “customer” the single most popular data domain being addressed, “product” a close second, followed by “financial data”. Almost two thirds said that their Big Data initiative was focused on a mix of structured and unstructured data, with nearly a third concentrating on structured data. Just under half had a business line driving things, though in a third of cases the IT department was in charge of Big Data.
Fully 45% of respondents regard Big Data as an “integral part” of broader data management initiatives including MDM, with only 16% regarded it as a stand-alone activity. This is encouraging, suggesting that most organizations are taking a joined-up view of their data management.
There were two quite surprising results from the survey. One was that there were significantly more live Big Data applications from European respondents than the US ones, which flies in the face of conventional wisdom on the subject. Another was that, just as beauty is in the eye of the beholder, “Big Data” is not always that big as far as customers are concerned. Only 21% were handling data over 100 TB in size, and just 5% over a petabyte. Over a third were tackling data of 10TB or less within their Big Data applications, data volumes that should be well within the capability of conventional databases. It may be that some companies are looking at technologies such as Hadoop from an economic viewpoint i.e. as an opportunity to cut costs, rather than purely as a response to giant data volumes.
Big Data appears to be still in its infancy, with just 16% of survey respondents having spent more than $1 million so far on their projects, though doubtless this will rise in due course. At this stage it is clearly early days, though it is encouraging that many organizations are positioning Big Data within their broader data management programmes, taking a strategic rather than a tactical approach to the subject.
About the author:
Andy Hayler is one of the world’s foremost experts on master data management. Andy started his career with Esso as a database administrator and, among other things invented a “decompiler” for ADF, enabling a dramatic improvement in support efforts in this area. He became the youngest ever IT manager for Esso Exploration before moving to Shell. As Technology Planning Manager of Shell UK he conducted strategy studies that resulted in significant savings for the company. Andy then became Principal Technology Consultant for Shell international, engaging in significant software evaluation and procurement projects at the enterprise level. He then set up a global information management consultancy business which he grew from scratch to 300 staff. Andy was architect of a global master data and data warehouse project for Shell downstream which attained USD 140M of annual business benefits.
Andy founded Kalido, which under his leadership was the fastest growing business intelligence vendor in the world in 2001. Andy was the only European named in Red Herring’s “Top 10 Innovators of 2002”. Kalido was a pioneer in modern data warehousing and master data management.
He is now co-founder and CEO of The Information Difference, a boutique analyst and market research firm, advising corporations, venture capital firms and software companies. He is a regular keynote speaker at international conferences on master data management, data governance and data quality.
He can be contacted via email: Andy Hayler