Articles

Master Data Management (MDM) has been around since the mid-1980s, but has really come to the fore in the last decade, with many of today’s data governance efforts built on top of existing MDM strategies. This has been driven by the advent of Big Data, an increased focus on Business Analytics and Intelligence, and growing adoption of Machine Learning and Artificial Intelligence.

For the past 25 years or so there have been no major leaps in how providers have built or provisioned their MDM offerings. Traditional MDM solutions still require you to implement strict controls over every aspect of your master data management process—from data acquisition to data storage, and from maintenance and modification to security and access control. These systems were built for the on-premises, siloed institutions of the past where data ownership lay almost exclusively with the IT department.

Modern approaches are more aligned to how most enterprises operate today - in a hybrid, highly distributed and fluid fashion. Data is a valuable business asset, which means that technology and business users are equally responsible for its maintenance and use. This does not mean that everyone in the business needs to be a data engineer or architect. What it does mean is that everyone is, to some extent or another, a data steward and a data citizen. It is the job of technology to enable these roles and ensure that everyone with a stake in an organisation's data benefits from its potential. Which is where Modern MDM comes in.

What is Master Data Management?

At a fundamental level, Master Data Management (MDM) is the process of creating and maintaining a single, consistent view of your organization's critical data. MDM is closely related to data governance, which can be thought of as rules for how data is collected, processed, stored and accessed. It also includes policies on how data should be handled, such as how long it should be retained and what access permissions are granted to different groups of people or individual data owners.

Master data is the set of identifiers that provides context about business data. The most common categories of master data are customers, employees, products, financial structures and locational concepts. Master data is different to reference data, which is data that refers to other objects, but does not contain identifiers that represent different types of master data entities. Whether there is still a need for reference data in the context of what can be achieved with modern MDM is debatable, but that's a discussion for another time.

What's the problem with traditional Master Data Management solutions?

It has been estimated by Gartner that up to 85% of MDM projects fail. That's a big number. Little wonder then that so many organisations have been burnt in the past and aren't exactly falling over themselves to start another MDM initiative.

There's a number of reasons why this number is so high:

1. The upfront planning process - data profiling, analysing and modelling is time consuming and expensive. Many traditional MDM projects take over a year to deliver any ROI at all.
2.  A domain-by-domain approach, such as that used by traditional MDM systems, causes complexity and creates new silos, restricting how the data can be used.
3. Traditional MDM demands high manual and technical intervention, which is both costly and time-consuming.
4. Because traditional MDM systems are built on relational databases with only direct relationships, connections are manual and add to the maintenance overhead.
5. Due to the upfront profiling and modeling requirements, you're always playing catch-up with your data as it changes. This adds to the complexity and need for manual intervention, further delaying projects.

In spite of all of the above, the fact remains that businesses need to be able to use their data to fuel the projects that will move them forward. Whether these are customer, product, supplier or employee focused initiatives, they all rely on data to provide insights to inform them. At the moment, many organisations are using their data in this way, but the data is neither consistent nor reliable. Which means that the results and recommendations aren't trusted either.

The modern approach to Master Data Management

Modern MDM seeks to solve the above issues in a number of ways.

• By managing all of your data - master, meta, reference, structured and unstructured. Suddenly, the potential use cases for your data have multiplied exponentially.
• By eradicating the need to model your data upfront. Modern MDM embraces data in its "raw" form from hundreds, if not thousands, of data sources. The potential cost and time savings are huge.
• By removing repetitive and manual tasks from the outset. Automating manual tasks like data cleaning reduces the burden on the client and frees time and resources to work on value-orientated tasks instead.
• By being truly Cloud-native. Most traditional MDM platforms were not born in the Cloud, they were built for an on-premises, highly structured environment and then tweaked for the Cloud. Modern MDM platforms were built for the Cloud - which means that getting up and running is quicker and easier, you can scale up or down at pace, and you benefit from the Cloud economic model.
• By providing proactive data governance. Establishing trust in data means having full visibility of its lineage and controlling what happens to your sensitive data in a transparent way. Meeting compliance requirements and demonstrating how data is protected won’t slow you down anymore.

You may be wondering what is so different about modern MDM systems that makes all of the above possible. One major difference is that modern MDM systems like CluedIn are built on a NoSQL, schema-less database called Graph. In the world of Graph, the relationships between the data are as important as the data itself.

A really simple way to think of it is similar to the difference between organising your data into neat rows and columns in Excel versus jotting it down on a whiteboard. With the whiteboard you can visualise the relationships between the data and add the connections as they emerge. This is exactly what Graph does - as the data is ingested, it allows the patterns and relationships to surface, and is then able to organise it into a natural data model. LinkedIn, Facebook and Google are all built on Graph, and the same principles of schema-less, scalable modeling now apply to MDM.

What does the future of Master Data Management look like?

In many ways, the future of Master Data Management doesn't look like Master Data Management at all. Where traditional MDM systems were siloed and slow, modern platforms are integrated and quick. Where the old way of approaching MDM dictated set rules and structures, the new way embraces freedom and flexibility. And if we accept that these concepts shouldn't only apply to Master data, but all data, then the concept of Master Data Management becomes almost entirely redundant.

At this point in time, CluedIn is the only MDM platform that uses Graph. This will change as established vendors and new market entrants recognise how powerful Graph can be when applied to the management of business data. And that's a good thing. Right now, forward-thinking businesses that want to use their data to react to market forces, competitive advancements and customer preferences have a very limited choice: traditional MDM or CluedIn. As the market continues in this direction, a new category will emerge and we will no longer talk about traditional or modern approaches to MDM. In fact, there's a very good chance that by that stage, we won't be talking about MDM at all.

The term “data-driven business” refers to an organisation that uses data to inform or enhance decision-making, streamline operational processes and ultimately to fuel revenue and growth. Whether or not it is possible for any business to be solely data-driven is another debate, but there is no doubt that those who get close to it are adept at turning data into insight, and at using that insight to propel the business forward. While most companies today would probably cite becoming data-driven as a crucial enabler of their wider goals, there aren’t many that have achieved it. Google, Facebook, McDonalds and UBER definitely fall into this group, but these are industry heavyweights and represent the exception rather than the rule.

What does that mean for everyone else vying to achieve data-driven status? Like many things in life, it starts with the basics and builds from there. Even the big boys had to start somewhere!
All truly data-driven businesses have something in common, aside from the obvious operational and competitive advantages. They can all answer six vital questions.

• What data do we have?
• Where is the data?
• What is the quality of the data?
• Who owns the data?
• Who is responsible for each step of the data journey from start to finish?
• What happened to the data as it transitioned from raw to insightful?

Why is it even important for you to be able to answer these questions in the first place? There are the obvious compliance and regulatory reasons why you should, but for now let’s focus on what your business could achieve if you had the answers to these questions.

What data do we have?

Once you have experienced one win as a result of seeing data really work for you, you’re hooked. This could be using data to optimise processes, lower operational costs, find more customers, attract great talent, monitor trends in the market and much more besides. Knowing what data you as a company have in your arsenal is the first trigger to inspiring these types of insights. Insights can come from manual discovery, or can come from using technology to find patterns in the data and bring them to your attention. We believe in being able to walk before you can run and it is not necessarily a bad thing to start gaining insights through manual discovery.

For example, if you have a list of customers and a list of support tickets, you might want to know which geography causes the most support tickets. With a pattern-driven approach, it is not so much about asking the questions of the data, but rather about allowing the data to reveal interesting trends. The likelihood is that there will be patterns hidden in the data that you would not proactively ask for – e.g. churned customers took over 54 hours to have their support tickets resolved. This insight may then lead you to hire more customer support representatives to bring down the average answer rate or have an internal SLA that no ticket takes more than 24 hours to answer.

Where is the data?

Knowing where the data is and where it has come from is an important regulatory requirement, but in the context of achieving some type of insight, knowing the answer to these questions is vital to establishing trust in the data from across the organization. If someone on the street handed you a credit card and said "Feel free to use this!” the first thing you’d probably ask is where it came from. Without this lineage, there is no trust. And most notably, in this analogy, you would want to know if the source of this credit card is reputable. 

Also, although duplicate data is not necessarily a huge storage cost issue anymore, it is a big operational issue. Of course, this also depends on exactly how much duplicated data you have – petabytes of it can be quite costly! Which also means that knowing where your data is can help you to reduce operational costs too.

What is the quality of the data?

In the era of fake news and AI bots that are indistinguishable to humans, it is more important than ever to establish integrity in the data you are using to make decisions. There are a plethora of shades of data quality, and every shade will correlate with a different level of confidence in the "usability" of the data. It should also be pointed out that there is no such thing as right or wrong when it comes to data, and no matter how high quality the data is deemed to be it will bring with it an inherent level of risk. 
In the spirit of keeping things technology-agnostic and high-level, think about the times you have made a decision with confidence. What gave you that confidence? Was it that your research came from a reputable source? Was it because the voice of the crowd all agreed with one approach? Was it your gut feeling?

Just like everyone else, you probably make decisions on a daily basis using a combination of these techniques to make your final judgement. It’s much the same with data - determining quality is about building up your confidence in making a decision. The challenge with data is that it doesn't have to adhere to any laws of physics, hence any judgement made on data quality is a heuristic attempt to provide metrics on which a decision can be made with an acceptable level of confidence and risk. You can read more about how CluedIn interprets and measures the shades of data quality here.

Why does data need ownership?

In many ways, it doesn’t. In fact, it needs much more than ownership. This is why we have frameworks in Data Governance like the RACI model, in which the four dimensions of "ownership" are defined as the minimum requirements for an ownership matrix relating to data and journey that data takes. Like any process you have within a business, if no-one is responsible for it, it often grinds to a halt. As you have probably experienced in other parts of your business, sometimes a task can be blocked by the most minuscule reason, but the bottom line is - it was blocked. This is often down to a lack of ownership for that part of the process. 

Who is responsible for each step of the data journey from start to finish?

The data journey from source to insight has some very distinguishable steps, and each of these steps requires you to attack the data from a different angle. Irrespective of the  technology you use to get from source to insight, the generic journey includes pulling data from a number of sources, integration, normalisation, standardisation, deduplication, linking, enrichment, profiling, mapping and transformation. (Honestly speaking, we could easily add another 10 or 15 stages, but let's stick with this list for now!). In many cases, each of these steps is a comprehensive task and responsibility in its own right. For example, the normalisation and standardisation of data is easily a full time job for many data stewards. Hence, if a full supply chain of ownership of the steps in the process is not established then it should not be a surprise that the flow of usable data can break down – often for the most mundane of reasons.

What happened to the data as it transitioned from raw to insightful?

Let’s consider for a moment why it is that data needs lineage, and different parties to take responsibility for the entire data journey, yet other processes we run within the  company don't demand the same level of stringent needs? Could it be that this lineage would actually be very useful in all parts of the business, but because of the digital nature of data it is inherently easier to build a digital footprint? The same cannot be easily said for passing around Excel sheets from department to department, for example. Any explanation of how this Excel sheet "came to be" simply isn’t something that can be achieved simply through the use of Excel. The audit trail of the transformation of data from source to insight is often just as useful for “explainability”  as it is for highlighting parts of the process that can be improved or are error-prone.

Summary 

Now that we have established the questions you need to answer in order to start your journey to being truly data-driven, we should look at how technology can help you to both answer the questions and use those answers to best effect. The best way to do this is to approach it from both the asset and the record level – which in effect means getting both the birds-eye and granular view, and bringing them together in a way that makes sense. One powerful and increasingly popular combination is to use Microsoft Purview and CluedIn. To some degree, both Purview and CluedIn answer all of the questions above, but at different levels. The bottom line is, you need both and in some ways, you can't have one without the other, particularly if your data technology stacks are all housed within Microsoft Azure.

High quality, trusted data is the foundation of Machine Learning (ML) and Artificial Intelligence (AI). It is essential to the accuracy and success of ML models. In this article, we’ll discover how CluedIn contributes to driving your Data Science efforts by delivering the high quality data you need.

CluedIn not only provides your teams with tooling that improves the quality of the data that is fed to your ML models, it also simplifies the iterations by which you can evaluate their effectiveness.

The five Vs of Data Quality

The term “data quality” is overused and can mean many things. As Machine Learning and Big Data are still both evolutionary fields with developments in each complementing the other, we’ll approach it from an angle you may already be familiar with – the five Vs of Big Data (Volume, Variety, Velocity, Value and Veracity).