Impress With Data Mesh

Data mesh is the nudge that puts us on a new trajectory in how we approach data: how we imagine data, how we capture and share it, and how we create value from it, at scale and in the field of analytics and AI – Zhamak Dehghani

There is a lot of excitement (and information) around Data Mesh at present; What is it? How can I benefit? How do I get data meshed and should I?

Well, they are very good questions. Just because it appears to be the next exciting development in data and will solve all my problems. Will it really?

We’ve been through a few interesting ‘next silver bullet’ developments in data over the years.

Data science (and Big Data)….it’s not that mysterious black box to generate magical opportunities to solve all your problems, though I think some Data Scientists would still like you to believe that. It does however, when done correctly provide insights into the opportunities available to operations and serve your customers better.

The pillars of the data mesh paradigm

Data Catalog….having a view of where all you data is and how it is used is critical. Particularly with compliance responsibilities and helping your Data Scientists find the most appropriate data more easily, is exceptionally useful. Fancy going to Amazon and not having any search or categorisation. How would you find those new Bluetooth headphones you want…you’d probably give up after a while of sifting through loads of products like a jumble sale. You can’t implement a Data Catalog and think your Data Management actions are complete or that’s all you need to do to obtain great insights!

Then today there is Data Mesh & Data Fabric. I originally wrote about the benefits of ‘Big Data Fabric’ in Aug 2017. Not much has changed in the concept since then. However, the technology to enable Fabrics and Meshes have evolved to a point where it’s now a realistic benefit.

The Basics

A data fabric and a data mesh both provide an architecture to access data across multiple technologies and platforms, a data fabric is technology-centric, while a data mesh focuses on organisational change.

Data fabric connects data from multiple sources and prepares it in a way in which you – a business user or a data professional – can access and analyse it easily. Think about it like putting a big blanket around your architecture to easily access what you need from the outside of the blanket.

According to Gartner, data fabric is a design concept that serves as an integrated layer (fabric) of data and connecting processes. It is a composable, flexible and scalable way to maximize the value of data in an organization. It’s not one tool or process, rather an emerging design concept that gives a framework to think about how to stack existing tools, resources, and processes.

Whereas, the data mesh paradigm stands for decentralised and domain-specific data ownership that is easily discoverable and ready for consumption for everyone in the organisation via a Marketplace.

Data mesh moves away from the concept of storing, transforming, and processing analytical data centrally. Instead, it advocates that each business domain is responsible for hosting, preparing, and serving their data to their own domain and larger audience. Don’t worry you don’t need to be highly technical to be, or remain a Data Owner. Data Ownership sits firmly under the business responsibilities and Data Mesh reinforces that.

Several companies have publicly presented their data mesh strategy – here are just a few:

Intuit’s breakdown of their data mesh strategy https://medium.com/intuit-engineering/intuits-data-mesh-strategy-778e3edaa017

JPMorgan Chase shared their experiences on how to get started with data mesh implementation https://www.dremio.com/subsurface/implementing-a-data-mesh-architecture-at-jpmc/

HSBC uses data mesh in their data strategy https://www.assetservicingtimes.com/assetservicesnews/dataservicesarticle.php?article_id=10675

Learn about data mesh deployments from the awesome companies listed https://datameshlearning.com/user-stories/

Choosing a fabric or a mesh essentially boils down to the problem your organisation is dealing with.

Is A Data Mesh Right For Me?

Here I take a look at the pros, cons and considerations of a Data Mesh.

Pros

Reinforces the Data Ownership model as Data Product Owners report into Data Owners in the Business, not IT. Data Owners drive Data products within a Data Mesh. IT or Technical experts serve the needs of the Product Owner and the business

Domain teams are in the driving seat of their own Data Products to explore more scenarios

Data Culture matures as each domain is invested in the quality of their data as it directly impacts their domains analytical and insights generation abilities

It is easy to identify what data is available for reuse in the Marketplace

Cons

If you don’t have a strong Data Ownership culture, it’ll be very difficult for a Product Owner to operate without the Data Owners direction

Having a Data Mesh doesn’t negate the need for robust Data Management, infact it necessitates the need as more critical

A central team developing Data Products could lead to a backlog, whereas distributed Domain teams could lead to under utilised resources

That investment you have made in creating data products, yet they are not used and colleagues still retain the use of their spreadsheets of siloed data. If the Data Products are not used your investment is a *White Elephant. This is why this must be a culture change not a build it and they will come….they won’t!

If sufficient data product maintenance is not in place the data product will become out of date. Another reason why they won’t come to your party without sound reassurance of effect maintenance

The pull for a Data Mesh approach needs to come predominantly from the business. The IT approach of if we build it they will come, won’t work with the culture change needed for these new business responsibilities and ways of working

Things To Consider

Data Product maintenance workload, should not be underestimated

Separate capabilities and technology for Data Management is required as Data Mesh does not provide Data Management for good solid foundations for your data

How will you measure adoption and usage, and track value of the utilising of the domains data products by other teams and attribute success back to the data domain owning the data product to demonstrate success?

Will your take a central or Domain developed Data Product development team and how will the skills and capabilities available dictate this in the short term?

Security around access to the Marketplace and Data Products needs to be carefully defined with the correct guardrails for what each function will be doing with the data for example utilise adaptive governance with those in a sandpit testing out innovation needing less governance than those creating external facing data solutions

Does the business understand data mesh and data products?

Is the Data Owner ready to run a pod team to create data products?

How to stop business focusing on routine and start considering the art of the possible

Imagination will often carry us to worlds that never were. But without it we go nowhere – Carl Sagan

Decentralise your domains slowly and carefully with a good level of change management support and data literacy….remember to bring people with you

The need of a data catalog to show what data is available

A data marketplace is needed to share data and the place where you would go to access the Data Products – This needs to be a virtual copy of the data product otherwise data is duplicated and cloud storage costs will spiral – my understanding is that the *hyperscales are still working on a solution for this, with Snowflake being the most mature in their thinking for a solution

Assessing the maturity and level understanding of the business to accept new ways of working with data, will considerably affect whether you are ready for Data Mesh. Take your time , there is no rush. Get good solid foundations in place first.

Data Mesh is not a silver bullet!

Data Science / Data Catalog / Data Mesh….they all have their place and benefits, none are that magic wand solving all your data challenges. The foundations for good quality, accurate, consistently understood and utilised data with the appropriate level of Governance and security remain critical to get in place before any other activities. I include transformation and migration projects in that bucket of, before any other activities.

Getting data right takes determination, commitment, investment in people, technology and time and mostly leadership to bring your colleagues with you on the journey to ‘Get Data Right’ and utilise it to generate value.

A few links to deepen your understanding and knowledge:

https://atlan.com/data-fabric-vs-data-mesh/

https://youtu.be/l_3RyxsoZks  101 of Data Mesh

https://atlan.com/data-fabric-vs-data-virtualization/

*The term white elephant refers to an extravagant, impractical gift that cannot be easily disposed of. The phrase is said to come from the historic practice of the King of Siam (now Thailand) giving rare albino elephants to courtiers who had displeased him, so that they might be ruined by the animals’ upkeep costs.

https://en.wikipedia.org White elephant gift exchange – Wikipedia

*Hyperscalers –

What are Hyperscalers? This term stems from hyperscale computing, which is an agile method of processing data. Depending on data traffic, scale can quickly go up or down. Hyperscalers have taken this computing method and applied it to data centers and the cloud to accommodate fluctuating demand. Examples of hyperscalers are Amazon AWS, Microsoft Azure, Google GCP, Alibaba AliCloud, IBM, and Oracle.