Starting fresh with Data Mesh

TechRadar 2023 Hot Topic – Data Driven Intelligence

This article is a selection of the 2023 Devoteam TechRadar, it’s designed to introduce the context of the chapter by covering a hot topic in the industry.

This year’s focus is on the Cloud Native era – a new era where technology and business are more interconnected than ever. Companies must prepare for disruptive models built with, in, and for the cloud. The technologies in TechRadar 2023 are mostly all participating in this movement.

From Business Intelligence to Data Mesh

Looking for a way to gain a detailed understanding of the relationships between your data? Enter Data Mesh. Based on the organisational strategies for scalability of large-scale software development, Data Mesh is a socio-technical approach that enables an organisation to scale with Data.

Large-scale data platforms face 3 challenges: The first is the use of centralised monolithic data architectures to respond to omnipresent data from operational systems. Secondly, hyper-specialised tools need a hyper-specialised staff. This creates hyper-specialised silos between data generation and consumption, generating operating friction. Lastly, centralised silo models disconnect the process causing consumers and platform providers to experience the most difficulty here. Either they have trouble bootstrapping large platforms or growing and differentiating.

So, what if we broke down this monolith around the concept of decentralisation and domains instead of based on pipeline and pipeline stages? 

Data Mesh fixes the problems with data warehouses by giving data owners more freedom and flexibility. This makes it easier for data owners to experiment and come up with new ideas, and it makes it easier for data teams to meet the needs of all data consumers through a single pipeline. Data Mesh is based on 4 main pillars namely, Federated governance, Decentralised Domain Ownership of Data, Data as a Product, Self-serve Infrastructure as a platform.

Devoteam’s Recommendation to get started on Data Mesh

Data mesh brings solid foundational solutions for a company to scale Data, as it’s already been done in software development. We anticipate that “embracing data mesh” should replace “having a data mesh” as the default phrasing for this concept in the near future. 

The implementation of data mesh should be a long-term goal and not all components of each pillar must be implemented immediately, but eventually. This will enable all employees to have access to the data whenever they need it. But, merely adhering to the pillars does not guarantee its implementation. Many organisations are deploying step by step practices according to Data Mesh pillars.

Data Mesh has many potential uses, including:

• Agility and scalability comes hand-in-hand through the use of data mesh as it works to support decentralised data operations to aid in the reduction of the amount of time needed to bring a product to market, increase scalability, and improve business domain agility.
• Facilitation of query execution is possible across multiple data sources for developers and DevOps teams through a self-service model
• Introduction of a universal, domain-agnostic, automated approach for data standardisation by data teams
• Cross-team transparency that is fostered by decentralised data ownership and shared across specialised groups working in different areas.

Since a data mesh is fundamentally a method of organising, it is not a product that can be purchased. However, technology is crucial since it enables data mesh, and only practical and straightforward solutions will win over domain teams. You can build a data platform for your data mesh using the existing offers of cloud providers, which include a sufficient set of strong self-serve data services.