In this post we explain what data mesh is, the advantages of applying it and a real example from the German company Zalando.
But before that, let's delve into the world of big data to understand where this concept known in Spanish as "data mesh" comes from.
When we talk about ETL processes , we explain that the data warehouse is like a raw materials warehouse but full of data.
These processes and warehouses were a great discovery and have helped many companies reach customers in ways that were unthinkable years ago. But as time goes by, innovations continue and there is a search to speed up processes, since the extraction, transfer and loading of data can sometimes be too slow.
To achieve a faster pace of management, what is known as a data lake was born.
What is Data Lake?
The data lake is a data warehouse, just like the chinese overseas asia database warehouse, but unlike the latter, it supports any type of data and preserves it as it has arrived in the cloud.
The advantage of the data lake is that it adapts quickly to changes because it is stored in a flat architecture, and not in the form of files as in the data warehouse.
Users access the data before it has been transformed and filtered, making it easier to access.
What is Data Mesh?
The data mesh divides the data stored in the data lake into many business domains that will have a team of professionals in charge of creating data products to deliver them to various areas of the organization.
The data mesh is a further step towards personalizing the products or services that companies offer to each client, as it prioritizes the levels of data and, in turn, makes processes faster for the digital needs demanded by the market.
After all, the user browses from more than one device and synchronizations between them must be agile. The data mesh allows this, allowing data products to be linked between domains allowing the exchange of data without storage. We will explain this in the next section.
Data Mesh Principles
To better understand how data mesh can revolutionize the world of data, let's explain the 4 principles that set it apart.
Managing data independently
We have already mentioned this, but the fact that there is no warehouse offers much more convenient and cheaper management through domains, without a warehouse. It is independent data management.
Usable and simple data as if it were a product.
Analytics data provided by domains should be treated as a product, and consumers of that data should be treated as customers.
For this to work, the domain data product owner must have a deep understanding of who the users of that data are , how they use it, and what methods they are comfortable using to consume it. This knowledge of users enables the design of data product interfaces that meet their needs.
Self-service data platform
Members of each domain can access it to make use of the data, it requires high technology if you want to access the data products correctly.
Data Mesh: What it is, advantages and examples
Federated governance
We've said that these data sets are independent, but to be truly independent, to not need a warehouse, they must be able to interoperate with each other . In this way, unions are created, intersections are found, graphs and large-scale operations are performed.
In our article on the profile of the data architect, we mentioned that big data is a very changing environment and that this professional would have to adapt to changing trends over time; the data mesh is another proof of this.
Now we are going to see an example of how a company has managed to change its way of working with data, from the data warehouse to the data mesh.
Data Mesh Example: Zalando
German company Zalando is achieving extraordinary results in the footwear niche, here we will break down how it has used data over time.
Originally, it was fairly easy to get connectivity to the back-end databases to get the information that the business actually needed and then bring it all together for reporting, so it ran in the data warehouse.
But there came a time when Zalando wanted to scale its business, so it started migrating its database to the cloud. In reality, this means that they were starting to build an architecture of microservices located in separate environments.
To unite this data, a data lake was started to be used , which was part of data management and integration and which made the processes more agile.
Zalando was looking for interoperability in data management, which is what a data warehouse offers, but at the same time, such a warehouse slowed down the steps in the management process and caused bottlenecks to form at some point in the task.
After observing that, although the use of data was good, it could be improved, they began to implement the data mesh.
The first thing they did was follow the principle of “ treating data as a product ” by asking themselves how to market their data. To do this, they created the position of “data product owner” because they needed someone to oversee each data product autonomously.
They then created real domains. This allowed people on the team to focus specifically on each domain and become experts in it.
When a company makes this transformation towards the data mesh, it is giving a lot of responsibility to its employees, which shows great confidence in them.
At Zalando, data was no longer archived by default, but rather shaped by studying the people in charge of it. This led them to realise that there was a huge amount of data that was not being used and was simply a burden on the cloud.