Assessment of the Distortion Propagation in Data Products in the Context of a Relational Data Mesh Architecture

Abstract

With a decentralized approach to data management named Data Mesh, interaction with data occurs at the level of individual data products. If errors or poor-quality data are used during the preparation of data products, their distribution can significantly increase across replicas, datamarts, services, and etc. This paper examines the quality of a data product in a relational Data Mesh architecture under conditions of detected input data distortions. An approach of constructing such an assessment based on representing data transformations within a data product as a chain of elementary relational algebra operations, such as extended projection, filtering, union, and Cartesian product, and analyzing the propagation of input data distortions as a result of applying these operations is proposed. A concept for "labeling" the source relation is presented, allowing tuples to be divided into types based on the nature of the distortions detected in them, as well as algorithms aimed at tracking the transformation of tuples of different types as a result of applying the above elementary operations. A complex algorithm is developed that implements the solution to the considered data product quality assessment problem. As part of the testing, an experimental setup was developed using the Open University open data set and the proposed algorithm was implemented. The experimental results confirmed that the data product quality assessment algorithm can be successfully used within the context of a relational Data Mesh architecture. In conclusion, several practical applications of this assessment for effective data quality management within the Data Mesh concept are proposed.