By Christian Poecher, Senior Solutions Engineer at Privitar

As modern enterprises struggle to manage their fast-growing amounts of data, they are also seeking more and more ways to modernize and optimize their data architecture stacks. Over the last year, we’ve not surprisingly seen a dramatic increase in interest in the data mesh – so much that it has become one of the biggest trends in data science and analytics. As my colleagues noted earlier this year, data mesh provides a well-balanced blueprint for enterprise data architecture and is suited to modern technologies and modern people organizations that enable successful enterprises.

Challenging the status quo

Many organizations realize that their traditional data operation models do not lend themselves well to a data-driven culture which is required these days to thrive. The data warehouse, with central control of a dedicated team enforcing highest quality, has been criticized as being too slow and inflexible in responding to business demands. This became one of the drivers for the data lake. However, high flexibility combined with minimal oversight of data lakes often led to a chaotic state that made it useless for all but the most data literate in an organization. Modern cloud architectures have eased technical limitations but created new concerns around data sovereignty and also provided little progress with regard for the human and organizational factors involved. The data mesh paradigm is a blueprint that includes guidance on how to solve these governance challenges.

The Data Mesh is federated like modern organizations

The data mesh is designed around a federated model of ownership. Each data owner and each domain can choose how to achieve their goals to a decent level of internal autonomy. In the end, their customers need to be happy, whatever it takes. Any organization should strive to maximize that autonomy and make decisions as locally as possible. But some cross-cutting concerns can only be accomplished by certain standards everyone has to adhere to. For example, security, interoperability, risk exposure, or legal obligations. In essence, this is what is called data governance. One example of this is the protection of personally identifiable information (PII) while preserving analytical utility. 

Shifting towards federated and computational governance

Federated ownership of data products also requires federated data governance, so data owners work together with data guardians for legal guidance and to achieve sufficient levels of data protection. As a team, they are collaboratively responsible – but not chaotic! They can apply just the right number of policies to the data products. Obviously, ground rules or processes help to structure the collaboration between the stakeholders, especially in today’s global and hybrid work environments.

Another critical success factor for governance in the enterprise is to minimize manual effort. Processes that require having a human in the loop should be avoided, as they bog down the agility of data provisioning and increase operational costs. Therefore, it is important to strive toward computational governance. This means that policies are put into place, which computationally assert the right level of governance depending on the usage context. 

For example, most internal use cases require the same amount of protection of personally identifiable customer data as defined in GDPR. When organizations share that same data with an external partner, for example, a marketing analysis agency, stricter data protection may be more advisable. In other use cases where data is used according to the declared intended use, applying fewer restrictions may be adequate. The principles of federated ownership, as well as computational governance, are force multipliers as they easily allow tailoring the policies depending on usage context.

Modern data provisioning minimizes the invasiveness of data governance

Most data executives want governance that is low maintenance and stays out of the way as a default. Taking a modern approach to data provisioning accomplishes this by uniting data, technology, people, and processes to enable efficient, effective, and responsible data use. By embracing modern data provisioning, organizations can gain the agility to innovate and adapt to evolving regulations. Stakeholders collaborate to maximize the value of data, and valuable data reaches those who need it, when they need it, while managing risks and demonstrating compliance with relevant laws and regulations.

Semantic metadata helps differentiate sensitive information like “date of birth” from regular data like “last change date.” Policies are managed by the appointed data guardians, and the system automatically infers the right policies to transform the data. Everyone can participate, whether they are a data owner, a data guardian, or just the actual data consumer.

Data is transformed automatically and individually depending on the usage context. Usage of data across the enterprise is automatically tracked supporting data lineage. The data can be provisioned using dynamic access at runtime as it flows to the consumer, meaning no additional copies need to be maintained. If required, batch or streaming data provisioning patterns can be applied as well. For global enterprises, data can be limited to be processed or accessed only in specific countries and regions to remain compliant with safe-harboring legislation or to protect intellectual property against espionage. All the while, it can deliver a simple and clean user experience through a single, unified management UI.

The critical role of safe data 

Simply leveraging a data mesh architecture does not guarantee success. Like most things in life, what you put in impacts what you get out—and if you are leveraging a data mesh as part of your analytics programs, you need to ensure that safe data is at the core to maximize the impact of your efforts. This is a critical success factor for organizations looking to take an agile approach to creating, managing, governing, and protecting data products—and to leveraging that data for analytics within their organizations and beyond.

Modern data provisioning puts safe and ethical data at the core of a data mesh

The most effective and efficient way to ensure that data remains safe is by taking a modern approach to data provisioning and leveraging data privacy tools as part of the process. This enables you to seamlessly de-identify sensitive information. It also allows your data scientists to analyze data without exposing sensitive, protected information. De-identifying sensitive data allows you to provision your data to a broader group of analysts while upholding customer privacy and ensuring regulatory compliance. This is essential for organizations to ensure that their data stays safe across their organization and beyond.

A data mesh will take the friction out of working with data and empower teams to realize their creativity and capabilities without being slowed down by excessive controls. Embracing a modern approach to data provisioning makes these privacy controls part of the process and implements the governance required in the data mesh to allow organizations to use 100 percent of their data ethically and safely across their business and beyond. 

Learn how the Privitar Modern Data Provisioning Platform can help you put safe and ethical data at the core of your data mesh and other modern data architectures