This is not a technology problem, it's an operations problem.

Your data is a product, treat it like one.

Organizations and engineering leaders have largely embraced and see the benefits of a distributed, micro-service architecture over the once popular mega monolithic code-bases. This is true both for customer-facing software applications and the internal systems that support business operations.

A distributed architecture decouples the individual components of your system, as well as the individual teams and domains of your organization, allowing them to operate autonomously, and businesses to move quickly.

Why is it that the data landscape is so far behind? The parallels between software product development and data product development are palpable. 

Whether looking at a dashboard of core business KPI reports, a machine-learning model, or a one-off analysis, all of these activities and the data produced by them are equally valuable to the organization as the applications we build and buy that generate the data. 

If we can't accept that basic premise, our pursuits of advanced analytics, AI and Machine Learning will be underwhelming at best and in the worst cases it will introduce a deep and painful organizational rift that is hard to identify.

This is not a technology problem, it's an operations problem. Teams, missions, skills, tools, and incentives need to be strategically designed to capture the full value locked up in our data and generate new value on top of it.

The Data Landscape of the Future

Recently, I've been tuned into the research of Zhamak Dehghani of ThoughtWorks. I want to share it with others working in data engineering, data analytics, machine learning, data science...etc. Her perspective and insights so perfectly capture the challenges I've experienced working in data, business intelligence, and analytics.

Read This: How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh

"The data mesh platform is an intentionally designed distributed data architecture, under centralized governance and standardization for interoperability, enabled by a shared and harmonized self-serve data infrastructure. I hope it is clear that it is far from a landscape of fragmented silos of inaccessible data."

Listen to This: Data Engineering Podcast: From Data Lake to Data Mesh