In this article, we discuss how you can avoid data pipeline breakdowns thanks to total observability through the use of dbt complemented with Kensu. Data quality problems tend to manifest in many ways. Here is an example.

Before the data era, data engineers and data scientists had few resources, few technologies, and few data to build something from. But they also had little pressure from the business to create new values, and above all, it was easier to find some time to write, check and implement their applications. It had the advantage of better control of quality.

Data has become the lifeblood of most organizations. Yet, despite using data almost daily to make critical business decisions, few organizations have complete visibility into the health and usage of their data. Moreover, as the acceleration of data usage has increased, so too has the complexity of data systems, increasing the risks of data-related issues and making it even more difficult to identify and resolve issues related to data quickly.

In our last article, we introduced the topic of SLAs (Service Level Agreements) and how they are necessary within organizations to help both consumers and producers agree on expectations around data usage and quality. Not only do SLAs provide visibility into what needs to be achieved to ensure data reliability and avoid surprises, but SLAs also create communication flows between consumers and producers that help ensure an alignment on expectations.