In this article, we discuss how you can avoid data pipeline breakdowns thanks to total observability through the use of dbt complemented with Kensu. Data quality problems tend to manifest in many ways. Here is an example.
API Observability isn't exactly new, however it's popularity has seen rapid growth in the past few years in terms of popularity. API Observability using open source is different from regular API monitoring, as it allows you to get deeper and extract more valuable insights. Although it takes a bit more effort to set up, once you've got an observability infrastructure running it can be immensely helpful not only in catching errors and making debugging easier, but also in finding areas that can be optimized.
A business is only as healthy as its data. Organizations rely on data not just to accelerate and adapt, but increasingly, to perform the most basic of business operations, from hiring new personnel to launching and moving products.
API providers need to observe their APIs to get meaningful data about whether and how they are consumed in practice. API observability is a form of monitoring that passively logs API traffic to an observability service. Different from traditional API monitoring, with API observability you: Monitor interactions to improve developer experience Understand how customers use your API Troubleshoot your API Observing REST APIs is well understood and supported, but not every API is a REST API.
Before the data era, data engineers and data scientists had few resources, few technologies, and few data to build something from. But they also had little pressure from the business to create new values, and above all, it was easier to find some time to write, check and implement their applications. It had the advantage of better control of quality.