Improving Data Pipeline Reliability with On-Call Data Teams
A big part of data teams’ responsibilities is dealing with the unpredictable. Data pipelines don’t always run without incident: you need to rerun processes and fix data processing issues—in other words, put out data fires—to keep stakeholders happy. For every significant roadblock, additional time and effort is given over to investigation and post-mortem reports to make sure the incident doesn’t reoccur. But naturally, they keep happening.