Systems | Development | Analytics | API | Testing

Data Pipelines

Streaming Pipelines With Snowflake Explained In 2 Minutes

Streaming data has been historically complex and costly to work with. That's no longer the case with Snowflake's streaming capabilities. Together, Snowpipe Streaming and Dynamic Tables (in public preview) break the barrier between batch and streaming systems. Now you can build low-latency data pipelines with serverless row-set ingestion and declarative pipelines with SQL. You can easily adapt to your business requirements to change latency as a single parameter.

Building a Real-time Snowflake Data Pipeline with Apache Kafka

In today's data-driven world, organizations seek efficient and scalable solutions for processing and analyzing vast amounts of data in real-time. One powerful combination that enables such capabilities is Snowflake, a cloud-based data warehousing platform, and Apache Kafka, a distributed streaming platform.

Streaming Data Pipeline Development

This Meetup will cover how to build applications from some common use cases and highlight tips, tricks, best practices and patterns In this interactive session, Tim will lead participants through how to best build streaming data pipelines. He will cover how to build applications from some common use cases and highlight tips, tricks, best practices and patterns. He will show how to build the easy way and then dive deep into the underlying open source technologies including Apache NiFi, Apache Flink, Apache Kafka and Apache Iceberg.

How to Monitor and Debug Your Data Pipeline

Picture this: during a bustling holiday season, a global e-commerce giant faces a sudden influx of online orders from customers worldwide. As the company's data pipelines navigate a labyrinth of interconnected systems, ensuring the seamless flow of information for timely product deliveries becomes paramount. However, a critical error lurking within their data pipeline goes undetected, causing delays, dissatisfied customers, and significant financial losses.


The Future of Data Pipelines: Trends and Predictions

The global data integration market size grew from $12.03 billion in 2022 to $13.36 billion in 2023, making it evident that organizations are prioritizing efficient data integrations and emphasizing effective data pipeline management. Data pipelines play a pivotal role in driving business success by transforming raw datasets into valuable insights that fuel informed decision-making processes.


No-Code Data Pipelines: Streamline Data Integration

Historically, connecting multiple data sources to a single destination required extensive experience as a computer programmer or data scientist. Today’s no-code data pipelines have changed that perspective. Now, practically anyone – even those without any coding experience – can use no-code pipelines to streamline data processing without damaging data quality. You will, however, need the right ETL and ELT tools to manage real-time data flows.


Streamline Your Data Pipeline with No-Code ETL Tools

Staying competitive in today’s business world means having access to a greater volume of data than ever before. No-code ETL tools give businesses of all sizes the opportunity to connect to SaaS, social media platforms, and all types of digital marketing data to keep their data warehouses up to date and improve the effectiveness of business intelligence (BI) tools.


5 Tools to Build Modern Data Pipelines

Data pipelines are a critical element of any modern, data-driven organization. With the right tools in hand, data analysts can quickly build resilient data pipelines for your analytics infrastructure. From orchestration to monitoring, these tools can march your business towards advanced levels of automation, as well as improved transparency into how your pipeline functions along every step of its journey.