See this demo of Cloudera Data Engineering which builds upon Apache Spark and allows us to load, transform, and enrich our datasets and has built-in workload orchestration to automate these pipelines at scale.
The demo will also illustrate how easy it is to go from streaming to enrichment and data pipeline automation all in an end-to-end data platform. To see the other demos of this use case, view the full ClouderaNow event, here: https://bit.ly/3eyMTDR