As businesses increasingly leverage data-driven decision making, the ability to use and understand data at the company-wide level becomes mission critical. While tech behemoths like Netflix, Airbnb, and Spotify have strong data cultures built over the last decade, most companies often face challenges getting up and running with data.
The Insurance industry is in uncharted waters and COVID-19 has taken us where no algorithm has gone before. Today’s models, norms, and averages are being re-written on the fly, with insurers forced to cope with the inevitable conflict between old standards and the new normal.
The only certainty in today’s world is change. And nowhere is that more apparent than in the way organizations consume data. A typical company might have thousands of analysts and business users accessing dashboards daily, hundreds of data scientists building and training models, and a large team of data engineers designing and running data pipelines. Each of these workloads has distinct compute and storage needs, and those needs can change significantly from hour to hour and day to day.
It’s time, again, to look at a chart that you might not be using, but that you definitively should consider using when doing data visualization. The waterfall chart is great at visualizing the cumulative effect from positive and negative changes, as, for example, you would see in a Profit and Loss (P&L) report.
On average, data-driven companies grow more than 30% every year. Because of the competitive advantage that data confers to incumbents who are capable of extracting value from it, it has been called the new oil. Companies are tapping into this well of resources because of the advantages that it has to offer: But using data to run your operations poses its own set of challenges.
Today we are introducing a new series of blog posts that will take a look at recent enhancements to Apache Impala. Many of these are performance improvements, such as the feature described below which will give anywhere from a 2x to 7x performance improvement by taking better advantage of all the CPU cores. In addition, a lot of work has also been put into ensuring that Impala runs optimally in decoupled compute scenarios, where the data lives in object storage or remote HDFS.
