Calculating log analytics ROI is often complicated. For many teams, this technology can be a cost center. Depending on your platform, the cost of a log management solution can quickly add up. For example, many organizations use solutions like the ELK stack because the initial startup costs are low. Yet, over time, costs can creep up for many reasons, including the volume of data collected and ingested per day, required retention periods, and the associated personnel needed to manage the deployment.

Organizations are building data lakes and bringing data together from many systems in raw format into these data lakes, hoping to process and extract differentiated value out of this data. However, if you’re trying to get value out of operational data, whether on prem or in the cloud, there are inherent risks and costs associated with moving data from one environment to another.

Javascript Object Notation (JSON) is becoming the standard log format, with most modern applications and services taking advantage of its flexibility for their logging needs. However, the great flexibility for developers quickly turns into complexity for the DevOps and Data Engineers responsible for ingesting and processing the logs. That’s why we developed JSON FLEX: a scalable analytics solution for complex, nested JSON data.

On top of their industry-leading cloud infrastructure, Amazon Web Services (AWS) offers more than 15 cloud-based analytics services to satisfy a diverse range of business and IT use cases. For AWS customers, understanding the features and benefits of all 15 AWS analytics services can be a daunting task - not to mention determining which analytics service(s) to deploy for a specific use case.

Can DataOps help data consumers reveal and take action on powerful product insights hidden in operational data? For many companies, the answer is yes! The emerging practice of DataOps applies Agile development principles and DevOps best practices (e.g. collaboration, automation, monitoring and logging, observability) to data science and engineering, making it faster and easier for organizations to uncover valuable product insights that enable innovation.

Streaming analytics is an invaluable capability for organizations seeking to extract real-time insights from the log data they continuously generate through applications and cloud services. To help our community get started with streaming analytics on AWS, we published a piece last year called An Overview of Streaming Analytics in AWS for Logging Applications, where we covered all the basics.

In the past, querying a database required Structured Query Language (SQL) skills, or knowledge of other database query languages, such as Kibana Query Language (KQL). Today, with the emergence of generative AI (GenAI), teams can query their analytic database using natural language — and get plain English results in return. Or, if you prefer to still use SQL, many teams use GenAI for database query optimization, making queries faster and more efficient.

Mobile and video games generate huge amounts of data that can be captured and analyzed by game developers to power gaming analytics initiatives and improve games.

Recently at re:Invent, Amazon unveiled S3 Express One Zone for AWS. Express Zone for S3 responds to the demand for faster analytical query speeds, with the convenience of centrally storing all of your application telemetry data in cloud object storage. In the past, for data-intensive applications, data access speeds were slower than desired.

In 2009, as the world became increasingly data-driven, organizations began to accumulate vast amounts of data — a period that would later be characterized as the Big Data revolution. While most organizations were used to handling well-structured data in relational databases, this new data was appearing more and more frequently in semi-structured and unstructured data formats.