Unstructured text is everywhere in business: customer reviews, support tickets, call transcripts, documents. Large language models (LLMs) are transforming how we extract value from this data by running tasks from categorization to summarization and more. While AI has proved that real-time conversations in natural language are possible with LLMs, extracting insights from millions of unstructured data records using these LLMs can be a game changer. This is where batch LLM inference becomes essential.

Earlier this year the Kong Konnect Analytics team was looking to leverage the stability and flexibility of our own Kong Gateway to handle the entire load of our Analytics ingest firehose. Almost all Konnect API traffic flows through a Kong Gateway and most of it has from the very beginning. For some legacy and differing protocol reasons, our Analytics ingest service was not included in our initial Kong Gateway configuration.

The typical data engineer is balancing many priorities. In addition to the day-to-day tasks like pipeline construction, connecting sources, and establishing data trust and governance, pressure is increasing to contribute to the organization’s bottom line. This may happen by unlocking the full value of the organization’s data or devising ways to accelerate business outcomes. One way to accomplish both is through data products.

Budgeting and planning are the backbone of your organization’s success. However, manual processes, endless spreadsheets, and disconnected systems can bog down your finance team, creating bottlenecks that waste time and divert focus from strategic growth. To stay competitive, you need a smarter approach—one that streamlines workflows, enhances accuracy, and maximizes ROI.

Financial services companies are leveraging data and machine learning to mitigate risks like fraud and cyber threats and to provide a modern customer experience. By following these measures, they are able to comply with regulations, optimize their trading and answer their customers’ needs. In today’s competitive digital world, these changes are essential for ensuring their relevance and efficiency.

With the advent of modern Large Language Models (LLMs), Retrieval Augmented Generation (RAG) has become a de-facto technology choice, employed to extract insights from a variety of data sources using natural language queries. RAG combined with LLMs presents many new possibilities for integrating Generative AI capabilities within existing business applications, specifically opening up many new use cases within the data streaming and analytics space.