OpenSearchCon India 2025

In modern big data ecosystems like Databricks, logs from different jobs, stages, and components can be highly diverse, making it difficult to gain comprehensive insights into system health, performance, and operational flow. This presentation will explore how to consolidate these disparate logs into a unified OpenSearch platform, enabling advanced log analysis and real-time visibility. By leveraging large language models (LLMs) and contextual dashboards, we will demonstrate how to create a "single pane of glass" for monitoring entire operational workflows. Attendees will learn how this approach improves visibility, traceability, and transparency, enabling quicker actions and decisions based on knowledge-driven insights. Key topics include log mapping, lineage tracking, and using OpenSearch to facilitate both broad and granular insights for operational teams. By the end of this session, you'll understand how to streamline complex log data from systems like Databricks, empowering your teams with smarter, faster decision-making and more efficient troubleshooting.