OpenSearchCon India 2025

Cost for running OpenSearch also always been a huge factor. In spite of OpenSearch being completely open-source and no licensing cost, when you are dealing with terabytes and petabytes of data, cost can quickly stack.

This talk aims to empower OpenSearch users by taking a holistic view using best practices and with practical steps & checklist for reducing the infrastructure cost and to do more with less with OpenSearch regardless of the workload (be it Search/Log Analytics/Observability/Semantic Search).

We'll discuss

1/ Gaining optimal performance from existing resources
2/ How to right size a domain & when to Horizontal scale vs. Vertical scale
3/ How to scale down without impacting performance
4/ How much heap memory is too much or too little and what is the correct vCPU ratio for the given heap memory
5/ When to de-compose larger clusters into smaller clusters Vs. When to aggregate smaller clusters to few larger ones
6/ How to write cache efficient queries, optimize mapping and the index settings
7/ Knobs to improve ingestion and search speed
8/ Budget friendly ways to implement Semantic Search

Learn with examples the variety of OpenSearch driven - Vector, Neural, Conversational, Semantic and Multimodal Search and the difference to cost , resiliency and response time in a Managed Vs Self run Cluster. Lessons, Challenges and Solutions would enable you to architect and run it successfully to be the data exhaust fingerprint for your product.

OpenSearch is a powerful open-source search and analytics engine, but running it effectively on Kubernetes requires a well-structured approach. In this session, I will explore how to deploy, configure, and operate OpenSearch in a Kubernetes-native way. I will dive into best practices for maintaining OpenSearch clusters, optimizing performance, and handling scaling challenges.

Additionally, I will cover how to ship Kubernetes logs to OpenSearch, enabling centralized log management for a single cluster and also across multiple clusters. I will discuss scalable log ingestion strategies involving off the shelf as well as custom made tools. The session will also include practical demonstrations on building insightful dashboards for observability and setting up alerts to proactively monitor system health.

This talk will provide a comprehensive guide to managing OpenSearch effectively within Kubernetes.

I hope to share our experience of managing petabyte-scale log data on OpenSearch, achieving 30% cost savings while ensuring performance and scalability. I’ll cover key strategies, challenges, and optimisations, including searchable snapshots, compression techniques, and index management to efficiently store and query massive datasets. I’m also looking forward to engaging with the OpenSearch community to exchange ideas and refine best practices for large-scale deployments.

Attendees will learn practical insights into migrating large workloads with no downtime, optimising storage through compression and index management, and balancing cost and performance using searchable snapshots. I’ll also share best practices for ensuring high availability across multiple availability zones and building strong disaster recovery strategies in OpenSearch.

By sharing real-world challenges and solutions, I aim to help organisations adopt OpenSearch confidently. Demonstrating its scalability and cost efficiency can encourage wider adoption while driving improvements in data management and operational resilience within the OpenSearch ecosystem.

In this talk, we’ll share Uber’s journey in scaling OpenSearch to handle large scale data across 100+ clusters while ensuring high availability, fault tolerance, performance and zero-downtime upgrades. We’ll dive deep into our strategies for overcoming regional and zonal failures, observability and fleet-upgrade strategy with zero-downtime, at scale. Attendees will learn about our approaches to sharding, indexing, and data distribution that enables resilience across a global infrastructure. Additionally, we’ll cover key best practices for managing OpenSearch clusters at Uber’s scale, ensuring reliability, and achieving cost-efficiency.

Learn the story of OpenSearch, and how we evolved the OpenSearch snapshot process to support sub-minutely granularity at Petabyte scale without sacrificing on cost-effectiveness, performance and availability. We'll share our journey of establishing core principles and breaking down complex challenges into manageable deliverables.

We will start by talking about how snapshots work in the OpenSearch world. We will cover some of the key design decisions that helped us to allow backward compatibility with existing snapshots, without major overhaul of the system. We will walk through our open-source first journey, illuminating common pitfalls and lessons learnt in scaling, correctness and performance, which helped shape the eventual solution.
We will lastly cover step by step as to how users can get started on enabling the new sub-minutely snapshot mechanism for their workload to achieve high data protection for their use cases

Learn the story of AWS OpenSearch Service, and how we evolved the OpenSearch engine to offer high durability at Petabyte scale without sacrificing on performance. We’ll share our journey of establishing core principles and breaking down complex challenges into manageable deliverables. We will cover some of the key design decisions that helped us to allow new and prevalent software engines to co-exist, without major overhaul of the system. We will walk through our open-source first journey, illuminating common pitfalls and lessons learnt in scaling, correctness and performance, which helped shape the eventual solution. Finally, we’ll reveal how we revolutionised our periodic data backup strategy, transforming a process that handles half a million partitions into one that completes in mere seconds—enabling granular, minute-by-minute backups.

Key Take-aways
- Define your tenets
- Don’t try building an ideal system
- Don’t re-invent the wheel
- Why high judgement calls matters
- Invent to simplify

At Uber, OpenSearch is a core component of our search infrastructure. We manage ~100 clusters across several thousands of nodes. We platformized OpenSearch to ensure scalability and reliability for Uber’s search needs.

This talk covers:

Automated Build & Deployment: Uber maintains a custom OpenSearch fork with performance optimizations, scalability improvements, and bug fixes. We’ll discuss our build and release pipelines for deploying OpenSearch at scale.

Batch & Streaming Ingestion: We built batch ingestion using Spark and OpenSearch-Hadoop and real-time streaming with Kafka/Flink, scaling to ingest a billion vectors in X hours.

Elasticsearch to OpenSearch Migration: Insights into compatibility challenges, query performance tuning, and scaling ingestion throughput.

Security: Implementing fine-grained authentication and authorization for multi-tenant OpenSearch clusters.

Community Contributions: How we contribute optimizations, bug fixes, and features upstream to improve OpenSearch for everyone.

Operating OpenSearch in multi-tenant environments presents challenges in performance, security, and data isolation. This session explores best practices for optimizing OpenSearch for high-scale, multi-tenant architectures, covering index sharding, access control, and query performance tuning. Attendees will gain insights from real-world deployments and learn how to ensure cost efficiency and high availability in OpenSearch clusters.