OpenSearchCon India 2025

In this talk, we’ll share Uber’s journey in scaling OpenSearch to handle large scale data across 100+ clusters while ensuring high availability, fault tolerance, performance and zero-downtime upgrades. We’ll dive deep into our strategies for overcoming regional and zonal failures, observability and fleet-upgrade strategy with zero-downtime, at scale. Attendees will learn about our approaches to sharding, indexing, and data distribution that enables resilience across a global infrastructure. Additionally, we’ll cover key best practices for managing OpenSearch clusters at Uber’s scale, ensuring reliability, and achieving cost-efficiency.

Learn the story of OpenSearch, and how we evolved the OpenSearch snapshot process to support sub-minutely granularity at Petabyte scale without sacrificing on cost-effectiveness, performance and availability. We'll share our journey of establishing core principles and breaking down complex challenges into manageable deliverables.

We will start by talking about how snapshots work in the OpenSearch world. We will cover some of the key design decisions that helped us to allow backward compatibility with existing snapshots, without major overhaul of the system. We will walk through our open-source first journey, illuminating common pitfalls and lessons learnt in scaling, correctness and performance, which helped shape the eventual solution.
We will lastly cover step by step as to how users can get started on enabling the new sub-minutely snapshot mechanism for their workload to achieve high data protection for their use cases

Learn the story of AWS OpenSearch Service, and how we evolved the OpenSearch engine to offer high durability at Petabyte scale without sacrificing on performance. We’ll share our journey of establishing core principles and breaking down complex challenges into manageable deliverables. We will cover some of the key design decisions that helped us to allow new and prevalent software engines to co-exist, without major overhaul of the system. We will walk through our open-source first journey, illuminating common pitfalls and lessons learnt in scaling, correctness and performance, which helped shape the eventual solution. Finally, we’ll reveal how we revolutionised our periodic data backup strategy, transforming a process that handles half a million partitions into one that completes in mere seconds—enabling granular, minute-by-minute backups.

Key Take-aways
- Define your tenets
- Don’t try building an ideal system
- Don’t re-invent the wheel
- Why high judgement calls matters
- Invent to simplify

At Uber, OpenSearch is a core component of our search infrastructure. We manage ~100 clusters across several thousands of nodes. We platformized OpenSearch to ensure scalability and reliability for Uber’s search needs.

This talk covers:

Automated Build & Deployment: Uber maintains a custom OpenSearch fork with performance optimizations, scalability improvements, and bug fixes. We’ll discuss our build and release pipelines for deploying OpenSearch at scale.

Batch & Streaming Ingestion: We built batch ingestion using Spark and OpenSearch-Hadoop and real-time streaming with Kafka/Flink, scaling to ingest a billion vectors in X hours.

Elasticsearch to OpenSearch Migration: Insights into compatibility challenges, query performance tuning, and scaling ingestion throughput.

Security: Implementing fine-grained authentication and authorization for multi-tenant OpenSearch clusters.

Community Contributions: How we contribute optimizations, bug fixes, and features upstream to improve OpenSearch for everyone.

Operating OpenSearch in multi-tenant environments presents challenges in performance, security, and data isolation. This session explores best practices for optimizing OpenSearch for high-scale, multi-tenant architectures, covering index sharding, access control, and query performance tuning. Attendees will gain insights from real-world deployments and learn how to ensure cost efficiency and high availability in OpenSearch clusters.