OpenSearchCon India 2025

Cost for running OpenSearch also always been a huge factor. In spite of OpenSearch being completely open-source and no licensing cost, when you are dealing with terabytes and petabytes of data, cost can quickly stack.

This talk aims to empower OpenSearch users by taking a holistic view using best practices and with practical steps & checklist for reducing the infrastructure cost and to do more with less with OpenSearch regardless of the workload (be it Search/Log Analytics/Observability/Semantic Search).

We'll discuss

1/ Gaining optimal performance from existing resources
2/ How to right size a domain & when to Horizontal scale vs. Vertical scale
3/ How to scale down without impacting performance
4/ How much heap memory is too much or too little and what is the correct vCPU ratio for the given heap memory
5/ When to de-compose larger clusters into smaller clusters Vs. When to aggregate smaller clusters to few larger ones
6/ How to write cache efficient queries, optimize mapping and the index settings
7/ Knobs to improve ingestion and search speed
8/ Budget friendly ways to implement Semantic Search

This session explores how Gaia’s decentralized AI platform integrates with OpenSearch to create scalable, privacy-preserving solutions for semantic search, generative AI applications, and real-time analytics.

In this session, I intend to demonstrate a use case where Gaia’s federated learning models enhance OpenSearch’s vector database capabilities, enabling dynamic personalization and efficient retrieval-augmented generation (RAG) workflows.

Gaia’s AI Framework: How Gaia’s decentralized architecture complements OpenSearch’s ecosystem, particularly in scenarios requiring data privacy and distributed model training (e.g., healthcare or financial analytics).

Demo a RAG Pipeline: Build a real-world demo/example using Gaia’s language models to generate embeddings stored in OpenSearch’s binary vector indexes (new in v2.19), coupled with OpenSearch Flow for automated pipeline configuration.

Community Impact: A look at how this integration supports OpenSearch’s goal of becoming the preferred backend for generative AI while adhering to open governance principles.

OpenSearch is a powerful search and analytics engine. But many organizations unknowingly expose their OpenSearch clusters, leaving them vulnerable to data leaks, privilege escalation, and even full system compromise. In this session, we’ll look into real-world OpenSearch security flaws, exploit them live, and secure them.

This talk is hands-on - we’ll spin up two OpenSearch environments: one intentionally vulnerable, the other properly secured. You’ll see exactly how attackers break in, followed by a step-by-step hardening process.

The key takeaways from this session will be:
1. Live demonstrations of common attack techniques (unauthenticated access, query injection, privilege escalation, and container compromise)
2. Best practices for securing OpenSearch, including TLS, RBAC, and security configurations

In this session, Uber, a premier member of the OpenSearch Foundation, will share how the unique demands of their low-latency, high-throughput search platform have driven impactful contributions to the OpenSearch project.

The discussion will explore Uber’s collaboration with the OpenSearch community to integrate critical learnings and innovations from their own search platform. Key highlights include the development of a modular, cloud-native architecture with clear separation between readers and writers, a high-performance messaging protocol for inter-service communication, and a pull-based indexing model.

These advancements, along with contributions from the broader OpenSearch community, are shaping OpenSearch into a cutting-edge, serverless open-source search engine tailored for enterprise needs. Join this session to discover how pioneering work from Uber and OpenSearch is paving the way for the next generation of high performance and scalable search technology in open source.

OpenSearch is a powerful tool for log storage, analysis, and dashboard reporting. In the context of Retrieval-Augmented Generation (RAG) applications with Large Language Models (LLMs), OpenSearch serves as a vital component, acting as a vector database. This presentation explores an architecture and process flow to integrate graph-based context with OpenSearch, enhancing the correlatability of data and improving the context and performance of search results.

Through experimentation with synthetic data from a travel app, we’ll demonstrate how customer activity data—such as travel points, preferences, and interactions with destinations—can be integrated into OpenSearch. By introducing graph points and correlating them with other data (e.g., maps and historical data), we can provide personalized, more relevant search results for users. Additionally, we’ll compare metrics before and after the introduction of graph context to showcase how it improves search performance and the overall user experience in RAG-based systems.

What if your observability tools could predict breaches before they happen? We built a self-learning OpenSearch pipeline that ingests 10M+ logs/sec, detects zero-day threats using unsupervised ML models, and auto-generates incident reports. Learn how we optimized OpenSearch’s anomaly detection plugin to reduce false positives by 60%, integrated threat intelligence feeds, and built a custom Kibana dashboard that even your CISO can love.

Spoiler: We’ll share how a single line of bad regex almost took down our cluster (and how we fixed it).

Snapshots are the backbone of data resilience and portability in OpenSearch. This session will provide a comprehensive exploration of snapshot capabilities in open-source OpenSearch, covering recent advancements and best practices for implementing robust backup, restore, and data migration strategies.

Key areas we'll delve into:
1. Fundamentals: Core architecture and mechanics of snapshot operations.
2. Repositories: Comparing storage options across file system, S3, Azure, and GCS implementations.
3. Management: Automated policy-based snapshot scheduling and lifecycle management.
4. Cross-cluster: Efficient data movement techniques for disaster recovery and migrations.
5. Searchable Snapshots: Instant access to historical data without full restoration.
6. Performance: Optimizing snapshot and restore operations for minimal cluster impact.
7. Scaling: Advanced techniques for managing snapshots in large-scale deployments.
8. Security: Repository access controls, encryption, and multi-tenant considerations.
9. Monitoring: Tools and strategies for maintaining healthy snapshot operations.
Walk away with practical knowledge to master OpenSearch snapshots, from basic to advanced solutions.

In the era of vector search and semantic similarity, returning highly relevant results is only half the battle. When search results are too similar, users must wade through redundant information to find diverse perspectives. This talk introduces Maximum Marginal Relevance (MMR) implementation in OpenSearch, a powerful technique that optimally balances result relevance with diversity.

Key takeaways will include:
- Implementing MMR reranking with OpenSearch's vector search
- Optimizing performance for large-scale deployments
- Measuring and tuning diversity metrics
- Real-world applications and success patterns

I’ll share how we built a highly available, multi-tenant, multi-AZ search platform using OpenSearch, supporting modern capabilities like vector search for semantic search. I’ll cover key challenges and how we ensured fault tolerance and scalability across a distributed environment. Additionally, I look forward to learning from the community’s experience in scaling OpenSearch for advanced search use cases.

Attendees will understand how to design a fault-tolerant OpenSearch architecture and integrate vector search to enhance accuracy and search performance. I’ll provide insights into optimising distributed setups for high availability and resilience, ensuring OpenSearch meets enterprise-scale demands.

By sharing best practices and challenges, I hope to drive discussions on multi-region deployments and hybrid search solutions. Highlighting OpenSearch’s capabilities for scalable, AI-powered search can help expand its adoption and evolution for next-gen search applications.

Learn the story of AWS OpenSearch Service, and how we evolved the OpenSearch engine to offer high durability at Petabyte scale without sacrificing on performance. We’ll share our journey of establishing core principles and breaking down complex challenges into manageable deliverables. We will cover some of the key design decisions that helped us to allow new and prevalent software engines to co-exist, without major overhaul of the system. We will walk through our open-source first journey, illuminating common pitfalls and lessons learnt in scaling, correctness and performance, which helped shape the eventual solution. Finally, we’ll reveal how we revolutionised our periodic data backup strategy, transforming a process that handles half a million partitions into one that completes in mere seconds—enabling granular, minute-by-minute backups.

Key Take-aways
- Define your tenets
- Don’t try building an ideal system
- Don’t re-invent the wheel
- Why high judgement calls matters
- Invent to simplify

If you’re still manually sifting through logs to spot issues, it’s time to rethink your observability strategy. This session will show how AI-driven OpenSearch transforms your observability practices by turning logs, metrics, and traces into actionable insights.

Using vector and neural search, how AI models can detect patterns, anomalies, and trends in real time. Learn how to reduce troubleshooting time, predict potential issues before they become critical, and empower data-driven decisions, all while scaling your observability pipeline with OpenSearch’s powerful capabilities.

Operating OpenSearch in multi-tenant environments presents challenges in performance, security, and data isolation. This session explores best practices for optimizing OpenSearch for high-scale, multi-tenant architectures, covering index sharding, access control, and query performance tuning. Attendees will gain insights from real-world deployments and learn how to ensure cost efficiency and high availability in OpenSearch clusters.