What is Milvus?Milvus is an open-source vector database designed for storing, indexing, and managing billions of embedding vectors. It specializes in high-performance similarity search over these vectors. Milvus is built to handle the unique challenges of vector data, such as: - High-Dimensional Data: Efficiently stores and searches vectors with hundreds or thousands of dimensions.
- Scalability: Supports distributed architectures to scale out for massive datasets and high query throughput.
- Real-time Performance: Provides low-latency query responses for similarity searches.
- Diverse Indexing: Supports various indexing techniques to optimize search performance based on specific needs.
- Data Management: Offers features like data persistence, CRUD operations (create, read, update, delete) for vectors, and data filtering capabilities.
Milvus is commonly used in applications requiring semantic search, recommendation systems, image/video recognition, drug discovery, and anomaly detection, where the core task is to find similar items based on their vector representations. 
What is FAISS?FAISS (Facebook AI Similarity Search) is a library developed by Facebook AI Research for efficient similarity search and clustering of dense vectors. It provides a highly optimized collection of algorithms that can search through billions of vectors in sub-second time. Key characteristics of FAISS include: - Algorithm Variety: Offers a wide range of indexing algorithms, including exact search (brute-force) and approximate nearest neighbor (ANN) methods like IVF_FLAT, IVF_PQ, LSH, HNSW, etc., each with different trade-offs between speed, memory usage, and accuracy.
- High Performance: Implements these algorithms with C++ for maximum speed, with Python wrappers for ease of use. It also supports GPU acceleration for further performance gains.
- Memory Efficiency: Designed to work with large datasets that might exceed available RAM by employing techniques like compression.
- Locality: Primarily a library for indexing and searching within a single process or machine, though its indexes can be distributed manually.
FAISS is an engine that focuses purely on the mathematical task of finding nearest neighbors given a set of vectors. Why Milvus is Paired with FAISS for ML?Milvus is not merely paired with FAISS; it integrates FAISS (and other libraries like HNSWlib) as its core indexing and search engine. Milvus leverages FAISS's highly optimized algorithms to perform the actual approximate nearest neighbor (ANN) search on vector data. 
When you insert vectors into Milvus, Milvus takes care of: - Distributed Storage: Storing the vectors across its distributed cluster.
- Data Management: Handling metadata, filtering, and CRUD operations.
- Orchestration: Managing data shards and query distribution.
- Index Building: Internally, for each shard of data, Milvus uses an indexing library like FAISS to build an ANN index. When you specify an index type (e.g.,
IVF_FLAT), Milvus configures and utilizes FAISS (or a similar engine) to construct that specific index from the vector data. - Query Execution: When a search query comes in, Milvus routes it to the relevant shards, performs a search on the FAISS indexes residing on those shards, aggregates the results, and returns them to the user.
In essence, FAISS provides the powerful, low-level algorithms for vector similarity search, and Milvus wraps these algorithms within a complete, scalable, and manageable vector database system. Benefits of Paring Milvus and FAISSThe integration of Milvus and FAISS offers a comprehensive and robust solution for machine learning applications requiring vector similarity search: - Scalability and Performance: FAISS provides optimized search capabilities for individual data segments, while Milvus manages these searches across a distributed cluster. This allows querying billions of vectors with low latency and scales horizontally as data volume grows.
- Ease of Use and Management: Milvus abstracts away the complexities of managing FAISS indexes directly. Users interact with a database system, not just a library. This includes simplified data insertion, deletion, updates, and schema management.
- Robustness and Reliability: Milvus offers features like data persistence, high availability, and fault tolerance, which are not inherent in FAISS alone. This ensures data integrity and continuous service for production ML systems.
- Unified Ecosystem: Milvus provides a unified API and ecosystem for vector search, allowing users to switch between different indexing algorithms (including those from FAISS) and manage their vector data efficiently without needing to implement the underlying distributed logic themselves.
- Rich Features: In addition to basic search, Milvus offers features like scalar filtering, real-time data ingestion, and multi-tenancy, which enhance FAISS's core search functionality and help build complete ML-powered applications.
- Developer Productivity: Developers can focus on building their ML models and applications, using Milvus as a reliable backend for vector search, instead of spending time optimizing FAISS indexes for distributed environments or managing data at a low level.
Data Storage in MilvusMilvus is a vector database that leverages indexing libraries like FAISS (or HNSW, Annoy, IVF_PQ, etc.) for efficient similarity search. While FAISS provides the algorithms and structures for building and searching indexes in memory, Milvus is the system responsible for the comprehensive management, persistence, and distribution of the data. Who is actually storing data? - Milvus Itself: Milvus is the primary system storing and managing your vector data and its associated metadata. It orchestrates the entire data lifecycle, from ingestion to indexing, querying, and persistence.
- Decoupled Storage Layer: Milvus's architecture decouples compute and storage. It doesn't store data directly on the local disks of its query or index nodes. Instead, it utilizes a dedicated, highly scalable, and persistent storage layer.
- Underlying Persistent Storage:
- Object Storage: For the actual vector data and generated index files, Milvus primarily relies on cloud-native object storage solutions such as Amazon S3, MinIO, Azure Blob Storage, or Google Cloud Storage. This provides high durability, availability, and cost-effective scalability.
- Structured Storage (Meta Store): For metadata (like collection schemas, partition information, segment details, and operational logs), Milvus typically uses a robust metadata store like etcd. This ensures consistency and coordination across its distributed components.
- Message Queue: A message queue (e.g., Kafka, Pulsar) is used for reliable data transfer between components, ensuring data integrity during ingestion and processing, and contributing to the system's fault tolerance.
- Data Nodes: Within the Milvus cluster, DataNodes are responsible for ingesting incoming vector data, converting it into internal segments, and persisting these segments to the underlying object storage. They ensure that all ingested data is reliably written to a durable store before being acknowledged.
- Query Nodes and Index Nodes: These nodes operate on data that has been persisted. IndexNodes read raw vector data from object storage to build indexes (e.g., FAISS indexes) and then write these index files back to object storage. QueryNodes load these index files (or parts of them) from object storage into their memory to serve search queries.
In summary, Milvus stores data persistently on a distributed object storage system, coordinated by a metadata store and a message queue. FAISS is an indexing engine used by Milvus, not the storage engine itself. 
|
