Detailed Guide to MongoDB Vector Embedding Similarity Search

Estimated reading time: 7 minutes

Performing similarity searches using vector embeddings in MongoDB allows you to find documents that are semantically or conceptually similar based on the numerical representations of their content. This technique is powerful for applications like recommendation systems, semantic search, and anomaly detection. For a general introduction to MongoDB, you can visit the official MongoDB website.

Understanding the Fundamentals

At its core, similarity search with vector embeddings relies on the idea that similar items can be represented by vectors that are close to each other in a high-dimensional space. The “closeness” is quantified by distance metrics.

Deep Dive into Distance Metrics:

• Cosine Similarity:

  Measures the cosine of the angle between two vectors. It’s particularly useful when the magnitude (length) of the vectors is not as important as their orientation. Values range from -1 (opposite) to 1 (identical), with 0 indicating orthogonality (no similarity in orientation).

  Use Cases: Text similarity, document retrieval, where the length of the document might vary. You can learn more about cosine similarity on resources like Wikipedia’s page on Cosine Similarity.

• Euclidean Distance:

  Calculates the straight-line distance between the endpoints of two vectors in Euclidean space. Smaller values indicate higher similarity. It’s sensitive to both the orientation and the magnitude of the vectors.

  Use Cases: Image similarity based on feature vectors of the same dimensionality, where the intensity or magnitude of features matters. More details on Euclidean distance can be found on its Wikipedia page.

• Dot Product:

  The sum of the products of the corresponding elements of the two vectors. If the vectors are normalized to unit length, the dot product is equivalent to cosine similarity. It’s computationally less expensive than cosine similarity.

  Use Cases: Efficient similarity calculations when vectors are already normalized or when magnitude differences are relevant.

Strategies for Efficient Similarity Search:

• MongoDB Atlas Vector Search (Recommended):

  Leverages specialized indexing techniques optimized for high-dimensional vector data. It provides the $vectorSearch aggregation stage, which significantly simplifies and accelerates similarity queries. You can explore the capabilities of MongoDB Atlas on the MongoDB Atlas product page.

  Key Advantages: Scalability, high performance, integration within MongoDB’s aggregation framework, support for various distance metrics, Approximate Nearest Neighbor (ANN) search for speed. Learn more about Vector Search in the MongoDB Atlas Vector Search documentation.

  Index Creation (Atlas Example):

  
      db.collectionName.createIndex(
        {
          embedding: "vector"
        },
        {
          name: "vectorSearchIndex",
          vectorOptions: {
            type: "hnsw", // Hierarchical Navigable Small World graph (a type of ANN index)
            dimensions: 768, // Replace with the dimensionality of your vectors
            metric: "cosine" // Or "euclidean", "dotProduct"
          }
        }
      );
      

  For details on creating vector search indexes, refer to the MongoDB documentation on creating vector search indexes.

• Manual Implementation (Less Efficient):

  Involves retrieving all vector embeddings from the database and performing the similarity calculations in your application code. This approach becomes prohibitively slow and resource-intensive for large datasets.

  Steps Involved:

  1. Query MongoDB to fetch all documents containing vector embeddings.
  2. For each document, calculate the distance (e.g., cosine similarity) between its embedding and the query vector.
  3. Sort the documents based on the calculated similarity scores.
  4. Return the top-k most similar documents.

  Limitations: High latency, significant memory usage, increased computational load on the application server.

Detailed Breakdown of $vectorSearch in MongoDB Atlas:

db.collectionName.aggregate([
  {
    $vectorSearch: {
      queryVector: [0.1, 0.5, 0.2, ...], // The vector you are searching for similar vectors to
      path: "embedding", // The name of the field in your documents that contains the vector embeddings
      numCandidates: 10, // The number of approximate nearest neighbor candidates to consider during the search. Higher values can improve accuracy but may increase latency.
      limit: 5, // The maximum number of most similar documents to return in the results
      index: "vectorSearchIndexName", // The name of the vector search index you created (optional if only one vector index exists on the collection)
      distanceMetric: "cosine", // Specifies the distance metric to use. Options: "cosine" (default), "euclidean", "dotProduct". Ensure this matches the metric used when creating the index.
      // Optional filter to narrow down the search space before vector search
      // filter: { category: "electronics" }
    },
  },
  {
    $project: {
      _id: 1,
      otherFields: 1,
      similarityScore: { $meta: "vectorSearchScore" }, // Adds a field containing the similarity score calculated by $vectorSearch
    },
  },
]);

Explanation of Parameters: For a comprehensive explanation of the $vectorSearch stage, refer to the MongoDB documentation on the $vectorSearch aggregation stage.

• queryVector: This is the vector you want to find similar vectors to. Its dimensionality must match the dimensionality of the vectors in your embedding field.
• path: Specifies the field in your MongoDB documents that holds the vector embeddings (as an array of numbers).
• numCandidates: This parameter controls the trade-off between search speed and accuracy. A higher value tells the ANN algorithm to explore more potential neighbors, which can lead to more accurate results but might take longer.
• limit: Determines the number of the top most similar documents to be returned by the $vectorSearch stage.
• index: If you have multiple vector search indexes on your collection, you need to specify the name of the index to use for this query. If only one exists, this is optional.
• distanceMetric: Specifies how similarity between vectors is measured. It’s crucial that this matches the metric specified when you created the vector search index.
• filter (Optional): Allows you to apply a standard MongoDB query filter before the vector search is performed. This can be useful for narrowing down the search space based on other criteria.

Post-processing the Results:

The $vectorSearch stage adds a similarityScore to the matching documents (accessible via $meta: "vectorSearchScore" in a $project stage). The meaning of this score depends on the distanceMetric used:

• For "cosine", higher scores (closer to 1) indicate greater similarity.
• For "euclidean", lower scores (closer to 0) indicate greater similarity (it’s a distance).
• For "dotProduct", higher scores indicate greater similarity (especially if vectors are normalized).