Qdrant

My Master Thesis:

One of the newer databases that was released in 2021 is Qdrant. This database is written in Rust and uses a custom version of HNSW. It supports both constrained and hybrid search. Qdrant uses rule-based optimization to determine what type of index to use for query execution. Qdrant leverages heavily quantization of vectors to speed up its search.

• Qdrant Academy
  • Free self-paced courses from beginner to expert with certification
• Multitenancy
• computer_science/databases/Hybrid Cloud
• Distributed Qdrant
  • Raft Consensus
    • https://raft.github.io/
  • Distributed Deployment of Qdrant Cluster with Sharding & Replicas

# Qdrant Sync With SQL

• Approaches for syncing Qdrant with SQL database like Postgres

• Practice of combining multiple searches with some weighting scheme
• RRF with custom weighting
  • Weight adjustment for hybrid prefetch statements
  • FormulaQuery with prefetch results

IDF Modifier

For many search algorithms, it is important to consider how often an item occurs in a collection. Intuitively speaking, the less frequently an item appears in a collection, the more important it is in a search.

This is also known as the Inverse Document Frequency (IDF). It is used in text search engines to rank search results based on the rarity of a word in a collection.

IDF depends on the currently stored documents and therefore can’t be pre-computed in the sparse vectors in streaming inference mode. In order to support IDF in the sparse vector index, Qdrant provides an option to modify the sparse vector query with the IDF statistics automatically.

The only requirement is to enable the IDF modifier in the collection configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")

client.create_collection(
    collection_name="{collection_name}",
    vectors_config={},
    sparse_vectors_config={
        "text": models.SparseVectorParams(
            modifier=models.Modifier.IDF,
        ),
    },
)

Qdrant uses the following formula to calculate the IDF modifier:

Where:

• N is the total number of documents in the collection.
• n is the number of documents containing non-zero values for the given vector element.