Hybrid Search
Properties
- My general description
This type of search combines sparse retrieval (e.g. BM25 algorithm) with dense retrieval (e.g. ANN index). The outputs of these two retrievals are combined through some weighting scheme (e.g Combination of Rankings) to obtain the final result.
Modern vector databases usually employ the BM25 algorithm for hybrid search. This algorithm allows you to specialize keywords to search ( Sparse Retrieval) for in document chunks. Therefore, when performing hybrid search chunks with these keywords will be prioritized in the search result. Most vector databases nowadays allow the specification of the weight that should be given to sparse and dense retrieval, respectively.
This search is very beneficial when you can estimate important keywords (in relation to your dataset) that appear within the user prompt. You can then leverage these keywords to improve the search results. Hybrid search was shown to improve performance on document retrieval (The Chronicles of RAG).