GeezerVison is showing github.com — refactor: count number of documents using hnswlib · Issue #1759 · docarray/docarray
This repository was archived by the owner on Sep 9, 2026. It is now read-only.
This repository was archived by the owner on Sep 9, 2026. It is now read-only.
refactor: count number of documents using hnswlib #1759DescriptionData storage in HnswDocumentIndex works in the following way:
- Vectors are stored on disk using
hnswlib.
- All other types of data are saved in an SQLITE database.
One of the operations we frequently perform is determining the total number of documents (num_docs()). However, the only way to get number of documents from SQLITE is by scanning the entire table. Even though we've made efforts to reduce the number of times we use this functionality (#1729), it's still a time-consuming process.
For better performance, let's do the following: instead of scanning the SQLITE table, we can use hnswlib's get_current_count function to quickly get the number of documents in the index.
But there's a potential issue with this approach. What if documents don't have associated vectors? get_current_count would return 0.
We have two potential solutions:
- Notify/Warn users about this behavior and return 0.
- Use to the older method of counting using the SQL table if vector-less documents are detected.
Activityshobhit9957 commented on Aug 24, 2023 /attempt #1759
I would like to solve this issue. Can I get some more help. JoanFM commented on Aug 24, 2023 Hey @shobhit9957 ,
Sure, the main refactoring to be done is to change the code here:
def num_docs(self) -> int:
"""
Get the number of documents.
"""
if self._num_docs == 0:
self._num_docs = self._get_num_docs_sqlite()
return self._num_docsto something that does not rely on the _get_num_docs_sqlite but from a new private method _get_num_docs_hnsw shobhit9957 commented on Aug 24, 2023 so I should replace the _get_num_docs_sqlite to _get_num_docs_hnsw is that correct?
I'm just a beginner, to this open-source-community. Just trying my hands out there...please help me out. JoanFM commented on Aug 24, 2023 Yes, this would be the right approach shobhit9957 commented on Aug 24, 2023 Thanks! will do the PR Soon. shobhit9957 commented on Aug 24, 2023 Hey Joan. Submitted the PR, please check if there are any other issues or mistakes I've done in the PR, I would be happy to solve my mistakes and submit the PR again Joan. Thanks! shobhit9957 commented on Aug 24, 2023 def num_docs(self) -> int:
"""
Get the number of documents.
"""
if self._num_docs == 0:
# Replace the old method with the new one
self._num_docs = self._get_num_docs_hnsw()
return self._num_docs
def _get_num_docs_hnsw(self) -> int:
"""
Get the number of documents using the HNSW method.
This method should return the count of documents using the new technique.
"""
# Implement your logic here to get the count using the HNSW method
# For example, you might access some data or perform calculations
# that help you quickly determine the number of documents.
# Then return the count.
return calculated_num_docs
is this correct? munish0838 commented on Oct 22, 2023 Hi, is this issue fixed or needs contribution yet.
Would like to contribute JoanFM commented on Oct 22, 2023 munish0838 commented on Oct 24, 2023 I would like to start working on this issue JoanFM commented on Oct 24, 2023 be my guest. I believe there are parts of the plan that were not applied yet. Kulraj69 commented on Oct 3, 2025 Hi Joan, Is this issue still open? I would like to contribute.
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|