GeezerVison is showing github.com — HnswDocumentIndex treats document IDs as string, they can be str, int, ID · Issue #1850 · docarray/docarray
This repository was archived by the owner on Sep 9, 2026. It is now read-only.
This repository was archived by the owner on Sep 9, 2026. It is now read-only.
HnswDocumentIndex treats document IDs as string, they can be str, int, ID #1850DescriptionInitial Checks
- I have read and followed the docs and still think this is a bug
Description
I noticed this behavior when I wanted to access multiple documents in the index:
@requests(on='/find')
def find(self, docs: DocList[QuoteFile], **_) -> DocList[QuoteFile]:
return self._cache_di[docs.id]And when I issue POST /find with body {"data":[{"id":"300055"}]}, this code yields:
"/Users/oytuntez/motaword/jina-documents/venv/lib/py…
line 544, in _get_docs_sqlite_doc_id
hashed_ids = tuple(self._to_hashed_id(id_) for
id_ in doc_ids)
File
"/Users/oytuntez/motaword/jina-documents/venv/lib/py…
line 544, in <genexpr>
hashed_ids = tuple(self._to_hashed_id(id_) for
id_ in doc_ids)
File
"/Users/oytuntez/motaword/jina-documents/venv/lib/py…
line 445, in _to_hashed_id
return
int(hashlib.sha256(doc_id.encode('utf-8')).hexdigest…
16) % 10**18
AttributeError: 'int' object has no attribute
'encode' Upon investigation, I saw that most of HnswDocumentIndex treats IDs as str. However, it is my understanding that IDs can be int, see this type definition:
class ID(str, AbstractType):
"""
Represent an unique ID
"""
@classmethod
def _docarray_validate(
cls: Type[T],
value: Union[str, int, UUID],
...I think ID values should be cast to str if necessary (it would be in _to_hashed_id case).
Example Code
No response
Python, DocArray & OS Version
Python 3.8.12
docarray==0.40.0
Affected Components
ActivityJoanFM commented on Feb 8, 2024 hey @oytuntez ,
thanks for the issue reported, would it be possible for you to share a code snippet showing the issue with DocArray code, avoiding as much as possible other dependencies?
Thanks ai-naymul commented on May 4, 2024
I noticed this behavior when I wanted to access multiple documents in the index:
@requests(on='/find')
def find(self, docs: DocList[QuoteFile], **_) -> DocList[QuoteFile]:
return self._cache_di[docs.id]And when I issue POST /find with body {"data":[{"id":"300055"}]}, this code yields:
"/Users/oytuntez/motaword/jina-documents/venv/lib/py…
line 544, in _get_docs_sqlite_doc_id
hashed_ids = tuple(self._to_hashed_id(id_) for
id_ in doc_ids)
File
"/Users/oytuntez/motaword/jina-documents/venv/lib/py…
line 544, in <genexpr>
hashed_ids = tuple(self._to_hashed_id(id_) for
id_ in doc_ids)
File
"/Users/oytuntez/motaword/jina-documents/venv/lib/py…
line 445, in _to_hashed_id
return
int(hashlib.sha256(doc_id.encode('utf-8')).hexdigest…
16) % 10**18
AttributeError: 'int' object has no attribute
'encode' Upon investigation, I saw that most of HnswDocumentIndex treats IDs as str. However, it is my understanding that IDs can be int, see this type definition:
class ID(str, AbstractType):
"""
Represent an unique ID
"""
@classmethod
def _docarray_validate(
cls: Type[T],
value: Union[str, int, UUID],
...I think ID values should be cast to str if necessary (it would be in _to_hashed_id case).
Hi @JoanFM,
Hope you are doing great as well. I gotta this issue, I think the _to_hashed_id method in the HnswDocumentIndex class expects a string input for hashing, but it receives an integer. That's why this occurs. Here is the method that cause the error:
Here is my proposed solution:
Let me know what's your thought on that.
@oytuntez @JoanFM JoanFM commented on May 4, 2024 can you please provide an isolated code to reproduce the issue? ai-naymul commented on May 5, 2024
can you please provide an isolated code to reproduce the issue?
Hey I have tried to simulate the scenario where an integer ID is passed to a method that expects a string ID and tries to hash it, here it is:
import hashlib
def _to_hashed_id(doc_id):
return int(hashlib.sha256(doc_id.encode('utf-8')).hexdigest(), 16) % 10**18
doc_ids = [300055] # This should be a string, not an integer
hashed_ids = tuple(_to_hashed_id(id_) for id_ in doc_ids)
That would be great if @oytuntez can give us the actual reproducable code!! ai-naymul commented on Jun 12, 2024 JoanFM commented on Jun 12, 2024 no, waiting for reproducible code.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|
