GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — refactor: hnswlib performance by JoanFM · Pull Request #1727 · docarray/docarray

This repository was archived by the owner on Sep 9, 2026. It is now read-only.

Conversation

JoanFM commented Jul 26, 2023

Copy link

Copy Markdown

Member

No description provided.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Joan Fontanals Martinez added 2 commits

July 26, 2023 12:44


          refactor: hnswlib performance

9aebf2e

Signed-off-by: Joan Fontanals Martinez <[email protected]>


          Merge branch 'main' of https://github.com/docarray/docarray into refa…

02849c4

…ctor-hnswlib-performance

github-actions Bot added size/s area/core labels

Jul 26, 2023

codecov Bot commented Jul 26, 2023 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Codecov Report

Patch coverage: 76.59% and project coverage change: -0.01% ⚠️

Comparison is base (896c20b) 84.46% compared to head (c062c8c) 84.45%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1727      +/-   ##
==========================================
- Coverage   84.46%   84.45%   -0.01%     
==========================================
  Files         134      134              
  Lines        8798     8834      +36     
==========================================
+ Hits         7431     7461      +30     
- Misses       1367     1373       +6

Flag	Coverage Δ
docarray	`84.45% <76.59%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
docarray/index/backends/hnswlib.py	`74.28% <76.59%> (+0.58%)`	⬆️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          fix: reconstruct on filter

3f56d44

Signed-off-by: Joan Fontanals Martinez <[email protected]>

JoanFM force-pushed the refactor-hnswlib-performance branch from f7bd4d0 to 3f56d44 Compare

July 27, 2023 01:11

github-actions Bot added the area/testing label

Jul 27, 2023


          fix: reconstruct from proper type

f8bb5f9

Signed-off-by: Joan Fontanals Martinez <[email protected]>

JoanFM force-pushed the refactor-hnswlib-performance branch from 4f6c270 to f8bb5f9 Compare

July 27, 2023 03:17

Joan Fontanals Martinez added 3 commits

July 27, 2023 05:56


          fix: get the proper embedding back

ae33fe6


          chore: avoid extra debugging (#1730)

a91e23e

Signed-off-by: Joan Fontanals Martinez <[email protected]>


          Merge branch 'main' of https://github.com/docarray/docarray into refa…

fa9b9ce

…ctor-hnswlib-performance

github-actions Bot added size/m and removed size/s labels

Jul 27, 2023


          Merge branch 'get-current-count-hnswlib' of https://github.com/docarr…

79b0fea

…ay/docarray into refactor-hnswlib-performance

Signed-off-by: Joan Fontanals Martinez <[email protected]>

JoanFM force-pushed the refactor-hnswlib-performance branch from 3b0922a to 79b0fea Compare

July 27, 2023 05:33

Joan Fontanals Martinez added 5 commits

July 28, 2023 02:02


          chore: add comment to take over later

58ee168


          test: refactor hnswlib test subindex

83e55f5

Signed-off-by: Joan Fontanals Martinez <[email protected]>


          Merge branch 'test-refactor' of https://github.com/docarray/docarray …

281cf7e

…into refactor-hnswlib-performance


          chore: fix test

2ea5804


          Merge branch 'main' of https://github.com/docarray/docarray into refa…

1c8337e

…ctor-hnswlib-performance

JoanFM force-pushed the refactor-hnswlib-performance branch 2 times, most recently from fdfcb56 to 74a187e Compare

July 28, 2023 02:27


          chore: apply optim

8c0bf6c

Signed-off-by: Joan Fontanals Martinez <[email protected]>

JoanFM force-pushed the refactor-hnswlib-performance branch from 74a187e to 8c0bf6c Compare

July 28, 2023 02:32

JoanFM marked this pull request as ready for review

July 28, 2023 18:03


          Merge branch 'main' into refactor-hnswlib-performance

c062c8c

github-actions Bot commented Jul 30, 2023

Copy link

Copy Markdown

📝 Docs are deployed on https://ft-refactor-hnswlib-performance--jina-docs.netlify.app 🎉

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

JoanFM requested a review from jupyterjazz

July 30, 2023 13:16

JoanFM requested review from JohannesMessner, jupyterjazz and samsja and removed request for JohannesMessner, jupyterjazz and samsja

July 30, 2023 13:16

samsja reviewed

Jul 31, 2023

View reviewed changes

Comment thread

docarray/index/backends/hnswlib.py

		rows = self._execute_filter(filter_query=filter_query, limit=limit)
		return DocList[self.out_schema](self._doc_from_bytes(blob) for _, blob in rows) # type: ignore[name-defined]
		hashed_ids = [doc_id for doc_id, _ in rows]
		embeddings: OrderedDict[str, list] = OrderedDict()

samsja Jul 31, 2023

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why specifically OrderedDict? I think normal dict in python will already be ordered by default

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

JoanFM Jul 31, 2023

Copy link

Copy Markdown

Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would still like to be specific about the requirement for order

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

jupyterjazz reviewed

Jul 31, 2023

View reviewed changes

jupyterjazz left a comment

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have any measure on how much faster hnsw works now?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

docarray/index/backends/hnswlib.py

		self._hnsw_indices[col_name] = self._create_index(col_name, col)
		self._logger.info(f'Created a new index for column `{col_name}`')
		if self._hnsw_indices[col_name].space == 'cosine':
		cosine_metric_index_exist = True

jupyterjazz Jul 31, 2023

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we care about this?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

JoanFM Jul 31, 2023

Copy link

Copy Markdown

Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because for cosine, HNSWLib normalizes the vectors, and then if we retrieve, they have chanded, so no consistent API can be provided

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

JoanFM commented Jul 31, 2023

Copy link

Copy Markdown

Member Author

do we have any measure on how much faster hnsw works now?

We have observed around 10% improvement

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

JoanFM requested review from jupyterjazz and samsja

July 31, 2023 08:36

jupyterjazz approved these changes

Jul 31, 2023

View reviewed changes

JoanFM merged commit a643f6a into main

Jul 31, 2023

JoanFM deleted the refactor-hnswlib-performance branch

July 31, 2023 08:39

jupyterjazz mentioned this pull request

Aug 1, 2023

Release Notes v0.37.0 #1740

Closed

Reviewers

JohannesMessner

Awaiting requested review from JohannesMessner

samsja

Awaiting requested review from samsja

1 more reviewer

jupyterjazz

jupyterjazz approved these changes

Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

area/core area/testing size/m

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

JoanFM commented Jul 26, 2023

Uh oh!

codecov Bot commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Jul 30, 2023

Uh oh!

samsja Jul 31, 2023

Choose a reason for hiding this comment

Uh oh!

JoanFM Jul 31, 2023

Choose a reason for hiding this comment

Uh oh!

jupyterjazz left a comment

Choose a reason for hiding this comment

Uh oh!

jupyterjazz Jul 31, 2023

Choose a reason for hiding this comment

Uh oh!

JoanFM Jul 31, 2023

Choose a reason for hiding this comment

Uh oh!

JoanFM commented Jul 31, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Jul 26, 2023 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.