GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — feat(v2): tensor column mode by samsja · Pull Request #886 · docarray/docarray

This repository was archived by the owner on Sep 9, 2026. It is now read-only.

Conversation

samsja commented Dec 2, 2022 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Member

Context

introducing the stacked mode for faster computation and saving memory when dealing with tensor when needed:

class Image(Document):
    tensor: TorchTensor[3, 224, 224]

batch = DocumentArray[Image](
    [Image(tensor=torch.zeros(3, 224, 224)) for _ in range(10)]
)

batch.stack()
assert (batch.tensor == torch.zeros(10, 3, 224, 224)).all()
assert batch.is_stack()
batch.unstack()

We expose as well a context manager

with batch.stacked_mode():
    ...

with batch.unstacked_mode():
   ...

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

github-actions Bot added size/m area/core area/testing component/array labels

Dec 2, 2022

samsja marked this pull request as draft

December 2, 2022 10:26

samsja changed the title ~~Feat tensor column mode~~ feat(v2): tensor column mode

Dec 2, 2022

github-actions Bot added size/l and removed size/m labels

Dec 5, 2022

samsja force-pushed the feat-tensor-column-mode branch from fe10440 to 31d85d8 Compare

December 5, 2022 15:27

samsja requested a review from alexcg1

December 5, 2022 16:40

samsja marked this pull request as ready for review

December 5, 2022 16:41

samsja force-pushed the feat-tensor-column-mode branch from 915406d to 342a726 Compare

December 5, 2022 16:41

JohannesMessner suggested changes

Dec 5, 2022

View reviewed changes

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/mixins/attribute.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/typing/tensor/abstract_tensor.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

github-actions Bot added the area/typing label

Dec 5, 2022

samsja force-pushed the feat-tensor-column-mode branch 3 times, most recently from 63c74c7 to 62671f3 Compare

December 6, 2022 10:01

JohannesMessner suggested changes

Dec 6, 2022

View reviewed changes

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

alexcg1 suggested changes

Dec 6, 2022

View reviewed changes

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py



		Unstacked mode (default):
		In this case a DocumentArray is a list of Documents and each Document owns its data.

alexcg1 Dec 6, 2022

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does it mean to for a Document to "own" data?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

samsja Dec 6, 2022

Copy link

Copy Markdown

Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it means that the data is bound to the lifecycle of the Document. If the Document go out of scope the data as well

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

alexcg1 Dec 6, 2022

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it also be accurate to say the Document stores the data?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

samsja Dec 6, 2022

Copy link

Copy Markdown

Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think here owning the the good term from a computer science point of view

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

alexcg1 reacted with thumbs up emoji

All reactions

👍 1 reaction

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

samsja force-pushed the feat-tensor-column-mode branch from 78b7968 to b6770d2 Compare

December 6, 2022 12:58

samsja added 10 commits

December 6, 2022 13:59


          fix: inherit from cls in da

a574d84

Signed-off-by: Sami Jaghouar <[email protected]>


          fix: move class atribute to top torch tensor

05b90e9

Signed-off-by: Sami Jaghouar <[email protected]>


          feat: add stack and unstack mod

df5fe28

Signed-off-by: Sami Jaghouar <[email protected]>


          feat: add blocker for stack mod

915cbf5

Signed-off-by: Sami Jaghouar <[email protected]>


          feat: add blocker for stack mod

fd7d8a7

Signed-off-by: Sami Jaghouar <[email protected]>


          fix: fix issue with dict iterationg

95316af

Signed-off-by: Sami Jaghouar <[email protected]>


          refactor: init column in stacked mode entry and not init

70d6430

Signed-off-by: Sami Jaghouar <[email protected]>


          fix: fix typo

b290040

Signed-off-by: Sami Jaghouar <[email protected]>


          fix: fix typo

ac26cda

Signed-off-by: Sami Jaghouar <[email protected]>


          feat: add column mode for tensor

f50c27b

Signed-off-by: Sami Jaghouar <[email protected]>

samsja and others added 15 commits

December 6, 2022 14:01


          feat: add warning in list like funciton

bcaad73

Signed-off-by: Sami Jaghouar <[email protected]>


          fix: fix docstring column fields

ce34689

Signed-off-by: Sami Jaghouar <[email protected]>


          feat: add warning in list like funciton

1f57835

Signed-off-by: Sami Jaghouar <[email protected]>


          feat: apply jihannes suggestion

a6abd5e

Signed-off-by: Sami Jaghouar <[email protected]>


          feat: apply aleaddine suggestion

834fada

Co-authored-by: Johannes Messner <[email protected]>
Signed-off-by: samsja <[email protected]>
Signed-off-by: Sami Jaghouar <[email protected]>


          feat: apply jihannes suggestion

bc76613

Signed-off-by: Sami Jaghouar <[email protected]>


          feat: apply jihannes suggestion

17802b3

Signed-off-by: Sami Jaghouar <[email protected]>


          feat: apply jihannes suggestion

052288f

Signed-off-by: Sami Jaghouar <[email protected]>


          fix: docstring for blocked method

b0dd31d

Signed-off-by: Sami Jaghouar <[email protected]>


          feat: add better docstring for context manager

5fd7864

Signed-off-by: Sami Jaghouar <[email protected]>


          fix: mypy

c1bfa6f

Signed-off-by: Sami Jaghouar <[email protected]>


          feat: apply alex suggestion

9a75e80

Co-authored-by: Alex Cureton-Griffiths <[email protected]>
Signed-off-by: samsja <[email protected]>
Signed-off-by: Sami Jaghouar <[email protected]>


          feat: apply alex suggestion

d0d75c0

Co-authored-by: Alex Cureton-Griffiths <[email protected]>
Signed-off-by: samsja <[email protected]>
Signed-off-by: Sami Jaghouar <[email protected]>


          feat: apply alex suggestion

e0fec6b

Signed-off-by: Sami Jaghouar <[email protected]>


          fix: mypy

0b3f538

Signed-off-by: Sami Jaghouar <[email protected]>

samsja force-pushed the feat-tensor-column-mode branch from b6770d2 to 0b3f538 Compare

December 6, 2022 13:01

samsja added 2 commits

December 6, 2022 14:02


          fix: rebase

228277d

Signed-off-by: Sami Jaghouar <[email protected]>


          fix: rebase

6d07939

Signed-off-by: Sami Jaghouar <[email protected]>

alexcg1 suggested changes

Dec 6, 2022

View reviewed changes

alexcg1 left a comment

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed a few things last time round 😅

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py



		Unstacked mode (default):
		In this case a DocumentArray is a list of Documents and each Document owns its data.

alexcg1 Dec 6, 2022

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it also be accurate to say the Document stores the data?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/array.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/mixins/attribute.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/array/mixins/attribute.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

docarray/typing/tensor/ndarray.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

samsja and others added 4 commits

December 6, 2022 15:37


          refactor: fix problem

65ba015

Signed-off-by: Sami Jaghouar <[email protected]>


          feat: apply alex suggestion

baa7f53

Co-authored-by: Alex Cureton-Griffiths <[email protected]>
Signed-off-by: samsja <[email protected]>


          feat: apply alex suggestion

d7c0458

Co-authored-by: Alex Cureton-Griffiths <[email protected]>
Signed-off-by: samsja <[email protected]>


          fix: apply black to suggestion

0049d66

Signed-off-by: Sami Jaghouar <[email protected]>

alexcg1 approved these changes

Dec 6, 2022

View reviewed changes

alexcg1 left a comment

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions


          fix: update docstring

Signed-off-by: Sami Jaghouar <[email protected]>

JohannesMessner approved these changes

Dec 6, 2022

View reviewed changes

github-actions Bot commented Dec 6, 2022

Copy link

Copy Markdown

📝 Docs are deployed on https://ft-feat-tensor-column-mode--jina-docs.netlify.app 🎉

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

samsja merged commit ae03664 into feat-rewrite-v2

Dec 6, 2022

samsja deleted the feat-tensor-column-mode branch

December 6, 2022 15:55

JohannesMessner mentioned this pull request

Dec 13, 2022

Meta: DocArray v2 Roadmap #780

Closed

42 of 47 tasks

Reviewers

2 more reviewers

alexcg1

alexcg1 approved these changes

JohannesMessner

JohannesMessner approved these changes

Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

area/core area/testing area/typing component/array size/l

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

samsja commented Dec 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexcg1 Dec 6, 2022

Choose a reason for hiding this comment

Uh oh!

samsja Dec 6, 2022

Choose a reason for hiding this comment

Uh oh!

alexcg1 Dec 6, 2022

Choose a reason for hiding this comment

Uh oh!

samsja Dec 6, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexcg1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexcg1 Dec 6, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexcg1 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Dec 6, 2022

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

samsja commented Dec 2, 2022 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.