GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — feat(v2): tensor column mode by samsja · Pull Request #886 · docarray/docarray

This repository was archived by the owner on Sep 9, 2026. It is now read-only.

feat(v2): tensor column mode #886

Merged

samsja merged 53 commits into feat-rewrite-v2 from feat-tensor-column-mode

Dec 6, 2022

Uh oh!

There was an error while loading. Please reload this page.

Merged

feat(v2): tensor column mode #886

Show file tree

Hide file tree

Changes from 1 commit

Commits

Show all changes

53 commits

Select commit Hold shift + click to select a range

a574d84

fix: inherit from cls in da

samsja Nov 29, 2022

05b90e9

fix: move class atribute to top torch tensor

samsja Nov 29, 2022

df5fe28

feat: add stack and unstack mod

samsja Dec 1, 2022

915cbf5

feat: add blocker for stack mod

samsja Dec 1, 2022

fd7d8a7

feat: add blocker for stack mod

samsja Dec 1, 2022

95316af

fix: fix issue with dict iterationg

samsja Dec 1, 2022

70d6430

refactor: init column in stacked mode entry and not init

feat: add column mode for tensor

feat: add setter as property

feat: add numpy to stack mod

samsja Dec 5, 2022

4f73b9e

feat: add stack classmethod

samsja Dec 5, 2022

9cec782

feat: embedding type (#877)

JohannesMessner Dec 1, 2022

3506501

refactor: halfway trough mypy not complaining

samsja Dec 5, 2022

f148857

refactor: almost the end of mypy

chore: add docstring for stack mode

samsja Dec 5, 2022

f3d91c7

chore: add docstring for document array

fix: apply johannes suggestion

samsja Dec 5, 2022

52a5e69

fix: apply johannes suggestion

samsja Dec 5, 2022

3b91d61

fix: apply johannes suggestion

samsja Dec 5, 2022

6866bc6

fix: shorten docstring

samsja Dec 6, 2022

c801689

fix: apply johannes suggestion

samsja Dec 6, 2022

93e1892

refactor: apply johannes suggestion

samsja Dec 6, 2022

73d2fda

fix: apply johannes suggestion

samsja Dec 6, 2022

bcaad73

feat: add warning in list like funciton

samsja Dec 6, 2022

ce34689

fix: fix docstring column fields

samsja Dec 6, 2022

1f57835

feat: add warning in list like funciton

samsja Dec 6, 2022

a6abd5e

feat: apply jihannes suggestion

samsja Dec 6, 2022

834fada

feat: apply aleaddine suggestion

samsja Dec 6, 2022

bc76613

feat: apply jihannes suggestion

samsja Dec 6, 2022

17802b3

feat: apply jihannes suggestion

samsja Dec 6, 2022

052288f

feat: apply jihannes suggestion

samsja Dec 6, 2022

b0dd31d

fix: docstring for blocked method

samsja Dec 6, 2022

5fd7864

feat: add better docstring for context manager

feat: apply alex suggestion

samsja Dec 6, 2022

d0d75c0

feat: apply alex suggestion

samsja Dec 6, 2022

e0fec6b

feat: apply alex suggestion

refactor: fix problem

samsja Dec 6, 2022

baa7f53

feat: apply alex suggestion

samsja Dec 6, 2022

d7c0458

feat: apply alex suggestion

samsja Dec 6, 2022

0049d66

fix: apply black to suggestion

samsja Dec 6, 2022

1286876

fix: update docstring

samsja Dec 6, 2022

File filter

Filter by extension .py (3) All 1 file type selected

Viewed files

Clear filters

Conversations

Failed to load comments.

Uh oh!

There was an error while loading. Please reload this page.

Jump to

Jump to file

Failed to load files.

Uh oh!

There was an error while loading. Please reload this page.

Diff view

Unified

Split

Hide whitespace

Diff view

Unified

Split

Hide whitespace

Previous commit

Next commit

feat: apply alex suggestion

Co-authored-by: Alex Cureton-Griffiths <[email protected]>
Signed-off-by: samsja <[email protected]>

Loading branch information

Uh oh!

There was an error while loading. Please reload this page.

samsja and alexcg1 authored Dec 6, 2022

commit baa7f537fe0ddaa7bc03a4e5530486444b155f9c

There are no files selected for viewing

28 changes: 14 additions & 14 deletions 28 docarray/array/array.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -66,48 +66,48 @@ class Image(Document):


		If your DocumentArray is homogeneous (i.e. follows the same schema), you can access
		the field at the DocumentArray level (for example `da.tensor`). You can also set
		them, with `da.tensor = np.random.random([10, 100])`
		fields at the DocumentArray level (for example `da.tensor`). You can also set
		fields, with `da.tensor = np.random.random([10, 100])`


		A DocumentArray can be in one of two modes: unstacked mode and stacked mode.


		Unstacked mode (default):
Comment thread samsja marked this conversation as resolved. Resolved Uh oh! There was an error while loading. Please reload this page.
		In this case a DocumentArray is a list of Documents and each Document owns its data.
Comment thread alexcg1 Dec 6, 2022 Copy link Copy Markdown Contributor There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. What does it mean to for a Document to "own" data? Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. All reactions samsja Dec 6, 2022 Copy link Copy Markdown Member Author There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. it means that the data is bound to the lifecycle of the Document. If the Document go out of scope the data as well Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. All reactions alexcg1 Dec 6, 2022 Copy link Copy Markdown Contributor There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. Would it also be accurate to say the Document stores the data? Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. All reactions samsja Dec 6, 2022 Copy link Copy Markdown Member Author There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. I think here owning the the good term from a computer science point of view Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. alexcg1 reacted with thumbs up emoji All reactions 👍 1 reaction
		The getter and setter shown above returns a list of the fields of each Document
		(or a DocumentArray if the field is a nested Document). This list/DocumentArray
		The getter and setter shown above return a list of the fields of each Document
		(or DocumentArray if the field is a nested Document). This list/DocumentArray
		is created on the fly. The setter sets the field of each Document to the value
		of the list/DocumentArray/Tensor passed as parameters.

		This list-like behavior is not always optimal, especially when you want
		to process data in batches or perform operations involving matrix computation.
		This is where the stack mode of the DocumentArray comes in handy.

		In stacked mode tensor-like fields of every Document are stored on the
		DocumentArray level, as one stacked tensor. This enables operations on the entire
		In stacked mode, tensor-like fields of every Document are stored on the
		DocumentArray level as one stacked tensor. This enables operations on the entire
		batch without iterating over the DocumentArray.
		In this mode the Documents in in the Document don't own the data anymore but just
		In this mode the Documents in the DocumentArray no longer own the data but just
		reference the data in the DocumentArray's tensor.
		Operations like `da.append` are not supported because
		they are too slow. For these operations you should use unstacked
		mode instead.

		To switch from stacked to unstacked mode (or vice-versa), call `da.unstack()` or
		`da.stack`. There are also two context manager for these modes:
		`da.stack`. There are also two context managers for these modes:
		`with da.stack_mode():` and `with da.unstack_mode():`
Comment thread samsja marked this conversation as resolved. Resolved Uh oh! There was an error while loading. Please reload this page.

		see {meth}`.stack` and {meth}`.unstack` for more information.

		You should use unstacked mode, if:
		You should use unstacked mode if you want to:

		* You want to insert, append, delete, or shuffle Documents in your DocumentArray
		* Intend to separate the DocumentArray into smaller batches later on
		* Insert, append, delete, or shuffle Documents in your DocumentArray.
		* Separate the DocumentArray into smaller batches later on.

		You should use stacked mode, if you:
		You should use stacked mode if you want to:

		* You want to process the entire DocumentArray as one batch
		* Are using the DocumentArray in an ML model for training or inference
		* Process the entire DocumentArray as one batch.
		* Use the DocumentArray in an ML model for training or inference.

		"""

Expand Down

4 changes: 2 additions & 2 deletions 4 docarray/array/mixins/attribute.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -51,10 +51,10 @@ def _set_array_attribute(
		field: str,
		values: Union[List, AbstractDocumentArray, 'TorchTensor', 'NdArray'],
		):
		"""Set all document if this DocumentArray with the passed values
		"""Set all Documents in this DocumentArray using the passed values

		:param field: name of the fields to extract
		:values: the values to set at the document array level
		:values: the values to set at the DocumentArray level
		"""
		if (
		self.is_stacked()
Expand Down

2 changes: 1 addition & 1 deletion 2 docarray/typing/tensor/ndarray.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -196,5 +196,5 @@ def _flush_tensor_to_proto(pb_msg: 'NdArrayProto', value: 'NdArray'):
		def __docarray_stack__(
		cls: Type[T], seq: Union[List['NdArray'], Tuple['NdArray']]
		) -> T:
		"""Stack a sequence of ndarray into a single ndarray."""
		"""Stack a sequence of ndarrays into a single ndarray."""
		return cls.from_ndarray(np.stack(seq))

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

feat(v2): tensor column mode #886

Uh oh!

feat(v2): tensor column mode #886

Uh oh!

Uh oh!

Diff view

Diff view

Uh oh!

There are no files selected for viewing

Uh oh!

alexcg1 Dec 6, 2022

Choose a reason for hiding this comment

Uh oh!

samsja Dec 6, 2022

Choose a reason for hiding this comment

Uh oh!

alexcg1 Dec 6, 2022

Choose a reason for hiding this comment

Uh oh!

samsja Dec 6, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!