Description

kushalkolar

opened on Nov 25, 2025

Issue body actions

We can extend #868 to create a universal widget that can map multiple-arrays of n-dimensional data to arbitrary graphics. Need to brainstorm to get this right.

One idea: add arrays in as "groups". For a given single "group of arrays", I think one way to organize things is that this "array group" should be added in together and they all map to the same Graphic type. The ImageWidget implementation in #868 is an example of how 1 group (representing images) can be represented. I do not think it makes sense to have multiple arrays within a group that map to different graphics, because then you can't define the graphical representation in a simple way.

For example, let's say you want display videos in a few subplots, scatters in a few subplots (representing tracking/keypoints for example), and polygons. Not every subplot will have an image, scatter, and polygons. But for a given graphical-representation, the arrays can be added in as a "group" if they should be managed together (still not entirely sure what this would look like).

If multiple arrays that map to the same type of graphic are to be added to the same subplot, they should be in separate groups, for example: multi-channel imaging data.

The current index displayed would of course be the same across all groups, and all arrays.

Another idea, gives way more flexibility: Allow adding n-dimensional arrays along with a specified graphical representation to arbitrary subplots, NDImageProcessor-like objects are created under the hood and automatically managed and used to generate the sliders and get/set indices to set the currently displayed slice. Maybe a crazy idea, but the implementation might be simpler without requiring specified "groups" of arrays, and it might allow for more customized viz.

Graphics we want to map to:

• Image
• Scatter
• Line (need to think about LOD)
• Heatmap (need to think about LOD and setting visible segments)
• Polygons or image overlays

More thoughts about lines:

• mode to display the entire line with a linear selector
• mode to display a chunk of full time series or other line-data as a window centered around the current timepoint
  • this would also be useful for display (x, y) behavior tracks, with a point selector that denotes the current timepoint and the line is only drawn for a certain window of time around the current time index.
• similar ideas for heatmap
• would also be useful for displaying neural trajectories

Thoughts about scatters:

• Would also be useful to display the scatter for all timepoints (or whatever dimension), and use a point selector tool that denotes the current index in that dimension.

Thoughts on LOD and out of core rendering:

• NDProcessor should have spatial LOD levels for all graphical representations
  • supply additional data arrays that can be loaded for specific extent ranges in the subplot.
  • auto-subsample data based on the extent of the currently visible space in the subplot and the number of screen pixels used. We could pre-define a certain number of levels that are used to auto subsample the data in space.
• Mode to display the "visible window of data", dynamically slice in space (the last 2 or 3 dims). This would be one form of out of core rendering for images, but also useful for lines and scatters. It corresponds to the "mode to display a chunk of time series data as a window centered around the current timepoint" for lines.
  • This mode and the ability to auto-slice in space should be managed in the backend by the NDProcessor. The corresponding mode in the widget would use canvas pointer events to move around the full data.
  • We can display a mini-map of the full data in the corner of the subplot, useful for all graphics.
  • I don't think this mode makes sense with scatters, if you want to view them as sub-sampled densities a heatmap should be used instead.

Some more potential features:

• ability to switch between graphic representations, for example an image vs a line-stack, useful for ephys data

The constructor of this class should probably pass **kwargs to Figure since this will deal with multiple types of graphics.

Once this is in a release, we could probably keep ImageWidget for 1 release and deprecate it, remove it in the following release.

Activity

1. kushalkolar

   changed the title [-]multi-array, n-dimensional widget that maps groups of arrays to arbitrary graphics[/-] [+]multi-array, n-dimensional widget that maps arbitrary arrays to arbitrary graphics[/+] on Nov 25, 2025
2. kushalkolar

   pinned this issue on Dec 3, 2025
3. kushalkolar commented on Dec 10, 2025

   

   kushalkolar

   on Dec 10, 2025

   MemberAuthor

   More actions

   Ideas from discussing with @clewis7 :

   NDProcessors

   An NDProcessor for each type of graphical representation, for example: images, line stacks, scatters.

   Need to think about whether a line-stack timeseries representation would be represented by the same NDProcessor class or have separate subclasses for line-stack and time-representing heatmaps.

   The time-series representations can have a "display_window" property that determines the chunk size of the currently displayed slice over a dimension (usually the time dimension). Can have options to align the graphical representation so that the current index is at the start or middle of the line-stack/heatmap's x-axis dimension.

   Scatters can be represented in different ways, need to think about how to implement these NDProcessors and graphical representations:

   • Data where one scatter is shown and a slider highlights a datapoint (or many datapoints maybe?)
   • The scatter itself changes along a dimension, ex: keypoint tracking data where n-points move in [x, y, z] space over time.

   NDWidget orchestrator class

   NDWidget manages a figure. It manages orchestration of NDProcessors and graphics, as well as the sliders and the "reference indices" that these sliders represent.

   The NDWidget can store a "reference index range" for each dimension that it manages, and these index ranges are used to set the sliders. For example, the index range could be a scientifically relevant unit such as seconds (instead of the usual frame/array index). Each NDProcessor instance has a callable that maps sliders in "reference indices" units to their data array's index. This will allow for arbitrary mapping of time (or other units) to an array index, accounting for non-uniform sampling rates or chunked recordings etc.

   NDWidget manages NDProcessors as individual units, so things like window_funcs have to be defined per-NDProcessor instance.

   Using NDWidget

   Ideas of how to populate the widget:

   ndw = NDWidget(..., **kwargs)  # kwargs passed to Figure
   ndw["subplot-name"].add_nd_image(
     data=nd_img_data,
     name="volume-movie",
     window_funcs=...,
     rgb=True,
    **kwargs  # kwargs get passed to ImageGraphic
   )

   I think the **kwargs could get passed to the Graphic that's used to display it, and all the other args go to the processor, can think and decide this later.

   Each NDProcessor could have a class attribute that defines what graphic it should use?

   class NDImageProcessor:
       graphic = ImageGraphic
   
   class NDLineStackProcessor:
       graphic = LineStack
   
   class NDHeatmapProcessor(NDLineStackProcessor):
       graphic = ImageGraphic
   
   ...

   We can have add_<nd_type> for all the default "NDProcessor-Graphic" pairs that we have premade (for images, time series line stacks or heatmaps, scatters, etc.).
   For custom NDProcessors we can maybe do something like:

   processor = CustomNDProcessor(data, ...)
   ndw["subplot-name"].add(processor, name=...)

   Setting NDProcessor or Graphic properties of managed objects in NDWidget

   An NDProcessor-Graphic pair added to a NDWidget subplot always has to be named. So then the way to access it and set things is:

   ndw["subplot-name"]["nd-data-name"].processor.window_funcs = ...
   ndw["subplot-name"]["nd-data-name"].processor.rgb = False
   
   # for line stacks or a heatmap time series
   ndw["subplot-name"]["neural-traces"].processor.display_window = 5
   
   # change the name
   ndw["subplot-name"]["neural-traces"].name = "new_name"

   Accessing the graphic could be like this maybe?

   ndw["subplot-name"]["neural-traces"].graphic.cmap = "jet"

   This may seem tedious but it makes clear separation of responsibilities.
4. kushalkolar

   self-assigned this

   on Dec 12, 2025
5. kushalkolar commented on Dec 25, 2025

   

   kushalkolar

   on Dec 25, 2025

   MemberAuthor

   More actions

   Need to think a bit about swappable representation of time-series data as lines vs. a heatmap.

   If we have a "linestack", the dims are [n_lines, n_datapoints, 2]. Heatmaps are just images with dims of [rows, col]. We can turn a "linestack" array to a heatmap by just using the y-values, i.e. slice line stack data with: [..., 1]. However one piece of information that is lost is the x-axis positions. We could scale it like this if the values are uniform over the x-axis:

   heatmap = fig[0, 0].add_image(heatmap_data)
   heatmap.world_object.world.scale_x = line_stack_data[:, -1, 0][0] / line_stack_data.shape[1]

   Need to think about the best way to handle this. Do we just rely on the user to know if their data are uniform over x? And we apply the scaling under the hood when viewing as a heatmap? Also need to consider how to scale in x if "reference index" mapping function is used (ex, to map time units such as seconds to integer array indices).

   EDIT: I think the best option is to sample along the x-axis to create evenly spaced columns for displaying heatmaps, can just use linear interpolation. And then scale the x-dim according to the last x-axis value as I showed above.
6. kushalkolar

   closed this as completedon Sep 23, 2026
7. kushalkolar

   unpinned this issue on Sep 23, 2026

Issue actions