Mastodon beta-26: plays nice with Fiji, the branch-graph, and the grapher views.

This release ships a large number of new features, focused on ease-of-life for the user, new data structures, interoperability with Fiji, scripting and handling of images stored remotely. It was built with new contributors that joined the governance. We put some highlights below. But in short:

• Mastodon integration with Fiji is better now. You can create a Mastodon project directly on a 'normal' image opened in Fiji; you can open a Mastodon project file by drag and drop; and Mastodon can be scripted using the Fiji script editor.

• There is a new view called the Grapher. It allows plotting and exploring numerical feature values in a XY graph, and can be synchronized with other view types.

• The data model has a new component: the branch-graph. The branch graph is built on the lineage graph and represent each branch (for instance, the trajectory of a cell between two divisions) as a single object: a branch. This allows views that are based on the branch graph to be more synthetic (and more responsive; fewer objects to render). This allows also computing numerical features on the branch themselves, for instance the duration of a cell cycle. The TrackScheme and BDV views can display and interact with the branch-graph. And it is used in a new TrackScheme mode: the hierarchy view.

The features described below:

• Create a Mastodon project directly from an image opened in Fiji.
• A new data structure and the associated views: the branch graph.
• Support for OME-NGFF and remotely hosted datasets.
• The launcher has a list of recent projects.
• Open Mastodon projects by drag & drop on the Fiji toolbar.
• A grapher for feature values.
• Scripting Mastodon in the Fiji script editor.
• Mastodon can open a project even if the image data is missing.
• Movies can be recorded from the BDV views.
• Auto-linking mode in BDV views.
• Ask to save before closing Mastodon.
• Extract a Fiji time-lapse following a track.
• Other changes.

Create a Mastodon project directly from an image opened in Fiji.

It is now possible to create a new Mastodon project directly from an image opened in Fiji. The core code to wrap a SpimData around an ImagePlus is based on existing code by Tobias in the BDV Fiji plugins. The proper handling of this feature in Mastodon mainly requires dealing with warning the user and ultimately offering exporting the image to a BDV file.

It works like this:

The launcher "New Mastodon project" as a new option, listing the images opened in Fiji as possible source.

Project created this way looks like classic Mastodon project, as if the image source had only 1 resolution level. The LUT and display scale are imported from the ImagePlus as well.

The image data in Mastodon wraps the ImagePlus data. If the user modify the image in the ImagePlus window, the modifications will be shown, and used, in Mastodon:

Because of this we have to be extra careful when the user closes the ImagePlus. So the ImagePlus window is modified to listen to the user closing the image, and shows this message: If the user clicks 'Yes', both the image and all Mastodon windows are closed.

When the user wants to save such a Mastodon project, they are offered to export the image to a BDV file:

It then shows the familiar dialog of the BDV export (the code is copied from the BDV Fiji plugin):

After the export, the user is prompted to save the Mastodon project in a standard way. The Mastodon session will now run using the new BDV file as source for the image data, not on the ImagePlus anymore. The warning that appeared when trying to close the image is removed.

If the user chooses not to export to a BDV file, a standard project file will be created, but with a dataset field pointing to where the image file is saved (as inferred from the FileInfo of the ImagePlus). Such a project can now be reloaded by Mastodon normally, without warning.

A new data structure and the associated views: the branch graph.

What is a branch-graph?

A branch graph is a simplified view of a source graph, called here the linked graph, that offers a coarser level of details by representing branches of the linked graph as a single vertex in the branch graph.

A branch in the linked graph is defined as sequence of connected vertices, where all except the last vertex have exactly one outgoing edge, and all except the first vertex have exactly one incoming edge.

If the linked graph has a branch like this:

v0 → v1 → v2 → v3

It's represented in the branch graph as a single vertex: bv0 Where bv0 links to all the nodes, and edges: v0 → v1 → v2 → v3.

A small graph like this:

                 v3 → v4 → v5
              /
 v0 → v1 → v2
              \
                 v6 → v7 → v8

will be represented by the following branch graph:

       bv1
     /
 bv0
     \
       bv2

The vertices and edge of the branch graph are linked to the vertices and edges of the original graph as follows:

• bv0 is linked to v0 → v1 → v2
• bv1 is linked to v3 → v4 → v5
• bv2 is linked to v6 → v7 → v8
• The edge bv0 → bv1 is linked to the edge v2 → v3
• The edge bv0 → bv2 is linked to the edge v2 → v6

In the context of Mastodon, the branch-graph is important for two main reasons:

• Views that are based on the branch-graph have a good performance, because the branch-graph is sparse compared to the core-graph it is based on.
• Biology applications need the branch-graph. There are many important features that are defined on the branch-graph, such as cell-cycle length, cell excursion, cell hierarchy, etc. The branch-graph offers a convenient and sensible basis to compute and reason on them.

The branch-graph in Mastodon.

With this new version, Mastodon offers support for the branch-graph, and several views to interact with it. Also, numerical features can be computed and interrogated on the branch-graph. We list here the new views related to the branch graph, and the new coloring modes and features associated to it.

The new TrackScheme view for branch graph.

To create a TrackScheme view of the branch graph, either shift-click on the track scheme button on the main window, or use the Window > New TrackScheme Branch menu item

For instance, here is a dense lineage viewed in TrackScheme:

and here is its equivalent visualization with the branch-graph:

By zooming in, we can see that the branches are represented by only one vertex in the branch graph. Again comparing with the normal graph we have:

Notice that the TrackScheme window has a button Regen branch graph. The modifications made to the linked graph are not automatically synchronized to the branch graph, for performance reasons. This button forces the branch-graph getting in sync.

The new Hierarchy view.

We made a new view, derived directly from the branch TrackScheme: the TrackScheme Hierarchy. It is a view of a lineage like TrackScheme, except that the Y displays the level of the branch. That is: how many times the cell divided since the first mother cell. By default it is painted with curved lines:

The new table view.

The branch graph called for a new table view, that could display items and features also from the branch graph. The table window has now 4 tables, 2 for the linked graph and 2 for the branch graph:

Navigation and selection events with the branch graph.

The branch graph views still share selection and events with the normal views:

Selection

• Selecting a branch spot selects all the spots and links that are linked to it.
• Selecting a branch link selects the link it is linked to. And conversely.
• Conversely, a branch spot will be shown as selected if and only if all the spots and links that are linked to it are selected.

Highlight.

• Highlighting a branch spot will highlight the last spot it is linked to. And conversely.
• Highlighting a branch link will highlight the link it is linked to. And conversely.

Focus.

• Focusing on a branch spot will focus the last spot it is linked to. And conversely.

Group-handle and navigation.

• Navigating to a branch spot will navigate to the last spot it is linked to. And conversely.
• Navigating to a branch link will navigate to the link it is linked to. And conversely.

Branch features and tag-sets.

Tag-set for branch items.

Again, the branch-graph does not have a separate model for tag-sets, but adapts the tag-set model of the core graph. This is done in a similar manner than for the selection:

• Tagging a branch spot tags all the spots and links that are linked to it.
• Tagging a branch link tags the link it is linked to. And conversely.
• Conversely, a branch spot will be shown as tagged with a certain tag if and only if all the spots and links that are linked to it are tagged with the same tag.

This gives this kind of view:

But observe below for instance that the branch tagged in blue in the core-graph does not appear as blue in the branch graph becomes some links are not tagged:

Branch-graph features.

The Mamut feature computation framework is augmented so that it can deal with feature computers for branch items, and provide the parameters they need for computation. A branch-graph feature can depend on a core-graph feature and conversely. Their computation is configured in the same UI:

Notice that most of the branch features are defined for the branch spots. A branch (a cell between two divisions) is really meant to be represented by a branch spot.

Coloring core-graph and branch graph with branch features.

The Feature color mode collection.

Now that we have numerical features defined over branches, we can use them to color the views. In Mastodon you can create feature color modes, built on the spot and link features. Normally we should manage a second collection of coloring modes for the branch-graph views. But we wanted to avoid multiplying the number items to configure.

So there is still only one collection of feature color modes, common to the linked graph and the branch graph. The coloring menu simply filters them based on whether a mode is suitable for a certain view. For instance, it is possible to use a color mode defined for a branch on a normal TrackScheme, because it will apply the branch color to all the spots and links of the branch. However, color modes that are defined over spots will not be active for the branch graph views. For instance:

The data table view is special, as it displays both the core-graph and the branch-graph. So there are two coloring menus:

The Feature color mode creation UI.

But now, how to create these color modes?

Since the feature-model is happy to welcome within in features that apply to different object types, we reuse the UI we have to specify feature color modes. Again, the goal was to avoid multiplying the number of UI windows we show to the user. The vertex and edge mode selectors are now in a scroll pane to show all the modes, and the non selected ones are greyed for clarity. We also tried to improve the labels, so that we show 'Spot' and 'Link' in the Mamut app, instead of 'Vertex' and 'Edge', again to make it simpler and clearer to the user:

Support for OME-NGFF and remotely hosted datasets.

Mastodon can now be opened on images that are hosted on a server (such as ZARR files hosted on a S3 server). This feature extends the support of the images hosted on a BDV server and supports login/password authentification.

To do so we use the OMEZarrS3Opener of the MoBIE project. The Mastodon update-site in Fiji ships the mobie-io artifact required for this feature. This artifact is normally accessible in the MoBIE update-site, but because it also ships its own version of the mastodon artifact, these two update sites are incompatible in Fiji. You need to active one or the other, or none, but not the two sites in the same Fiji.

It works as follow. There is a new tab in the Mastodon launched called new project from URL.

It requires a URL that points to a ZARR file stored on a S3 storage server. For instance, the IDR hosts examples of such datasets on the embassy cloud of the EBI. This path needs to be entered in the Image URL field.

When you click the open button, Mastodon tries to retrieve basic image information from the URL and gives an error message when it does not succeed. Depending on the access rights in effect, you might be prompted for a password.

Mastodon will then create a lightweight BDV file, that simply points to the data stored remotely. This file needs to be saved on your computer, and the path to the target file must be entered in the second field.

This BDV file will simply point to the remote image data, and contains basic information about it. The image data will not be stored on your computer:

tinevez@METALLICA MINGW64 ~/Desktop
$ cat ImageFromIDR.xml
<?xml version="1.0" encoding="UTF-8"?>
<SpimData version="0.2">
  <BasePath type="relative">.</BasePath>
  <SequenceDescription>
    <ImageLoader format="bdv.ome.zarr.s3" version="0.2">
      <ServiceEndpoint>https://uk1s3.embassy.ebi.ac.uk</ServiceEndpoint>
      <SigningRegion>us-west-2</SigningRegion>
      <BucketName>idr</BucketName>
      <Key>zarr/v0.1/4495402.zarr</Key>
    </ImageLoader>

Once done, click the create button, wait for a bit, and voila: a new Mastodon project that points to a possibly very large image, stored remotely:

When you will save this project to a Mastodon file, it Mastodon will point to the BDV file you created, which in turns point to the remote storage.

The launcher has a list of recent projects.

The open Mastodon project tab shows a list of the recently opened Mastodon projects. Clicking on the red button removes an entry from the list. Double-clicking on the path opens the location of the file.

The list of recent projects is saved in a YAML file inside the ~/.mastodon/ folder:

tinevez@METALLICA MINGW64 ~/.mastodon
$ ls
colormodes.yaml          debug.yaml  **recentprojects.yaml**               trackschemestyles.yaml
colormodes-post-pr.yaml  keymaps/    rendersettings.yaml
datagraphstyles.yaml     Plugins/    selectioncreatorexpressions.yaml

$ cat recentprojects.yaml
C:\Users\tinevez\Development\Mastodon\mastodon\samples\Celegans.mastodon
--- C:\Users\tinevez\Google Drive\Mastodon\FromVlado\mette_e1.mastodon
--- C:\Users\tinevez\Google Drive\Cours\MastodonI2K\CTC_TRIF_trainingVideo02_jy-tracked.mastodon
--- C:\Users\tinevez\Development\Mastodon\mastodon\samples\drosophila_crop.mastodon

Open Mastodon projects by drag & drop on the Fiji toolbar.

Drag and drop a Mastodon project file on the Fiji toolbar to open it directly.

A grapher for feature values.

There is now a new view in Mastodon, fully fledged. It lets the user plot a feature as a function of another one in a XY plot. This view is great to do a first analysis of your tracking data, for instance to plot intensity over time over a large number of cells.

It works with the same facilities that any other views (shared focus, shared selection, shared highlight, views synchronization, etc). This makes it easy to navigate from a dot in the XY plot to a cell in the BDV view, or in a lineage view. And the other way around as well. The grapher view uses similar keybindings that of the TrackScheme views.

https://user-images.githubusercontent.com/3583203/154713360-88f9a107-799e-4041-a00c-a3057a54a5a8.mov

It also works with context. You can set the grapher to generate a plot on the selection, the tracks of the selected items, or to take the content from a BDV context:

https://user-images.githubusercontent.com/3583203/154713425-75704999-c54f-4213-bf8d-c236ea2a0034.mov

Feature-based coloring and tag-based coloring work like for any other views. The plot itself can also be configured in a new panel in the preferences:

https://user-images.githubusercontent.com/3583203/154713462-49724aa7-fc98-4ea5-9a0b-be8004e8600a.mov

The view is a standard Mastodon view, strongly inspired by TrackScheme. So it:

• shares selection, highlight and focus with other views.
• shares object coloring.
• has styles.
• has roughly the same navigation shortcuts (mouse navigation, zoom, etc. are made ike TrackScheme).
• has Context support (though it is hacky compared to what TrackScheme has).
• has some LOD (spot size changes depending on the proximity with other spots, like for TrackScheme).

Right now this must be considered a first version, to be perfected. Ideally we want:

• a better renderer with better LOD, that exploits a KD-tree to avoid painting all the spots if the user requests.
• to review the Context support.
• make exporters to pdf and co.
• add histograms.

Scripting Mastodon in the Fiji script editor.

Mastodon can now be scripted, for instance in Jython, in the Fiji script editor. For instance:

#@ Context context

from org.mastodon.mamut import Mamut
import os

# Creating a new project from a BDV image data.
mamut = Mamut.newProject( bdvFile, context )
logger = mamut.getLogger()
# Detect with the DoG detector, a radius of 6 and a threshold on quality of 200.
mamut.detect( 6., 200. )
# Link spots with the simple LAP tracker, with a max linking distance of 10.
mamut.link( 10., 0 )

An example script describes the implemented commands: https://github.com/mastodon-sc/mastodon-app/blob/master/scripts/MastodonExampleScript.py

Mastodon can open a project even if the image data is missing.

Sometimes the path to a BDV file is faulty and prevents opening the project. Now, Mastodon will simply warn you and open the project with a blank image. When in a rush, this helps sharing tracking data with colleagues without having to send the image.

Movies can be recorded from the BDV views.

There is now two new commands to the BDV views. They toggle two new dialogs that let the user configure and record movies from a BDV view. Default shortcuts: ctrl R & ctrl shift R

The first one is used to record movies from the display, as it is currently shown in the BDV. The movie will record the BDV window, keeping the coordinates constant and iterating over time, abiding to any display settings currently selected.

The second one does the same, but can build a MIP image over a sandwich of images above and below the displayed plane.

The two dialogs look like this:

The code is of course derived from the classes with the same name in the BDV core by Tobias. We adapted it so that:

• The track overlay is also recorded. And possibly projected in the second case.
• The colorbar overlay is also recorded.
• When building the MIP, it includes planes above and below the one currently displayed, not just below.
• It can export the results as a collection of PNG files, like in the original BDV version.
• If required, generate a MP4, MOV or AVI movie on the fly, using the io.humble library (https://github.com/artclarke/humble-video). This requires the Humble-Video update site to be activated in Fiji.

Auto-linking mode in BDV views.

When on (toggle with control+L), the next spot added will be automatically linked to the one in the selection, if there is exactly one in the selection.

MaMuT users seem to be overly attached to this feature.

Ask to save before closing Mastodon.

When closing the main window, this dialog is shown. But only if there have been changes since the last save.

Extract a Fiji time-lapse following a track.

A new plugin (Plugins > Track image) allows extracting the image data that follows a cell in a track. Select 2 spots or 1 spot, run the plugin, and a new Fiji image will open (after some time) showing the image data that follows the cell tracked over time.

https://user-images.githubusercontent.com/3583203/202156405-ac6a0361-ebf7-4100-8818-220e75c6faef.mp4

Other changes.

• Tiny fix, NullPointerException in FeatureComputationPanel/Model by @maarzt in https://github.com/mastodon-sc/mastodon/pull/147
• CI: switch from Travis CI to GitHub Actions by @imagejan in https://github.com/mastodon-sc/mastodon/pull/149
• Fix FeatureComputationModel:getFeatureSpec(...) if no features are installed by @tinevez in https://github.com/mastodon-sc/mastodon/pull/148
• Improve min radius when editing spot. by @tinevez in https://github.com/mastodon-sc/mastodon/pull/163
• Menu and title fixes by @xulman in https://github.com/mastodon-sc/mastodon/pull/185
• Fix LineageTreeLayout to work for graphs containing merging spots by @maarzt in https://github.com/mastodon-sc/mastodon/pull/188
• ADD: MastodonDndLauncher to accept DnD .mastodon events by @xulman in https://github.com/mastodon-sc/mastodon/pull/195
• FIX: catches (warns and continues operation!) when restoring of BDV windows from gui.xml by @xulman in https://github.com/mastodon-sc/mastodon/pull/194
• Change the "edit label action" to rename all spots of a branch at once (Branch&Hierarchy TrackScheme) by @maarzt in https://github.com/mastodon-sc/mastodon/pull/186

New members of the Mastodon governance.

Ko Sugawara, Mathias Arzt and Vladimir UIman joined the Mastodon governance.

This new beta version contains several additions initiated by Ko Sugawara @ksugar to facilitate the deployment of the Elephant project https://github.com/elephant-track.

The latest changes can be summarised as follow:

• Listeners can be registered to the WindowManager to be notified when a new BDV view is created by the user.
• Fix a deadlock in TrackScheme occurring when TrackScheme was launched while spots or links were created.
• New settings for the BDV view:
  • stroke of spots
  • stroke of links
  • paint ellipses filled.
• Settings page are in a scroll pane, when they become very long.

Mastodon beta-18.

This new beta of Mastodon focuses mainly on bringing some clarity in the development process and a rework of the intensity-based feature computation. If you are interested in the development of Mastodon, read at the end of this changelog for the changes in development guidelines that come with this release.

---

Changelog.

New intensity features and removed ones.

Tobias made a new and convenient iterator that can iterate through all the pixels of an ellipsoid exactly once, regardless of the transform or shape of the ellipsoid (https://github.com/mastodon-sc/mastodon/blob/master/src/main/java/org/mastodon/mamut/feature/EllipsoidIterable.java). It is used in this release to rewrite and change all the intensity-based features:

• The Spot intensity feature computes the mean, min, max, sum, median and std of the pixel values in a spot for each channel.
• The sum intensity and median intensity feature in the mastodon-pasteur artifact were removed (replaced by above feature).
• The Spot gaussian-filtered intensity feature is now renamed in Spot center intensity (it does that). It now only returns the mean value, not the std anymore.
• There is now a Spot quick mean feature, that computes the mean intensity on the highest resolution level available, in a multi-threaded fashion. It brings faster computation speed at the cost of accuracy. Because so far we tested computation on HDF5 images, for which in Java there is no multithreaded access, its benefit are yet limited.

Better look of the window and dialog icons when minimized.

No more warnings about plugin commands without a description.

All the plugin commands that are were mapped by default and that were are regifted in the bdv, ts or table contexts generated an error message everytime preferences were edited.

This also allows these commands to be edited in the Mastodon Keymap editor.

Finally, the ts context is now renamed to trackscheme.

[Bugfix - core] Don't complaint when pressing 'Save' with a newly created project.

This happened because just after creating a project on a H5/xml file, there is not yet a Mastodon project to save to. With this fix, pressing the Save button asks the user for a .mastodon file to save to.

[Bugfix - mastodon-tracking] Mastodon blocks during semi-automatic tracking.

Sometimes when a TrackScheme window was opened and when semi-automatic tracking was done in a BDV window, Mastodon could become blocked. This was caused by conflicting graph update sent by the semi-automatic tracker and TrackScheme trying to repaint the whole model every-time.

This release contains a tentative fix that does update notification in a more timely manner.

Noticed by Jang-Mi Kim (Institut Pasteur) & Arianne Bercowsky (EPFL).

[Bugfix - core] Feature update mechanism does not work properly.

Mastodon can do feature value updates. The values for some features are not recomputed for an object if this object and its neighbor objects did not change between computations. This help greatly in the case of large models for the intensity features, that take ages to compute.

Normally when the user modifies a spot, its feature values that need recomputing should be deleted (e.g. as they appear from the tables) to avoid misleading the user with faulty feature values after modification. A bug prevented the features to be deleted when a spot was edited. This is fixed now.

Noticed by Ko Sugawara (ENS-Lyon).

---

Changes in the development practices.

Prior to this beta, there were two Mastodon versions uploaded to Fiji. The first one - Mastodonpreview - was in sync with the master but did not have many features. It brought really the core of Mastodon but did not have for instance the table views.

The second one - Mastodon-jungle - had a lot of features, including the table views, the visibility modes, etc.... These features were not (still are not) reviewed by Tobias, which violates somehow our development guidelines. However we (Tobias and JY) reasoned that in the beta phase they should be brought to the community in an early manner. So a branch named jungle3 was made that merged all the PRs with the new features, and uploaded to Fiji. This is the version the users have had playing with since a couple of months.

As noted by others, this deployed version was not in sync with the master, which started to cause some confusions. So:

• We forked the original master branch to another one to save its state (master-before-switch) and rebased the feature PRs against this one. This way we can analyze them a posteriori when we have time.
• We merged the jungle3 on the master branch.

From now on, we will test the following development guidelines:

• The master branch will remains in sync with the version uploaded to Fiji. Except for minor things such as the version specified in the pom.xml.
• In between releases, all development will be made on the dev branch. So if you make a PR, please make it against the dev branch.
• For a release, the dev branch will be merged on the master, tested then uploaded to Fiji. The next development cycle will happen again on the dev branch.