Jakob Troidl, Johannes Knittel, Wanhua Li, Fangneng Zhan, Hanspeter Pfister*, Srinivas Turaga*

Global Neuron Shape Reasoning with

Point Affinity Transformers

Research Presentation

( *equal advising )

Dorkenwald et. al 2023 – Animation by Tyler Sloan

The Connectomics Pipeline

3

Proofreading Connectomes

Dorkenwald, S., Schneider-Mizell, C.M., Brittain, D., Halageri, A., Jordan, C., Kemnitz, N., Castro, M.A., Silversmith,

W., Maitin-Shephard, J., Troidl, J. et al. CAVE: Connectome Annotation Versioning Engine. Nature Methods 2024.

Neurons come in many shapes

Motifs are recurrent connectivity patterns of neurons in the brain.

​

MICrONS Consortium, Bae, J.A., Baptiste, M., Bishop, C.A., Bodor, A.L., Brittain, D., Buchanan, J., Bumbarger, D.J., Castro, M.A., Celii, B. and Cobos, E., 2021. Functional connectomics spanning multiple areas of mouse visual cortex.

Neurons come in many shapes

Motifs are recurrent connectivity patterns of neurons in the brain.

​

MICrONS Consortium, Bae, J.A., Baptiste, M., Bishop, C.A., Bodor, A.L., Brittain, D., Buchanan, J., Bumbarger, D.J., Castro, M.A., Celii, B. and Cobos, E., 2021. Functional connectomics spanning multiple areas of mouse visual cortex.

Manual / Human

Error Identification

​

Two neurons

incorrectly merged

Representing Morphology

Mesh

Skeleton

Point Cloud

Difficult to make automatic higher-level inferences about neuron morphologies

Model

Automated Error

Correction

Automated

Cell Typing

Learned Feature

Why Point Clouds ?

(+) Easy to sample; either subsampling segmentation or skeleton vertices

(+) Scalable 3D data representation that captures global morphology

(-) No explicit representation of topology

Imperfect CNN-based

volumetric segmentation

Multi-Neuron Point Cloud

Corrected Labels

Our

Approach

Sampling

But what about Split Errors ?

We assume strong super-voxel agglomeration biased towards merge errors

Image Credit: Dorkenwald, S., Schneider-Mizell, C.M., Brittain, D., Halageri, A., Jordan, C., Kemnitz, N., Castro, M.A., Silversmith, W., Maitin-Shephard, J., Troidl, J. and Pfister, H. et al., 2024. CAVE: Connectome annotation versioning engine. to appear in Nature Methods.

Our Approach

Point Affinity Transformer

(Multi-) Neuron

Point Cloud

Affinity guided Point Clustering

(Multi-) Neuron

Point Cloud

Point Distance Metric

Clustering Labels

D = 1.0 -

Agglomerative

Point Cloud Clustering

Affinity = Point Proximity Metric

Point Affinity Transformer

Qualitative Results

Quantitative Results

Neurons come in many shapes

16

Motifs are recurrent connectivity patterns of neurons in the brain.

​

Dorkenwald, S., Matsliah, A., Sterling, A.R., Schlegel, P., Yu, S.C., McKellar, C.E., Lin, A.,

Costa, M., Eichler, K., Yin, Y. and Silversmith, W., 2023. Neuronal wiring diagram of an adult brain.

LO20

GNGME1

IPSME2

LO1

Tedious Labeling Process

Neuron Type Classification

_🔒

Input: Single Neuron Point Cloud

Pretrained Point Affinity Transformer Encoder

Neuron Types: Qualitative Results

Neuron Types: Confusion Matrices

Hemibrain

FlyWire OL

Caveats & Next Steps

(-) Trained & tested on simulated reconstruction errors

Next Steps

Test on real reconstruction errors (e.g., map initial FlyWire segmentation to post proofreading)

(-) So far only trained & tested on Drosophila connectomes

Establish automatic proofreading benchmark datasets

Combine our approach with additional data such as SegCLR, synapses or neuron types

Neuron morphology & connectivity aware motif discovery

References

Troidl, J., Knittel, J., Li, W., Zhan, F., Pfister, H. and Turaga, S.C., 2024. Global Neuron Shape Reasoning with Point Affinity Transformers. Under Submission at CVPR, 2024.

​

Dorkenwald, S., Schneider-Mizell, C.M., Brittain, D., Halageri, A., Jordan, C., Kemnitz, N., Castro, M.A., Silversmith, W., Maitin-Shephard, J., Troidl, J. and Pfister, H., 2023. CAVE: Connectome annotation versioning engine. To aopear in Nature Methods.

​

MICrONS Consortium, Bae, J.A., Baptiste, M., Bishop, C.A., Bodor, A.L., Brittain, D., Buchanan, J., Bumbarger, D.J., Castro, M.A., Celii, B. and Cobos, E. et al. , 2021. Functional connectomics spanning multiple areas of mouse visual cortex.

​

Shi, W. and Rajkumar, R., 2020. Point-GNN: Graph neural network for 3d object detection in a point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1711-1719).

​

Zhao, H., Jiang, L., Jia, J., Torr, P.H. and Koltun, V., 2021. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 16259-16268).

​

Dorkenwald, S., Matsliah, A., Sterling, A.R., Schlegel, P., Yu, S.C., McKellar, C.E., Lin, A., Costa, M., Eichler, K., Yin, Y. and Silversmith, W., 2024. Neuronal wiring diagram of an adult brain. Nature, 634(8032), pp.124-138.

​

Takemura, S.Y., Hayworth, K.J., Huang, G.B., Januszewski, M., Lu, Z., Marin, E.C., Preibisch, S., Xu, C.S., Bogovic, J., Champion, A.S. and Cheong, H.S., 2023. A connectome of the male drosophila ventral nerve cord. BioRxiv, pp.2023-06.

​

Scheffer, L.K., Xu, C.S., Januszewski, M., Lu, Z., Takemura, S.Y., Hayworth, K.J., Huang, G.B., Shinomiya, K., Maitlin-Shepard, J., Berg, S. and Clements, J., 2020. A connectome and analysis of the adult Drosophila central brain. elife, 9, p.e57443.

​

Dorkenwald, S., Li, P.H., Januszewski, M., Berger, D.R., Maitin-Shepard, J., Bodor, A.L., et al., 2023. Multi-layered maps of neuropil with segmentation-guided contrastive learning. Nature Methods, 20(12), pp.2011-2020.

​

​

​

jakobtroidl.github.io

Thank You!

Website: vcg.seas.harvard.edu/publications/neuron-shape-reasoning

Image Credit: Amy Sterling @ FlyWire

Code: github.com/jakobtroidl/neuron-shape-reasoning

23

24

25

26

27