1 of 97

Feature Extractor

May 1st, 2025

1

5/21/2025

2 of 97

Data

  • Total samples has increased *object detection complete (green)
    • 17 PAN
      • PANd7kid1_left (5)
      • PANd7kid1_middle(5)
      • P1-1_kidney (6)**
    • 24 Control
      • Contkid1_left (5)
      • Contkid1_middle (6)**
      • C1-1a_kidney_v7 (6)**
      • C1-1a_kidney_v (6)**

​

​

​

2

5/21/2025

3 of 97

METHODS: Semi-automated Object Detection

  • Workflow
    1. Binary thresholding - smooth with median kernel and threshold
    2.  Calculated connected components in order
    3. Include/exclude based on compactness value and number of pixels
    4. Manual intervention of results to remove unwanted detections

​

​

3

5/21/2025

4 of 97

Semi-automated Object Detection

​

​

​

​

​

4

5/21/2025

  1. Binary thresholding
    • Smooth with median kernel to remove small pixel variations
    • Threshold to defined hyperparameter in configuration file

threshold

Microscope Image

Thresholded Image

Contkid1_left_results3

​

​

​

​

5 of 97

Semi-automated Object Detection

2. Connected components function (sitk package, example segments below)

  • Groups pixels into 'components' or groups of touching pixels

3. Groups as below based on number of pixels in each group (hyperparameter)

  • compactness factor (ratio of perimeter to area, telling us how 'round' a shape is)

​

​

​

5

5/21/2025

Contkid1_left_results3

​

​

​

​

6 of 97

Semi-automated Object Detection

4. Manual intervention of results to remove unwanted detections

​

​

​

​

6

5/21/2025

Remove

Add

Contkid1_left_results3

​

​

​

​

7 of 97

Semi-automated Object Detection

  • Note: lots of manual edits required when artifacts are present
    • Very sensitive to hyperparameters

​

7

5/21/2025

8 of 97

YOLO Object Detection

  • Pretrained image net model
  • Retraining to refine with semi-automated bounding boxes
  • No current successful results
    • Data too sparse
    • Will try again with more data after we have confirmed segmentations from clinicians

​

8

5/21/2025

9 of 97

Full image feature extraction

  • All new samples added with ability to save features
    • Summary of Results *Rerunning with all control samples now
      • Statistical difference in Pyfeats and HOG
      • SimCLR feature extraction
  • Try comparing features surrounding glomerulus and renal tubes as two separate groups

​

9

5/21/2025

10 of 97

Updates

  • Confirm object detection with clinicians *on-going
    • All Segmentations complete!! To upload
  • Yolo attempt with half of the samples * not results
    • Maybe augmentation but leaving for now.
  • Renal tube / Glomerulus separation code (see next slides)
    • 10 control and 11 disease (code test set)
    • Renal tube only using 2 segmentations (space)
  • Move to other computer *takes time
  • Mannwhitneyu

​

10

5/21/2025

11 of 97

GROUPS

  • Using segmentations from 11 PAN, 10 CONT
  • Resample regions to 2048 by 2048
  • Features Completed
    • HOG
    • SOBEL
    • CANNY

​

​

11

5/21/2025

12 of 97

HOG - GLOMERULUS

12

5/21/2025

13 of 97

HOG – RENAL TUBE

13

5/21/2025

14 of 97

SOBEL - GLOMERULUS

14

5/21/2025

15 of 97

SOBEL - GLOMERULUS

15

5/21/2025

16 of 97

SOBEL - RENAL

16

5/21/2025

17 of 97

SOBEL - RENAL

17

5/21/2025

18 of 97

CANNY - GLOMERULUS

18

5/21/2025

19 of 97

SOBEL - RENAL

19

5/21/2025

20 of 97

Questions

  • Resample regions to 2048 by 2048
    • Maybe the above is causing the sampling to not make sense?
    • If we take the image and resample then normalize.. Should I not normalize?
    • To do Pyfeats and SimCLR I need to recalculate for each sample. Doing this next!
      • Loop in each image and calculate features for each section?
      • Should I downsample?

​

20

5/21/2025

21 of 97

To Do

  • Discuss – plan for 3 weeks
    • Run on all samples?
    • Report?
    • Document code?

21

5/21/2025

22 of 97

Statistics

  • Mann-Whitney U Test
    • Tests weather distributions differ between groups
    • Non parametric (no distribution assumptiosn)
    • Robust to outliers

22

5/21/2025

23 of 97

23

5/21/2025

Image Semi-Automated Results

Examples

24 of 97

24

5/21/2025

PAN GROUP

25 of 97

PANd7kid1-left�_result1

25

5/21/2025

26 of 97

PANd7kid1-left�_result2

26

5/21/2025

27 of 97

PANd7kid1-left�_result3

27

5/21/2025

28 of 97

PANd7kid1-left�_result4

28

5/21/2025

29 of 97

PANd7kid1-left�_result4

29

5/21/2025

30 of 97

PANd7kid1-left�_result4

30

5/21/2025

31 of 97

31

5/21/2025

CONTROL GROUP

32 of 97

Contkid1-left�_result1

32

5/21/2025

33 of 97

Contkid1-left�_result2

33

5/21/2025

34 of 97

Contkid1-left�_result3

34

5/21/2025

35 of 97

Contkid1-left�_result4

35

5/21/2025

36 of 97

Contkid1-left�_result4

36

5/21/2025

37 of 97

Clinical Details – from Professor Matsui

  • Glomerular boundary is relatively clear and easy to identify. Renal tube appearance varies depending on how they're sectioned. Tubules in simple cross-section are easier to annotate, while tangential sections are more difficult.
  • There are at least three types: proximal tubules, distal tubules, and cortical collecting ducts. In stained images, proximal tubules are relatively easy to distinguish. Proximal tubules are overwhelmingly more numerous.
  • "Proximal" refers to proximal tubules, while "Distal" likely refers to distal tubules or cortical collecting ducts.
    • If you're evaluating mitochondria, its recommended to distingish between proximal and other tubules.
  • In clinical settings, when diagnosing diseases, doctors primarily examine glomeruli, while tubules are assessed more broadly at low magnification.
  • Glomeruli are crucial for disease type differentiation, while tubular regions provide important information about kidney function.
  • The samples you're observing represent a disease model that cannot be distinguished from normal using conventional light microscopy.
  • Doctors do not make decisions based on renal tube proximity to glomeruli.

37

5/21/2025

38 of 97

Adding Clinical Annotations

  • Loop and calculate each and save crops!
  • Incorporating Clinical Annotations
    • Glo, Proximial, Distal
  • Features:
    • HOG
    • SIMCLR and Pyfeats

38

5/21/2025

39 of 97

Experiment Updates

May 23, 2025

39

5/21/2025

40 of 97

Recommendations from Prof.Bise-san

1. Patch-based analysis� 1-1. Filter bank (e.g., Gabor filters)�•Select representative (distinctive) patches (around 10 samples)�→ Use PCA or similar methods to check whether the features can be separated�  *Compare regions where the double lines are clearly visible in the control vs. diseased regions�  *Example patch size: For glomeruli, use a rectangular patch roughly the size of the minor axis of a tubular epithelium ellipse

 1-2. Use methods such as:�◦VAE (supervised)

◦SimCLR�→ Apply PCA → Use linear SVM or similar classifier

 2. Pixel-wise filter bank analysis�•For example, to investigate differences in mitochondria between the inner and outer regions of renal tubules:�•Extract pixel groups from the inner region and the outer  region of the ellipse (the renal tubule)

→ Perform clustering for each group using a Bag-of-Features-like approach

40

5/21/2025

41 of 97

Recommendations from Prof.Bise-san

1. Patch-based analysis� 1-1. Filter bank (e.g., Gabor filters)�•Select representative (distinctive) patches (around 10 samples)�→ Use PCA or similar methods to check whether the features can be separated�  *Compare regions where the double lines are clearly visible in the control vs. diseased regions�  *Example patch size: For glomeruli, use a rectangular patch roughly the size of the minor axis of a tubular epithelium ellipse

 1-2. Use methods such as:�◦VAE (supervised) ** looking for pretrained model don’t really exist easily - what about monai zoo pathology https://github.com/Project-MONAI/model-zoo/tree/dev/models/pathology_nuclei_classification �◦SimCLR ** tried resent 50�→ Apply PCA → Use linear SVM or similar classifier

 2. Pixel-wise filter bank analysis�•For example, to investigate differences in mitochondria between the inner and outer regions of renal tubules:�•Extract pixel groups from the inner region and the outer  region of the ellipse (the renal tubule)

→ Perform clustering for each group using a Bag-of-Features-like approach

41

5/21/2025

Completed / TO DO

42 of 97

Datasets

  • Clinician Annotation Regions
    • Split into Glomerulus, Proximal and Distal regions
    • 5 Control/5 Pan
  • ROI (region of interest)
    • Segmentations done for Glomerulus edges, inner and outer proximal tube regions
    • Same 5 Control/5 Pan ~ 5 per group

42

5/21/2025

43 of 97

Patch wise small analysis

  • 10 samples
    • 5 control / 5 Pan (~5 –10 per image per group *if they clearly exist)
      • Outer
      • Inner
      • Glomerulus

​

​

43

5/21/2025

Glomerulus​

Inner

      • Outer​

NOTE* visuals look different sizes but they are set to the same size this was just to illustrate locations​

44 of 97

Experiments

  • Clinician Annotation Regions and ROI (region of interest)
    • SimCLR (VGG and Resnet50)
    • Gabor (0.1 and 0.2 frequency) – only reports for ROI *running next
    • HOG – only reports for ROI
    • Pyfeats *running now
  • Other added capabilities
    • Combining channels 1-2-3 for PCA/Histogram analysis
    • Gabor feature extraction
    • Visualizations of histograms (zooming in/new overlays)

​

44

5/21/2025

45 of 97

RESULTS

1. Clinician Annotation Regions

A) SimCLR (Resnet50) - PCA visual grouping * Glomerulus only *

​

2. ROI (region of interest)

A) HOG – region means are different in Histograms

45

5/21/2025

46 of 97

SimCLR Resnet Features - GLOMERULUS

​

​

46

5/21/2025

Channel 5: Combine channels 1,2,3 *scattering components

Visual Separation in Channel 4 and 5.

Very interesting alone 1-2-3 could not tell, but together they can !

Clinician Annotation Regions​

  1. A)

47 of 97

HOG Features - GLOMERULUS

47

5/21/2025

ROI Annotation Regions​

2. A)

48 of 97

HOG Features - INNER

48

5/21/2025

Note differences

ROI Annotation Regions​

2. A)

49 of 97

HOG Features - OUTER

49

5/21/2025

Note differences

ROI Annotation Regions​

2. A)

50 of 97

Conclusions and Next steps

  • See next slides for details of all experiments and all data
  • Write up reports / read me for future student
  • ROI may not be the best after reshaping
    • Could redo?
  • Discuss evaluation of Gabor features
    • Is there a better way to compare due to the distribution?
  • Anything else to add :)

50

5/21/2025

51 of 97

Clinician Annotations EXPERIMENTS

ROI – small cut sections of glomerulus/proximal tubes/distal tubes

51

5/21/2025

52 of 97

HOG Features

  • Resampling crops to 256 by 256
    • Original image is 2048 by 2048 then cropped and resampled to 256 by 256
  • Parameters
  • #HOG Specific information see documentation.
    • orientations: 9
    • pixels_per_cell: 8 #assumes cubic shape
    • cells_per_block: 4 #assumes cubic shape
    • hog_intensity_rescale: 1 #0.1 #for visuals, rescale
  • Samples:
    • 5 control done by Doctor Matsui
    • 5 PAN done by semi-automated method
      • *redone for similar segmentations as test batch�

​

​

52

5/21/2025

53 of 97

SimCLR Features

  • 2048x2048
  • Ran for
    • VGG19 encoder
    • Resnet50 encoder

​

​

53

5/21/2025

54 of 97

SimCLR Vgg19 Features - GLOMERULUS

​

​

54

5/21/2025

Channel 5: Combine channels 1,2,3 *scattering components

55 of 97

SimCLR Vgg19 Features - DISTAL

55

5/21/2025

56 of 97

SimCLR Vgg19 Features - PROXMIAL

56

5/21/2025

Channel 5: Not added for this experiment

57 of 97

SimCLR Resnet Features - GLOMERULUS

​

​

57

5/21/2025

Channel 5: Combine channels 1,2,3 *scattering components

58 of 97

SimCLR Resnet  Features - PROXMIAL

58

5/21/2025

59 of 97

SimCLR Resnet Features - DISTAL

Not done yet

59

5/21/2025

60 of 97

Gabor Features - GLOMERULUS

To add

60

5/21/2025

61 of 97

Gabor Features - PROXMIAL

To add

61

5/21/2025

62 of 97

Gabor Features - DISTAL

To add

62

5/21/2025

63 of 97

ROI EXPERIMENTS

ROI – small cut sections of glomerulus/inner/outer sections

63

5/21/2025

64 of 97

Patch wise small analysis

  • 10 samples
    • 5 control / 5 Pan (~5 –10 per image per group *if they clearly exist)
      • Outer
      • Inner
      • Glomerulus

​

​

64

5/21/2025

Glomerulus​

Inner

      • Outer​

NOTE* visuals look different sizes but they are set to the same size this was just to illustrate locations​

65 of 97

SimCLR Resnet Features - GLOMERULUS

Reshaped to 256 by 256

** normalizing images before calculating features

65

5/21/2025

66 of 97

SimCLR Resnet Features - OUTER

66

5/21/2025

67 of 97

SimCLR Resnet Features - INNER

67

5/21/2025

68 of 97

HOG Features - GLOMERULUS

NOT normalizing

68

5/21/2025

Contkid1_leftkidney_2_featclassGlo_bb_annotation_2

69 of 97

HOG Features - GLOMERULUS

​

69

5/21/2025

70 of 97

HOG Features - GLOMERULUS

​

70

5/21/2025

Channel 1+2+3

**Mean values are .1 different per group

*Same graph different plots

71 of 97

HOG Features - GLOMERULUS

​

71

5/21/2025

72 of 97

HOG Features - OUTER

  • Added bins to 300 vs 150 for visualization
  • Resample to 256 by 256 patch size
  • Allows HOG orientations/cells

HOG Specific information see documentation.

orientations: 9

pixels_per_cell: 16 #assumes cubic shape

cells_per_block: 8 #assumes cubic shape

hog_intensity_rescale: 1 #0.1 #for visuals, rescale

​

​

72

5/21/2025

30 by 30

256 by 256

73 of 97

HOG Features - OUTER

73

5/21/2025

Channel 1+2+3

*Same graph different plots

74 of 97

HOG Features - OUTER

74

5/21/2025

Note differences

75 of 97

HOG Features - INNER

75

5/21/2025

76 of 97

HOG Features - INNER

76

5/21/2025

Channel 1+2+3

*Same graph different plots

77 of 97

HOG Features - INNER

77

5/21/2025

Note differences

78 of 97

Gabor Features

78

5/21/2025

​

​

​

  • Linear filtered defined by sine/cosine functions

​

  • Filter is seen at the left, different frequencies/thetas can be applied.

​

  • Properties
    • Sensitive to edges
    • Scale sensitive
    • Preserves spatial information

​

79 of 97

Gabor Features - GLOMERULUS

79

5/21/2025

Contkid1_leftkidney_3_bb_2

PANd7kid1_middlekidney_3_bb_4

Maybe we could also improve ROIS?

256 by 256 patches, 0.1 frequency

80 of 97

Gabor Features - GLOMERULUS

80

5/21/2025

Contkid1_leftkidney_3_bb_2

PANd7kid1_middlekidney_3_bb_4

256 by 256 patches, 0.1 frequency

81 of 97

Gabor Features - GLOMERULUS

81

5/21/2025

Contkid1_leftkidney_3_bb_2

PANd7kid1_middlekidney_3_bb_4

256 by 256 patches, 0.1 frequency

82 of 97

Gabor Features - GLOMERULUS

82

5/21/2025

How should we better compare these?

256 by 256 patches, 0.1 frequency

83 of 97

Gabor Features - GLOMERULUS

83

5/21/2025

Zoom in last slide, also blue is shaded red is outline

256 by 256 patches, 0.1 frequency

84 of 97

Gabor Features - GLOMERULUS

84

5/21/2025

Maybe some outliers?

256 by 256 patches, 0.1 frequency

85 of 97

Gabor Features - OUTER

85

5/21/2025

256 by 256 patches, 0.1 frequency

20250220_Contkid1_leftkidney 5_bb_2

20250223_PANd7kid1_middlekidney_1_bb_4

86 of 97

Gabor Features - OUTER

86

5/21/2025

256 by 256 patches, 0.1 frequency

87 of 97

Gabor Features - OUTER

87

5/21/2025

256 by 256 patches, 0.1 frequency

88 of 97

Gabor Features - INNER

88

5/21/2025

256 by 256 patches, 0.1 frequency

PANd7kid1_middlekidney_2_1patch

Contkid1_leftkidney_4_0patch

89 of 97

Gabor Features - INNER

89

5/21/2025

256 by 256 patches, 0.1 frequency

PANd7kid1_middlekidney_2_1patch

Contkid1_leftkidney_4_0patch

90 of 97

Gabor Features - INNER

90

5/21/2025

256 by 256 patches, 0.1 frequency

91 of 97

Gabor Features - INNER

91

5/21/2025

256 by 256 patches, 0.1 frequency

92 of 97

Gabor Features - INNER

92

5/21/2025

256 by 256 patches, 0.1 frequency

93 of 97

Gabor Features

93

5/21/2025

  • Experiments done with 0.2 frequency

​

  • No real differences in output metrics

Example Contkid1_leftkidney_3_bb_2

0.2 frequency​

0.1 frequency​

94 of 97

Pyfeat - GLOMERULUS

94

5/21/2025

95 of 97

Pyfeat  - OUTER

95

5/21/2025

96 of 97

Pyfeat  - INNER

96

5/21/2025

97 of 97

SUMMARY May 23 Report

1. Patch-based analysis� 1-1. Filter bank (e.g., Gabor filters)�•Select representative (distinctive) patches (around 10 samples)�→ Use PCA or similar methods to check whether the features can be separated�  *Compare regions where the double lines are clearly visible in the control vs. diseased regions�  *Example patch size: For glomeruli, use a rectangular patch roughly the size of the minor axis of a tubular epithelium ellipse

 1-2. Use methods such as:�◦VAE (supervised) ** looking for pretrained model – more likely what about monai zoo pathology https://github.com/Project-MONAI/model-zoo/tree/dev/models/pathology_nuclei_classification �◦SimCLR ** tried resent 50�→ Apply PCA → Use linear SVM or similar classifier

 2. Pixel-wise filter bank analysis�•For example, to investigate differences in mitochondria between the inner and outer regions of renal tubules:�•Extract pixel groups from the inner region and the outer  region of the ellipse (the renal tubule)→ Perform clustering for each group using a Bag-of-Features-like approach

97

5/21/2025