Modern Microscopy Control System-Design aimed towards Laser-Scanning-Microscopy and Widefield Whole-Slide-Scanning

Summary

Goals

• Distributed Computing for Acquisition Engines
• Simple and Custom GUI extensions for LSM in Panels and Napari
• Create documentation!

Project Outputs

Priya Updates

CustomDashboards_forMM2

Aarush Updates

Aarush Lab Notebook

Resources

Supplementary information 

Github repositories

• https://github.com/uw-loci/smart-wsi-scanner

Pending

• Better Metadata for OSc
• FLIM tools integration
• SPT tools (urgent -Jenu Nov 23, 2023 )
• Paquo: abstract read/write annotations into a project with image
• Ashlar for stitching RGB as 3 stacks

Aimed TaskList

• SLIM Atlas setup + SLIM Chamber Imaging setup
• CAMM/PPM Live “recording” and display in QuPath of image tiles
• Computations and Performance Evaluations for PPM-QuPath pipeline

Current

• Restructure smartpath library to “modular-format”
• Use something other than MicroManager for the GUI

• Jupyter-Panels
• Napari

• Processing on a remote GPU server

• Downsample and show cancer prediction or similar pixel classifier overlay.
• Resolution enhancement
• Denoising  and Compression (SPR grant- ask EFTY or Jun)

Updates

Sep 19, 2023Update Aarush starting

Sep 22, 2023Update (meeting with Mark, Jenu and Aarush:

• Install PyMMCore and get basics running => Create pixel size
• Get stitching running, call to QuPath command line
• Notebook exists for essential acquisitions
• Convert to PyMMCore+, look into Talley’s code
• Chains of commands, combine autofocus white balance, etc.

Sep 26, 2023Update (Napari-MM2) working on CAMM :

V3: posted : https://forum.image.sc/t/questions-about-using-pymmcore-plus/86880 

Sep 29, 2023 Mark T’s talk on MMCoreJ- pycromanager and pymmcore

Oct 5, 2023: pymmcore-plus works smoothly on CAMM : *working folder= mm2 folder for dl recognition

Oct 13, 2023: Im-switch discussion ImSwitch - requesting to join as community partner - Websites - Image.sc Forum

Oct 18, 2023: found all the weights*.pth files for denoising and SR testing

Oct 23, 2023: Look at autofocus algorithms for computer-vision aspect

Oct 28, 2023: MIT-Berkely published denoising metric https://arxiv.org/pdf/2310.16102.pdf 

Oct 31, 2023:  Qupath- paquo testing on CAMM worked

• https://forum.image.sc/t/annotations-not-saving-when-loading-through-paquo/67412/2
• https://forum.image.sc/t/paquo-read-write-qupath-projects-from-python/41892/14
• https://forum.image.sc/t/ome-types-rewritten-requesting-alpha-testers/83174

Oct 31, 2023: Script to test for startup on qu-path, :: startup scripts were removed from 0.5.0 so that does lock us to 0.4.4, which should be fine for what we are doing

•  https://forum.image.sc/t/memory-efficiency-when-creating-margin-annotations/86329/11

Nov 1, 2023 : GPU access granted? Need to setup vnc on PPM scope and CAMM

Nov 4, 2023: Meet with Kevin: Kevin wants to compare pancreatic cancer vs pancreatitis

Nov 6, 2023: Yuming found all Biomax slides are in use by Adib, but it's not anywhere to be found in the lab. Mike has an online spreadsheet for the pancreas database.

 Nov 7, 2023 :

• global slide registrations could enable super-modal annotated imaging at fastest scale possible?  

• Looking into slide scanners: SL160 Slide Loader - Prior Scientific ($28,190) or Products – Ludl Electronic Products, Ltd. or Automated Microscope - Zaber 
  Universal robots https://www.universal-robots.com/products/ur3-robot/ 20k

Nov 13, 2023:  

• GPU server connection working for most purposes
• Moving to a two-parent project:

Nov 14, 2023: Discussion with Mark:  narrower-scope and paper-projects

Nov 17, 2023:Ashlar stitching issues and Moving towards qupath stitching (for data upto 27GB RGB).

Nov 22, 2023: QuPath stitching seems to be working for RGB images. Python scripts written to poll or watch folders for file changes. https://github.com/uw-loci/PaQuO_test “polling_folder.py” and “watching_folder.py”. For our uses, assuming the additional dependency isn’t a huge issue, the watching_folder is probably the better option.

Nov 27, 2023: QuPath extension creation information

Forum post discussing https://forum.image.sc/t/debugging-qupath-extension-with-intellij/43899/7?u=mike_nelson

Github repo for extension https://github.com/qupath/qupath-extension-template

Building qupath https://qupath.readthedocs.io/en/0.4/docs/reference/building.html

Nov 30, 2023

Update - working plugin for stitching, but in QuPath 0.4.4.

Requires work -

• Increase stitching options with dropdown to support text files and basic Vectra
• Update for 0.5.0

Next goal, plugin to handle slide scanning -

Outline -

• Dialog 1

• slide box, 2 XY positions.
• Collect overview and some sort of indication of running/waiting.

• Support arbitrary python script
• Include settings section in Preferences for virtual environment location and other defaults

• Dialog 2

• Stitching then needs to take place (calling the stitching plugin).

• User selects regions of interest or runs automated region detection of some sort
• Dialog 3

• Collect regions in 20x brightfield

Dec 5, 2023 Update - working starter plugin

https://github.com/uw-loci/qp_scope

Takes input and outputs a text file proving the input is being read correctly.

Can be massively updated given a particular project structure to do things like maintain subfolders for tiling, pass those subfolders, get back commands, call tiling, etc. I will probably need help with some of those things though.

Dec 4, 2023

Kevin asking questions about direction of biological question: Differentiation of pancreatic ductal adenocarcinoma and chronic pancreatitis using graph neural networks on histopathology and collagen fiber features

Dec 23, 2023

Tested out the new var-shared extension GUI. First scan creates a slide-folder, a label-folder and a qupath-project inside the lable folder. The scantype+id will be generated for each scan, eg: slide\panc360\4x_BF_01

The additional tkinter and path dependency has been cleaned up on python side.

Pending: getting the annotations handled on the python side.

Jan 8, 2024

Goals:

1. Figure out pixel size created by images on the slide scanner, read the file name to figure out if they are 3600 or 7200, the only options.
2. Establish the workflow to use the top coordinates and the diagonal length to send a bounding box to the python script
3. Fix the send to the python script to send the csv file location.

Jan 10, 2024

• Committed changes on the wsi-scanner adding Mike’s qupath jars
• Cleaning up workflow and definitions for Pete’s meeting on 11th 8AM

Feb 8, 2024

• Have working QuPath + microscope for collection, but currently relies on many hardcoded shortcuts
• Currently working on -

• Preferences have been added, but working on accessing them
• Everything runs in 0.5.0(0.6.0 ?)
• Jenu working on schema to maintain the information about collection and the status of the overall project/collection

• Installation and setup for IntelliJ IDE for extension development: https://docs.google.com/document/d/1jkFaWcuS-XVgeKAmY_xe6rhTBICBGm1elYQFLrrydV4/edit

Feb 21, 2024

Useful forum posts related to getting grid/collection stitching working within QuPath: https://forum.image.sc/t/building-a-qupath-extension-that-relies-on-an-imagej-plugin/92396/3

https://forum.image.sc/t/plugin-or-class-not-found-stitching-grid/92397

Mar 25, 2024

Updates - Currently software is working on the CAMM for basic 4x collection and stitching, though the stitching is simply adjacent tiles, not resolving overlaps.

TODO:

• Rework basic-stitching to use Preferences and avoid Dialogs for 0.6.0 compatibility
• Update the bounding box and second modality functions to work with updated macro image functionality.
• Integrate the JSON/YAML file usage and writing as opposed to using Preferences for everything.
• Figure out how to add additional modalities beyond two?

May 15, 2024

Initial data selection for trial run of PPM with PathScope 2.0

PDAC and CP slides

Data PS low res scans whole slide scan overviews

Aim 1 - Two dimensional PPM imaging of PDAC H&E sections

Completed an early prototype of the QuPath-PycroManager-MicroManager pipeline and collected data on the brightfield/SHG multimodal system. Currently implementing the system on the PPM. Initially, our goal will be to image 5 slides each from patients with either PDAC or CP - the slides themselves have been acquired through Agnes Loeffler MD Ph.D, from Cleveland MetroHealth. Initial observations indicate that at least 3 whole slide images per modality will be required. H&E at full transmission (brightfield, BF), and two different scans at equal but opposite polarity offsets - which may allow for image subtraction and the identification of collagen against the H&E background. Using QuPath imageServers and ImageOps, we may be able to use the 6 input channels to either create a pixel classifier based on all 6 channels, or resave a new single channel image using averaging and subtraction of the two RGB PPM images - as the collagen pixels should have different values, while the non-collagen, or normal H&E pixels, should be roughly the same between imaging runs.

This aim will be achieved when the software pipeline can

1. Be implemented on new systems (as demonstrated by a third system) without major restructuring of the code
2. Be able to instruct a new user how to perform basic use within one to two hours.
3. Create a pixel classifier to detect the presence of collagen within a combination of the three brightfield WSI.
4. Collected sufficient data to perform an analysis of PDAC and chronic pancreatitis, such that

1. A pixel classifier can detect the difference between the two diseases with greater than 90% accuracy.
2. An object classifier using tabular data from detected collagen can distinguish the two disease states in the test cases - to be validated later.

5. Compare acquisition time for SHG across the same area to determine how effective the ability to shift the software to different modalities for various projects is.

May 18, 2024

Beginning tests - collected sample data at 10, 0, -10 degrees on the instrument (expected to be 20, 0 and -20 actually due to the entire rotation only being 180 degrees).

Average background at initial settings:

10 - 150, 145, 66

0 -  21, 20, 17

-10 - 82, 96, 40

Multiplying the -10 deg image by 1.5 then subtracting the 10 and -10 images unfortunately does not effectively remove the background to the point where the collagen is inherently obvious.

The 0 degree might be best currently given the color contrast, but with the H&E stain, especially the eosin, certain colors are going to be lost.

Attempt number 2 - color correct using the background for the two 10 degree offsets, then intensity balance,

https://github.com/pmascalchi/ImageJ_Auto-white-balance-correction

then use Image Math-Difference.

Intensity balance is 165/122 = 1.352

With some contrast adjustment, this shows more promise.

As a brute force method for collagen detection this may work, but it also means that the information should be there to use a 6 channel image to perform pixel classification. The issue there being that a lot of training data will be needed due to the large variety of color space that could represent collagen, especially since some of it will overlap with eosin.

Look into Bin’s code on PPM https://github.com/uw-loci/polychromatic-polarization/blob/master/ppm-process.py 

How it works

Shribak’s cover letter from 2022:

https://www.mbl.edu/sites/default/files/2022-01/PPM_2017-Shribak.pdf 

Test stage stability between +10 -10 acquisitions

Minor shading akin to DIC image, but not too far off

Jun 6, 2024

Discussion of json files

Objective needs to be within imaging mode

Offsets maybe need to be QuPath specific - should there be a QuPath section (based on camera frame size?)

Need to include camera frame size. Option for multiple detectors/cameras?

Include measurements in all names eg framewidth_px, stagesize_um, offset_px

Jul 22, 2024

Progress!

Still need to update to actually use the text file information, and provide useful errors when the text files are not present.

Preference should point to the text file(s)!?

Aug 15, 2024 Reviewing BX53 PPM system

PPM Olympus BX53 - modified

Adjusted Startup and ColorOn Groups/Presets, ColorOn should now perform white balance.

Oct 8, 2024

Goals - get PPM setup, but without the rotational stage (requires driver from Mark T.)

Code: swap over to using .json files

Rotational stage discussion

First stage PRM1Z8:

Recommended by Michael Shribak-

In light path, too large to fit under condenser

Second stage PDR1C + KIM001 -

https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=13798

PDR1C

https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=14576

Issues:

PDR1C (compact) fits nicely and does not overbalance the dial.

However the step size is inconsistent and there is no feedback loop to control the actual movement.

Attempting to move the stage between +5 and -5 degrees took either 3 or 4 “steps” of “250”, and the error was significant, shifting 1-2 degrees each “there and back”.

Discussion from Jenu:
Jenu Chacko: The PRM1Z8 is a DC servo rotation mount at 1° increments.

Jenu Chacko: If price is driving this decision, we could get a useful stepper motor with simple sensor for $450 controlled over serial port: https://optics-focus.com/motorized-rotary-stage-p-1094.html?zenid=tpnsqachdbkoi192glpurung47

Jenu Chacko: or the picard version for $985 https://picardindustries.com/products/optical-devices/usb-rotator-system/

Jenu Chacko: if you want performance with piezo, maybe the PI model: https://www.pi-usa.us/en/products/motorized-rotary-stages-goniometers/miniature-rotary-stages/u-651-rotation-stage-with-low-profile-design-703071#variants

Jenu Chacko: Newport has a ($970 + $2517) Piezo model for optics https://www.newport.com/f/agilis-series-piezo-rotation-mount

Email heather about returning stage

Feb 7, 2025

New stage arrived, adapter created by George Petry

Errors turned out to be due to friction, eventually will be fixed through iterations on the rotational motor and PPM.

Mar 25, 2025

Many iterations later, the rotational stage is connected to a motor and running!

There is some slippage, about 2 degrees, when going counter clockwise vs clockwise.

Reinstalling entire setup on my home computer after a critical OS failure due to a bad RAM stick, starting using qp-scope for development in 0.5.0

Well, there is already an issue, as things have changed for 0.6.0!

qp-scope for development in 0.6.0

Brand new area to store layouts for GUIs and string information. This will make things so much easier* maybe.

TODO

• Understand new QuPath extension layout
• Ask George for any description/layout of connection.
• Figure out how to handle micromanager rotational stage control from Pycromanager
• Deal with 2 degree error when rotating clockwise, possibly by overcorrecting or playing with backlash settings in Kinesis
• Add new settings/inputs for degrees for collection. Needs to be separate, yet integrated. Some kind of “special device” optional call, possibly to a different extension? Discuss how to implement. Maybe a list of device calls for any given microscope acquisition (stage, camera, ?rotational stage ?objective change ?filter change)
• Ask Alan about whether Groovy will continue to be supported or if I should switch everything over to Java

Apr 16, 2025

Calibration tests and data acquisition

PPMCalibration2025

Includes links to data