Bone Suppression Related Paper Highlights
Reporter: Benny Tseung-ho Chan Yifei Sun
Institute: Hangzhou Dianzi University
计算机视觉CS131：专题3-图像梯度、边缘检测（sobel、canny等）、Patch与Resize的区别
patch可以通俗地理解为图像块，当需要处理的图像分辨率太大而资源受限(比如显存、算力等)时，就可以将图像划分成一个个小块，这些小的图像块就是patch。
为何要划分patch而不使用resize缩小分辨率呢？
通常情况下，resize没有太大问题。但在处理图像分割问题时，由于是dense prediction，属于像素级的预测，因此会尽量要求精确。而resize操作大多是对图像进行插值处理，本质上一种滤波，在像素级别上会造成损失(对传统图像处理有了解的应该知道某些滤波效果会使图像变得模糊)，即：某些位置上的像素值是通过多个位置加权计算出来的，从而限制了模型预测结果的上限。相对地，划分patch只是把原来的大图分成一个个小图，而这些小图依然是原图的部分，像素值没有改动，因而在理论上，训练出来模型的上限能够比基于resize得到的图像训练来的高。
类似于图像的频率：灰度值变化剧烈程度的指标，是灰度在平面空间上的梯度。
通常我们称CT图像为DICOM(Digital Imaging and Communications in Medicine)图像，但是DICOM不是一个图像或者文件格式，它是包含了传输、存储、检索、打印、处理和显示医学图像信息方面的国际标准(ISO 12052)。
CT图像中的窗宽和窗位
窗位:60，窗宽:400时CT图像的显示效果。由图可以清晰地看出腹部器官肝脏(HU值在50到70之间)的纹理信息，而肺部的细节基本没有展现
窗位:-450，窗宽:1500时CT图像的显示效果，此时可以清楚地看出肺部的细节信息，而腹部器官肝脏的纹理信息将不再清晰
CRNet: Unsupervised Color Retention Network for Blind Motion Deblurring
Blind Image Decomposition (BID), which is to uniformly remove multiple types of degradation at once without foreknowing the noise type.
Towards performant and reliable under-sampled MR reconstruction via diffusion model sampling
Accelerate Sampling: From coarse to fine
重采样分为上采样和下采样，区分的依据是重新采样时新采样率与原先采样率的大小的比较。上采样和下采样都是针对信号最高频率而言。
上采样：采样频率高于信号最高频率的 2 倍，又称为插值/内插；
下采样：采样频率低于信号最高频率的 2 倍，又称为抽取。
重采样的方法：主要是最近邻法、双线性内插法以及三次卷积内插法。
欠采样与过采样主要是针对不平衡数据集。
欠采样：当数据集不平衡时，选择减少数据量过多类别的数量，例如对于一个只用0和1的二分类问题，样本标签1有10000个数据，样本标签0有6000个数据时，为了保持样本数目的平衡，可以选择减少标签1的数据量。过采样：就是对数量较少的类别，扩大它的数量，通过数据增强的方法增加它的数目。
基于 NVIDIA 的 PC 端到端人工智能：从 FP32 过渡到 FP16 优化人工智能https://zhuanlan.zhihu.com/p/638112323
Self-supervised
All-in-one VS CP
为了解决上述问题，我们结合 Masked AutoEncoder (MAE) 提出了一种高效且简单的预训练模式：Context-aware Pretraining (CP)，包含两个 pretext 任务：混合图像分离和损失图形重建。设图像的恢复过程遵循从结构到纹理的模式 (structure to texture)，我们的思路很简单，首先利用两个 pretext 任务在预训练阶段重建出结构信息，随后在 fine-tuning 阶段即可快速地基于结构进行纹理填充。
Structure与Mask的一一对应关系
Information Fusion Module
It can be observed
that our pretrained model can produce a restored structure. Meanwhile, the appearance flow can not only find the noise location but also indicate the sampling direction of textural features. Features containing vivid textures can “flow” to
the corrupted regions.
ShadowFormer: Global Context Helps Image Shadow Removal
基于Retinex理论的去阴影模型
引入了一种基于Retinex的阴影模型，提出了一种新的基于多尺度通道注意力框架的阴影去除Transformer(ShadowFormer)。
本文考虑了阴影部分与非阴影部分的全局相关性，并且提出了阴影部分与非阴影部分在光照等方面存在联系，设计了Shadow-Interaction Attention，总体架构是一个Encoder-Decoder的结构，在Encoder和Decoder都堆叠使用通道注意力来获得多尺度特征。同时在中间使用提出的Shadow-Interaction Module,每次特征图大小缩小到一半，通道数扩大一倍。
Obtain the low-level feature embeddings
Exploit non-shadow regions to help shadow region restoration
Shadow-Interaction Module模块中采用了类似窗口注意力的架构，使用池化操作减少了计算量，使用了异或操作来帮助获得更显著的权重。
类似于CP的思想
A Shadow-Interaction Module (SIM)
with Shadow-Interaction Attention (SIA) for exploiting the global contextual information across both spatial and channel dimensions from non-shadow region to help shadow region restoration in the bottleneck stage.
All-in-one Multi-degradation Image Restoration Network via Hierarchical Degradation Representation
Image restoration is a critical topic in low-level vision, which generates a high-quality image from a damaged image caused by degradation, e.g. , blurriness, noise, and low illumination.
The multi-degradation that this paper focuses on refers to a dataset with multiple degradations, differing from mixed degradations in an image.
From Sky to the Ground: A Large-scale Benchmark and Simple Baseline Towards Real Rain Removal
摘要结尾参考借鉴：
Extensive experiments verify the superiority of the proposed dataset and de-raining method over state-of-the-art.
Cascaded Latent Diffusion Models for High-Resolution Chest X-ray Synthesis
Deep Unrolling：深度网络与传统模型之间的桥梁
传统模型驱动的思路和端到端学习深度网络建模方法的区别与联系。具体来说：模型驱动方法通常首先利用领域知识为问题建立数学模型，进而通过特定的优化算法或求解策略得到模型的最优解。相对应地，端到端学习方法则希望直接建立输入数据与输出结果的非线性映射关系，其性能往往与训练数据的质量和规模有密切关系。总结起来，我们发现以上两种方法分别主要侧重于利用“领域知识”或者基于“数据（及经验）”来解决计算机视觉问题。很自然的，如果我们可以有效结合这两种不同的策略，将非常有希望设计出更加强大、且可解释性更强的深度网络模型。这也引出了我们后面将向大家介绍的Deep Unrolling技术。
近年来，Unrolling技术为“领域知识”与“数据（及经验）结合提供了一种有效手段。具体来说，所谓Unrolling是指我们将求解一个给定连续模型的迭代优化看成是一个动态系统，进而通过若干可学习模块来离散化这一系统，得到数据驱动的演化过程（Data-dependent Propagation）的方法。实际上，Unrolling过程与机器学习领域的元学习（Meta Learning）和近期LeCun等人提出的可微分编程（Differentiable Programming）概念也有着非常密切的内在联系。在Unrolling方法中，我们首先需要设计一个迭代框架（Schematic Scheme），在此基础上进一步将可学习模块嵌入到迭代中。更为重要的是，我们还需要对于所设计的演化过程进行必要的理论分析。
DATA AUGMENTATION FOR CHEST PATHOLOGIES CLASSIFICATION
In our work we used geometric, color and noise augmentations, which can be classified into such categories: 1) Affine transformations: horizontal/vertical flip, random degree rotations. 2) Color transformations: random brightness, contrast, gamma transformation. 3) Image quality transformations: additive Gaussian noise, random blur.
Keeping the natural bias of images is a necessary condition of data augmentation transformations, e.g. a transformation which somehow hides a lung nodule will negatively affect a classification model.
Improved detection of solitary pulmonary nodules on radiographs compared with deep bone suppression imaging
(A,C) Conventional chest radiography images show cancer partly obscured by ribs.
(B,D) Deep bone suppression imaging of soft tissues clearly show the cancer location, with different lengths of burrs.
(E,G) Dual-energy subtraction soft-tissue images also clearly show cancer.
(F,H) Lesion features are apparent on digital tomosynthesis(DTS) images, and even include pleural traction signs.
(I,J) Computed tomography(CT) images show the nodule more details, with shallow lobulation, different lengths burrs, and pleural stretch.
Subjective Evaluation:
Six observers, comprising 3 radiologists (10–30 years' experience each) and 3 radiology residents (2–3 years' experience each), evaluated the 256 study cases in different randomized orders.
In each reading session, the observers first analyzed the
CCR images, then CCR plus deep BSI images, then CCR
plus DES images, and finally, the DTS images, with the
reading time limited to 3 min.
Bone Suppression on Chest Radiographs for Pulmonary Nodule Detection: Comparison between a Generative Adversarial Network and Dual-Energy Subtraction
关注数据获取方式的描述
用设备和DL合成的骨抑制图像对比的主观评价指标
区分：ROC/AUC
IoU/ROI
1. IoU (Intersection over Union)，交集并集比
2. ROI (region of interest) , 感兴趣区域
3. ROC (Receiver Operating Characteristic curve) 受试者工作特征曲线
4. AUC (Area Under the Curve) ， 曲线下区域
首先要区分，前两个是目标检测领域的术语；后两个是从医疗领域引进的，但是所有机器学习准确率都可能用到该指标。
Value of bone suppression software in chest radiographs for improving image quality and reducing radiation dose
Bone suppression software significantly improved the image quality of bone suppression images with a relatively lower radiation dose, compared with dual-energy subtraction technique.
后续可以关注这篇论文，可以借鉴。
Computer-aided Detection of Lung Tumors in Chest X-ray Images Using a Bone Suppression Algorithm and A Deep Learning Framework
Segmentation: U-Net
Classification: VGG, the outline of the model is the second half of the segmentation model.
For this classification, we used an RGB image as input, where the labels obtained by segmentation are combined with the original chest X-ray image. Since the classification is performed using the output image of segmentation as input, the
segmentation result must be excellent.
The Curve (AUC) as evaluation method for classification. IoU is an index often used to evaluate object detection and segmentation tasks and it represents how large the predicted label is overlapped with its ground truth label.
在实验中，我们发现IoU的值随着输入图像大小的增大而下降，这是因为图像越大，各像素点之间的空间关系更加复杂，做精确的分割也更加困难。
Evaluation of Deep Learning Methods for Bone Suppression from Dual Energy Chest Radiography
Our results show that wavelet decomposition does not improve the rib suppression, “skip connections” modification outperforms baseline autoencoder approach with and without the usage of the wavelet decomposition, the residual models are trained faster than plain models and achieve higher validation scores.
The double-shot DES images are usually partially misaligned due to motion artifacts, e.g. patient breathing. To compensate for such misalignment, low energy diagnostic CXR were first non-rigidly registered to DES soft-tissue images.
The “skip connections” idea improves the standard autoencoder in a number of ways: it addresses the “vanishing gradient” problem and helps to preserve small details of the original image.
During the initial experiments, it was noticed that the higher contribution of MS-SSIM led to better results yet slower convergence. To account for this observation and train network efficiently, the weight of MSSSIM was gradually increased at every training epoch.
同时关注论文中对于数据增强的处理方式，可以借鉴。
使用L1正则而非L2正则的原因在于，L1正则比L2正则导致的图像模糊要少一些，但L1的缺点在于不能捕捉sharp edges和fine textures，它帮助学习general appearance of skeleton.
PatchGAN可用于图像的模糊重建
WHEN DOES BONE SUPPRESSION AND LUNG FIELD SEGMENTATION IMPROVE CHEST X-RAY DISEASE CLASSIFICATION?
In many applications combining different predictors can lead to improved classification results, which is known as ensemble forming.
DICOM的线性映射？
使用1st和99th percentile作为最小和最大值可以确保保留了大部分图像的信息，同时去除了一些异常值和噪声，使得图像更适合可视化和分析。
Image-to-Images Translation for Multi-Task Organ Segmentation and Bone Suppression in Chest X-Ray Radiography
A multitask deep learning model that generates simultaneously the bone-suppressed image and the organ-segmented image, enhancing the accuracy of tasks, minimizing the number of parameters needed by the model and optimizing the processing time, efficiency in learning optimizes the number of required parameters, memory or storage requirements, computational time and training convergence rate. all by exploiting the interplay between the network parameters to benefit the performance of both tasks.
交叉验证的目的：
从有限的学习数据中获取尽可能多的有效信息；交叉验证从多个方向开始学习样本的，可以有效地避免陷入局部最小值；可以在一定程度上避免过拟合问题；提升模型的泛化能力。
多器官分割用的模型：Pix2PixMTdG(d表示扩展卷积)
The Jaccard index is a metric that measures the percent overlap between the target ground-truth mask (GT) and our prediction mask (PM)==IoU
GAN-based disentanglement learning for chest X-ray rib suppression
关注模型的应用评估：
We employ two downstream applications, including lung disease classification and tuberculosis detection, to evaluate the quality of the imputed rib-suppressed CXR images.
Model Complexity
MACs (Multiply Accumulate Operations)
Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric."
Johnson, Justin, Alexandre Alahi, and Li Fei-Fei. "Perceptual losses for real-time style transfer and super-resolution."
Feature construction loss
Dilated conditional GAN for bone suppression in chest radiographs with enforced semantic features
Learning Bone Suppression from Dual Energy Chest X-rays using Adversarial Networks
Such a naive adoption of convolutional auto-encoder families(G.2017) often fails to capture the sharpness since the network misses high frequency details, which are the main reason of blurry images, in its encoding and decoding system. U-Net have overcome this limitation and achieved high performance on segmentation task with skip connection in the auto-encoding process.
In JSRT dataset, each image is of 2048*2048 resolution with three channels in PNG format. Considering the time and memory cost, we rescaled these images to 256*256 resolution, and the range of their pixel values has been linearly scaled to 0-255(more uniform).
Noise Power Spectrum (NPS) This metric gives a complete description of the noise with its amplitude over frequency
resolution.
关注penalty weight的选择
Learning Bone Suppression from Dual Energy Chest X-rays using Adversarial Networks
However, it still suffered from the lack of training data, which leads the model to often fail to capture the outline of the small blood vessels in the lungs and chest.
Haar 2D Wavelet Decomposition + GAN有效
Frequency information obtained from wavelet decomposition have a very critical role in training deep neural network. e.g., Pix2Pix
The artifacts processing: We manually examined the dataset since there was no post processing to handle this problem in acquisition of original images. 11 X-ray images were excluded from the training set and used for additional test. In this case, the ground truth obtained via DES can not be used as a reference image to evaluate the results. So in a qualitative manner, the motion artifacts due to heart beat did not appear and almost all information was maintained without blurry results.
先做linear transformation
再做linear mapping
这一过程叫linear windowing
It is equally important to preserve spatial information and image quality because they provide crucial insights on the size and area of infection, color accuracy, structural quality, etc.
Minimize the visibility of bones in CXRs while ensuring maximum retention of critical information
‘CheXNeXt’ for the screening of 14 categories of diseases such as pneumonia, fibrosis, emphysema, and effusion, etc. have developed a model ‘Covid-Screen-Net’ for mass screening of COVID-19.
普遍的模型存在以下缺陷：hard denoising, that leads to a reduction in the sharpness of an image, loss of texture, and introduction of small artifacts.
We pre-set the value of α as 1 and find the relative values of β and γ. 下一页PPT有详细解释参数选择实验
a resolution of 512 × 512 is sufficient for the detection of lung diseases and nodules. Therefore, we resized all the images to 512 × 512 × 3 for training SFRM-GAN
不同评价指标作用：
They used the combination of PSNR and NMSE to indicate the degree of bone suppression. Next, they calculate SSIM and Laplace Variance to indicate retention of spatial feature resolution. Further, the authors measure Entropy and BRISQUE(不用) score to indicate the preservation of spatial feature information.
Role: Detect edges and determine the gradient error. The model becomes more susceptible to the changes in high-frequency details, i.e., lung boundaries, blood vessels, infectious areas, etc.
Sobel Loss using Sobel operator
Mixed gradient error
Single image super-resolution based on a modified U-net with mixed gradient loss
Dual energy subtraction: Principles and clinical applications
在上页论文中效果不好，改成了Sobel loss
The aim of measure the gradient error is to learn a sharp edge which is close to the ground truth edge.
共同点：一次三张图片
双能成像实现方式---均可获得高低两种能量
Bone Suppression on Chest Radiographs With Adversarial Learning
While training with paired images yields slightly better performance than that of unpaired images when measuring with two objective image quality metrics, namely Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR), training with unpaired images(Unsupervised Learning) demonstrates better generalization ability on unseen anteroposterior (AP) radiographs than paired training.
Unpaired?----Unpaired training radiographs consist of a source set and a target set, with no information provided as to which standard CXR matches which bone suppressed CXR.(任意的两对)
We find that the GAN models introduce minimal motion artifacts compared with the dual-energy subtraction technique. The reason is that there is only a small portion of training data in the DE dataset contains motion artifacts. The image-to-image translation models tend to learn the majority of information from the entire data distribution. This characteristic of adversarial learning models can be considered as yet another main advantage of automatic bone suppression in addition to less radiation exposure.
Image2Image
Generating Dual-Energy Subtraction Soft-Tissue Images from Chest Radiographs via Bone Edge-Guided GAN
Detection of Bone-Edge
Considering that only a limited number of subjects are used for detecting bone edges, we employ the same architecture as the most compact FC-DenseNet56 model.(防止过拟合)
模型能力：In the test stage, our method only required about 1.04 seconds (0.83 s for edge detection and 0.21 s for image synthesis) to generate the bone edge map and DES-like soft-tissue image for an unseen CXR image.
Deep Learning Models for Bone Suppression in Chest Radiographs
Supervised:
Regressors are trained or optimized by teaching DES radiographs to estimate the soft-tissue or bone images.
Unsupervised:
Do not require teaching DES with radiographs, but these methods need segmentation and the border locations of bony structures. The bone-free images are reconstructed with the blind-source separation approach, or the gradient images modified according to the intermediate results.
MS-SSIM/SSIM loss pays attention primarily to local neighborhood changes.
CLAHE的作用: CLAHE for local contrast enhancement, so local details can therefore be enhanced even in regions that are darker or lighter than most of the image.
测试和训练数据一起做预处理操作的目的：
Bone suppression for chest X‑ray image using a convolutional neural filter
双能减影图像的另一缺点：
Organs with attenuation coefficients similar to bone are depicted on bone-extracted images along with bones themselves. If training is conducted based on such images, the output image will depict organ information as well as bone like the teacher image. Some organ information is thus lost in processing of the bone-suppressed image.
Bone-extracted images, which is the difference image between input image and ideal image, were used as teacher images.
The results of comparison between the proposed
method and the results described in those five papers.
Not the same dataset!
Bone suppression for chest X‑ray image using a convolutional neural filter
Applying a Conditional GAN for Bone Suppression in Chest Radiography Images
基于Patch的思想
The architecture of the CNF
Input to the CNF is 31 × 31 pixel values neighboring the target pixel. Output from the CNF is a value of the bone component of the target pixel.
Comparison:
Similar/Same 的dataset/technique
Autoencoder-based bone removal algorithm from x-ray images of the lung
Gusarev, M数据集描述
Deblurring Method:
1.Createa cascade solution, where a new neural network is included to remove blur from the output images.
2.It is also possible to use traditional image processing techniques to improve the quality of the output image. Simple example of such techniques are filters, like Bilateral, Wiener or Laplacian filter.
参考本篇论文中Different learning rate schedules
Acknowledgements: We would like to thank the researchers from the Innopolis University, Innopolis, Russia for sharing The Bone Suppression dataset.
Bone suppression on pediatric chest radiographs via a deep learning-based cascade model
Domain adaptation:是迁移学习的一种，主要解决数据不方便采集，数据标注耗时费力等情况。
不同输入数据集也可以做消融实验。
Visual scoring regarding bone suppression metrics
Score 5 implies that bones were almost fully invisible throughout; score 4 implies that bone suppression was good, although the remaining bone density was lower than the pulmonary right hilar vascular density; score 3 implies that bone suppression was good, although the remaining bone density remains at the level of pulmonary right hilar vascular density; score 2 indicates that bone suppression was partially incomplete; and score 1 indicates that bone suppression was not achieved.
DEEP FEATURE DISENTANGLEMENT LEARNING FOR BONE SUPPRESSION IN CHEST RADIOGRAPHS
An accurate bone suppression algorithm requires the model to identify the location of bony structures and discard them but retain the information about soft-tissue overlapping with the bones.
Guide借鉴:
Our goal is to retrieve the soft-tissue image S from the chest radiograph. Most of existing methods force the networks to map X to S without any supplementary guidance except for the reconstruction supervision. The lack of precise supervision signal makes learning inefficient and leads to suboptimal results. To address this issue, we propose a multi-task learning strategy to clearly guide the network to focus on the most important regions and separate the two components.
MSE缺点:
However, the MSE loss ignores the correlation between pixels and may lead to undesirable blurring of output images.
借鉴: Visualization of zoomed in images to display details.
MS-SSIM参数选择：
Cross-Entropy
Deep learning-based bone suppression in chest radiographs using CT-derived features: a feasibility study
激活函数PReLU
当a=0时，PReLU就变成了ReLU，而且 a 是一个可以学习的参数。如果 a 是一个小的固定值， PReLU就变成了 LReLU （Leaky ReLU)。LReLU目的是为了避免零梯度，实验表明LReLU相较 ReLU对准确率几乎没有提高，但是PReLU的参数 a 可以自适应学习，从而可以获得专门的激活函数
流程
含internal validation
From 3D to 2D: Transferring knowledge for rib segmentation in chest X-rays
GAN的其中一个作用：Domain adaptation or Transfer the knowledge
High-Resolution Chest X-Ray Bone Suppression Using Unpaired CT Structural Priors
Histogram matching is a method that can preserve the detail of the original image and match the intensity distribution of the target image.
Bone Suppressed CXR for Automatic Disease Classification using DensNet-121:
?We concatenate two original CXR images and one bone-suppressed CXR image of our method to form three channels. We also take only the bone-suppressed CXR images of the Shenzhen hospital dataset as input to test the performance.
Subjective Evaluation除了看骨抑制效果外，还可以判断图片中有无abnormal，即做诊断。
Bone suppression of lateral chest x-rays with imperfect and limited dual-energy subtraction images
Multi-scale and dual-domain
In fact, the gradients contain enough information for reconstructing the images. Moreover, the gradients are sparse, and the probability of both soft-tissue and bone components having large gradients of the same orientation at the same location is low (Chen et al., 2009). Therefore, the soft-tissue and bone components of CXRs in the gradient domain are more separable than in the intensity domain.
蒸馏学习（Distillation Learning）
后续关注MAP
Improving Tuberculosis Recognition on Bone-Suppressed Chest X-rays Guided by Task-Specific Features
Existing bone suppression models are usually trained on a relatively small dual-energy subtraction (DES) dataset with fewer variety of CXRs, thus ignoring the specificity of tuberculosis in CXRs from other datasets. Recent studies have shown that modeling the disease-associated image specificity helps improve disease detection performance.
Multi-scale CXR features, i.e., f1, f2, f3 and f4 can be explicitly transferred to the image synthesis network (via channel-wise addition) to guide the image synthesis process.
Shallow Network: ResNet18
DICOM格式图像的预处理方法
Four state-of-the-art methods that perform simultaneous TB diagnosis and TB area detection
In addition, the BCTR also achieves better results compared to BCTR-3 that finetunes the classification network in the second training process. This implies that using bone-suppressed CXRs to finetune a classification network pretrained on CXRs may not produce good results, which is different from previous findings that finetunning usually improves the classification performance. This may be due to the significant heterogeneity between CXRs and bone-suppressed CXRs.
Application: Classify and (to localize) pulmonary lesions
quantitative and qualitative
任务：分类Normal or COVID-19
AUROC, confusion matrix, Sankey diagram, and AUPRC curves
应用评估使用的数据集
COVID-19 CXR collection
RSNA CXR dataset
Ensure if the models learned to highlight regions containing COVID-19-consistent findings and not the surrounding context by using CRM Algorithms
F-EB0-BS bone suppression model is truncated and added with the classification layers
DeBoNet: A deep bone suppression model ensemble to improve disease detection in chest radiographs
For classification, we initially used a custom U-Net model[1] proposed in to segment the lung ROI on the CXRs. This approach ensures that the models learn relevant features from the lung ROI and not the surrounding context.
Cascade of multi-scale convolutional neural networks for bone suppression of chest radiographs in gradient domain
The task of predicting bone or soft-tissue components from a CXR is analogous to image denoising, rain drop
removal, and edge-aware filtering.
Gradient domain[1]优点：
The gradients contain enough information for reconstructing the images. Moreover, the gradients are sparse, and the probability of both soft-tissue and bone components having large gradients of the same orientation at the same location is low. Therefore, the soft-tissue and bone components of CXRs in the gradient domain are more separable than in the intensity domain. The mapping between the gradients of CXRs and their bone components may be less complicated than the mapping between the intensities.
From coarser to finer
区分几个概念：
尺度：图像的模糊程度
尺度越大，分辨率下降，是downscale的结果，显示的是全局特征。
Bone Suppression of Chest Radiographs With Cascaded Convolutional Networks in Wavelet Domain
Compared with previous work of CamsNet model which preforms bone prediction in gradient domain, the Wavelet-CCN model predicts the wavelet coefficients to reconstruct bone images and can avoid the inconsistent background intensity caused by 2D integration of gradients.
Wavelet decomposition results of (a) CXR image, (b) DES bone image, and (c) DES soft-tissue image. (d), (e), and (f) are the corresponding histograms of each sub-band images
in (a), (b), and (c), respectively.
The conclusion is remarkably consistent with Zhao et al. who reported that when the L2 or PSNR metrics were used to evaluate image quality, the network trained with the L1 loss function outperformed the one trained with the L2 loss function.
The motion artifacts are almost eliminated visually.
Cascade of multi-scale convolutional neural networks for bone suppression of chest radiographs in gradient domain
Visualization of the learned filters of a CamsNet
Visualization of the feature maps for different convolution layers.
The majority of the filters were similar to edge detectors of different scales from different directions. These edge detectors were used to reconstruct the gradients of the bone image, and they were considered to be the base for reconstruction.
In these maps, the details related to soft-tissue structures were canceled out. The output bone gradients were sharper than the input bone gradients。
Generating Dual-Energy Subtraction Soft-Tissue Images from Chest Radiographs via Bone Edge-Guided GAN
骨抑制存在的两大挑战：
However, eliminating artifact effects completely is difficult due to the challenge of accurately locating the motion artifact areas. Due to imperfect ground truth, some residual bone edge signals remaining in DES images, supervised deep models trained on DES data cannot suppress bone components of CXRs thoroughly.
Evaluation of Cross-Dataset Generalization
We further applied the proposed EGAN model to synthesize soft-tissue images from CXRs acquired by using different types of X-ray machines in our DES test set, the publicly available JSRT, NIH ChestXray 14(200 images), MC, and SH datasets.
Note that EGAN is trained on our DES dataset and directly applied to these five independent datasets.
Evaluation of Synthetic Images in Tuberculosis Diagnosis
For bone suppression, the selection of prior knowledge is essential in prior-guided models.(i.e. prior = bone)
1.A 5-fold cross-validation strategy is used.
2.Pre-processing:
Each CXR image was resized into a 256 × 256, followed by being cropped into 224 × 224(ResNet18的输入严格规定为3*224*224) and duplicated into three channels.
3.Even though a previous study have shown good results using pre-trained models on the NIH ChestXray14 or ImageNet dataset, we train the diagnosis models from scratch for the sake of fair comparison.
Separation of bones from soft tissue in chest radiographs: Anatomy-specific orientation-frequency-specific deep neural network convolution
Single-shot DE imaging VS dual-shot DE imaging
Less motion artifacts in soft-tissue and bone images. Limitations of single-shot DE imaging are, in general, a slightly increased noise level and slightly poor energy separation. Because virtual DE imaging based on deep learning requires pairs of input chest radiographs and corresponding “teaching” DE bone and/or soft-tissue images with no/little motion, single-shot DE imaging is suited for creating images for training deep learning models.
Two-fold cross-validation
描述我们模型优秀的一些写作参考：
As seen, our scheme was better able to suppress bones including the ribs near the lung wall and the clavicles, and specifically rib edges, than the reference technique. Additionally, our new scheme was better able to maintain the conspicuity of soft-tissue structures such as lung nodules of various sizes and vessels under the clavicles and with overlapping ribs. This was achieved through our model的哪些功能…
Isometric Convolutional Neural Networks for Bone Suppression of Multi-Planar Dual Energy Chest Radiograph
Illustration of the X-ray scanner used to create dataset
Vascular visibility, contrast or sharpness也是软组织生成效果好坏的指标
Image Post Processing
The reason for the images to have lower contrast while having high evaluation is that the contrast of an image is calculated based on the lowest and highest pixel in the image. A model could miss predicted only single pixel and that pixel will drastically affect contrast of the whole image. Concretely, the outlier prediction. Outlier prediction in this case is the overly high or low predicted value. Even if there is only one pixel of outlier, every pixel value in the image will be re-scaled such that the outlier pixel is the brightest or darkest pixel resulting in a reduction of the contrast.
作者分析了图像模糊和低对比度的原因；�作者提出了解决图像模糊和低对比度的方法。
One cause of the quality deficiency is the resolution of the target image. Since there are more target values for the model to fit with a fairly small number of trainable parameters, model’s bias is expected and result in blurriness.
Development and validation of bone-suppressed deep learning classification of COVID-19 presentation in chest radiographs
Resize的方法：Nearest-neighbor interpolation
Model: The pre-trained 2 class ‘COVID-Net CXR2’ method and the ‘VGG16-Modified’ (Rajaraman S) method
JSRT数据集处理：
To train the bone suppression models, the dataset was cleaned by removing image pairs where poor suppression (defined as observing at least 1 rib or clavicle with edges visible across the lung tissue) and image artifacts were observed. A total of 217 image pairs (131 with nodule, 86 without nodule) remained after data cleaning.
VGG16 had been chosen because it had been used in previous chest disease classification studies and had demonstrated improved results over alternatives such as DenseNet-121 and ResNet-50.
Development and validation of bone-suppressed deep learning classification of COVID-19 presentation in chest radiographs
几个检测方法
NPV is important because the public health consequences of missing a COVID-19 diagnosis was deemed to be greater than that of mistakenly identifying someone as infected.
敏感度就是有病的里边能看出来多少，特异度就是没病的里边能排除多少
阳性预测值就是某诊断实验说有病的人中有多少是真的有病的，阴性预测值(NPV)则反之
The trained bone suppression models and VGG16-Modified models used in this study are available for use from https://github.com/danielnflam
.
Future work: More image data could be acquired to improve both the classifier performance and the generality of the external test set. Other image enhancement techniques beyond bone suppression could be pursued to improve classification performance for other clinical conditions imaged using other modalities. Radiomics analysis of the enhanced images, and analysis of other non-deep learning features, could be combined with deep learning features to further improve classification performance.
COVID-Net CXR2 demands:
All images input into the classifier were processed as follows:
(I) the topmost 8% of the image was removed in order to remove metadata in the image; (II) the image was center cropped by removing both ends of the longer side of the image, such that the remaining image is square; (III) the image was resized using bilinear interpolation to 480×480 pixels; (IV) the image was input into the model’s neural network for classification.
使用N Cross-Validation
Generation of Virtual Dual Energy Images from Standard Single-Shot Radiographs using Multi-scale and Conditional Adversarial Network
CXR: As a noninvasive, low radiation dose, and low cost imaging modality,
Dual energy subtraction (DES) can separate high-density material, such as bone, from soft tissue
Image processing pipeline of bone suppression with cross projection tensor,这里的得到的软组织并不是直接相减
条件GAN
PSNR is commonly used for evaluating quality of image compression/reconstruction, where our algorithm can be viewed as a compression/reconstruction process.
Generation of Virtual Dual Energy Images from Standard Single-Shot Radiographs using Multi-scale and Conditional Adversarial Network
Normalized all images to the range of [−1, 1] per image，Ground truth intensity maximum and minimum is then applied to the normalized virtual bone image to recover the final result.
介绍JSRT: the size of a pixel was 0.175 * 0.175 mm
主观应用评估：
Lung nodule detection+ FROC 具体实施方式如下
怎样表明骨抑制效果好：
Cardiac motion artifacts are significantly reduced in the virtual DE images. High-quality and high-resolution virtual bone and soft tissue images with subtle details(病变区域) are produced.
和谁比？Ground truth/其他方法
参考下一页
和其他方法比
和Ground Truth比
Chest X-ray Bone Suppression for Improving Classification of Tuberculosis-Consistent Findings
An ImageNet-VGG-16 model is pretrained on a large collection of publicly available CXRs.
A previous study shows the benefits of using CXR modality-specific models retraining toward improving classification and localization performance and model generalization.
新的评价指标：马修斯相关系数（Matthews correlation coefficient）
马修斯相关系数是在使用机器学习作为二进制（2类）的质量的度量的分类指标。
SSIM provides a superior indication of prediction performance as it exemplifies human visual perception. The MS–SSIM measure is an extension of SSIM that computes structural similarity at various scales and combines them. On the other hand, MAE preserves luminance and contrast in the predicted image.
新的评价指标：Comparing the histogram of the predicted image using the proposed bone suppression models and the ground truth in pixel value
Chest X-ray Bone Suppression for Improving Classification of Tuberculosis-Consistent Findings
For the finetuning task, fourfold cross-validation is performed in which the baseline and bone-suppressed CXRs in the Shenzhen and Montgomery TB collections are split at the patient level into four equal folds. The hyperparameters of the models are tuned while training on the three folds and validating with the fourth fold. The validation process is repeated with each fold, resulting in four different models.
We observed that the filters in the first convolutional layer learned the edges, contours, orientations, and their combinations, specific to the input image. However, in the deepest convolutional layer, the filter activations were abstracted to encode class-specific information. This demonstrated that deeper convolutional layers encode class-specific details, while the initial layers contain image-specific activations. Additionally, the activation sparsity increased with model depth.
Visualizing the activations of the first 64 filters in (a) block1-Conv1 layer and (b) block5-conv3 layer.
We use this final improved model architecture as our default:
Variable width with 2 residual blocks per resolution, multiple heads with 64 channels per head, attention at 32, 16 and 8 resolutions, BigGAN residual blocks for up and down-sampling, and adaptive group normalization for injecting timestep and class embeddings into residual blocks.
Stochastic diffusion sampling，以y为条件, condition_mean的方法
有标签y
参考：
Recall (a measure of diversity)
IS (a measure of fidelity)
FID and sFID depend on both diversity and fidelity
Guidance的缺点：数据集必须要有label！
Classifier的训练：
We train these classifiers on the same noising distribution as the corresponding diffusion model, and also add random crops to reduce overfitting. After training, we incorporate the classifier into the sampling process of the diffusion model.
Palette: Image-to-Image Diffusion Models
Loss:
We find that while L2 and L1 losses in the denoising objective yield similar sample-quality scores, L2 leads to a higher degree of diversity in model samples, whereas L1 produces more conservative outputs.
Evaluation Metrics:
Specifically, we report Fréchet Inception Distance (FID), Inception Score, Perceptual Distance and Classification Accuracy for qualitative comparison. When computing FID scores, the choice of the reference distribution is important. In our work, we use the full validation set as the reference distribution. For Perceptual Distance, we use the Euclidean distance in the pool_3 feature space of the pre-trained InceptionV1network (same as the features used for calculating FID scores). We use EfficientNet-B0 top-1 accuracy for reporting ground truth Classification Accuracy scores.但分类用的是预训练好的ResNet-50!
所谓的Top-1 (Accuracy)是指排名第一的类别与实际结果相符的准确率，而Top-5 (Accuracy)是指排名前五的类别包含实际结果的准确率。
The ultimate evaluation of image-to-image translation models is human evaluation.
Self-Attention of U-Net in diffusion <Diffusion models beat gans on image synthesis>
We also find that removing self-attention layers from the U-Net architecture of Palette, to build a fully convolutional model, hurts performance. While self-attention layers provide a direct form of global dependency, they prevent generalization to unseen image resolutions. Generalization to new resolutions at test time is convenient for many image-to-image tasks, and therefore previous works have relied primarily on fully convolutional architectures.
个人理解：inpainting需要image resolution的generalization和a direct form of global dependency的平衡
CBCT-Based Synthetic CT Image Generation Using Conditional Denoising Diffusion Probabilistic Model
Our proposed conditional DDPM for sCT generation from CBCT, composed of unconditional forward(CT to CT) and conditional reverse processes.
CBCT-Based Synthetic CT Image Generation Using Conditional Denoising Diffusion Probabilistic Model
PSNR measures if the predicted sCT intensity is evenly or sparsely distributed. NCC is a measure of similarity between CT and sCT as a function of displacement.
新的评价指标 normalized cross-correlation
几种不同的条件diffusion
An unconditional DDPM is pretrained using CBCT images to translate the Gaussian noise to the CBCT image.即训练时不添加条件。
1.Iterative latent variable refinement (ILVR) take self-information (CBCT)
as condition 2.Weighted summation
Energy-Guided Diffusion Model for CBCT-to-CT Synthesis
Total loss:
借鉴loss的合成思想？怎么样修改diffusion传统的loss？
Change with time t
Diffusion Models for Memory-efficient Processing of 3D Medical Images
Coordinate encoding (CE) for the patches
Trains on randomly sampled patches of the input but can afterwards be applied to the full resolution volume during inference.
Benchmark：其性能被广泛研究，可作为标准方法来衡量其他方法的好坏；区别于SOTA（state-of-the-art），SOTA表明其性能当前最佳；如果 一个新算法最好以SOTA作为benchmark。
Baseline：一个指标/数据集/算法被称为baseline，基本上表示比这个算法性能还差的基本上不能接受，所以其自带的含义——“性能起点”；在算法优化过程中，表示你的算法能达到的一个基本性能，在算法继续优化和调参的过程中，你的目标是比这个性能更好。
总结：benchmark一般是和同行中比较牛的比较；baseline是在算法优化和调参过程中与自身的比较。
The diffusion model is optimized in memory efficiency and speed by training only on coordinate-encoded patches.
Improving Diffusion Model Efficiency Through Patching
We report both the sample quality measured by FID and the sampling speed. We find that P = 4 achieves a good trade-off between efficiency and quality, so we use this choice for our other models.
We study the effect of the output parameterization in diffusion models, both with and without patching, and find that data prediction is less fragile than noise prediction.
Therefore, we recommend that patched models use x or v prediction
在patch拼接后会存在：
Training:
关键
分割和检测任务在医学影像领域的评估指标Dice Score
Hyperparameter Sensitivity中的控制变量
i.e.
If L is chosen too large, this results in a destruction of the images. If L is chosen too small, the model does not have enough freedom to remove the tumor from the image.
另外一种情况：固定L，改变s
If we choose s too small, the tumor cannot be removed. However, if we choose s too large, additional artefacts are introduced to the image. Those artefacts are mainly at the border of the brain, and lead to a decrease in the Dice score.
两个贡献：
1.对任意大小的图片均可处理，实际上是把图片resize成r(类似于shifted window)的倍数
2.Single-weather和Multi-weather上的效果都很好
Deterministic Implicit Sampling
Number of implicit sampling steps S
Overlapping fixed-sized patches
本文Patch的优点：Without emerging edge artifacts
对于DDPM/DDIM/Guidance的参数设置：We did not perform task-specific parameter tuning or modifications to the neural network architecture.
Overall, we observed that a smaller number of implicit sampling time steps was sufficient for 64x64 patch-based models and determined S = 10 for weather-specific, and S = 25 for multi-weather models.
Exploiting Diffusion Prior for Real-World Image Super-Resolution
主要借鉴其中的patch思想
Progressive aggregation sampling strategy（渐进聚合采样策略）
克服预先训练的扩散模型的固定大小约束，从而能够适应任何大小的分辨率
一般方法：将较大的图像分割成几个重叠的较小patch，并分别处理每个patch。不适用于diffusion，在扩散迭代过程中，patch之间的差异会被放大，原因在于diffusion模型的随机性。
Meanwhile, we also de-fine a padding function f(·) that expands any patch of size 64×64 to the resolution of h×w by filling zeros outside the region Ω.------目的：unify the size
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models
A learning-free method to guide the generative process in DDPM to generate high-quality images based on a given reference image.
特点：
Enables control on high-level semantic similarity of generated images to the reference from coarse to fine.
Reference images can be selected from unseen data domains.
两个检测指标的介绍：越低说明越相似
We evaluated the quality and diversity of our generated images with widely used FID and LPIPS. The FID score evaluates the visual quality and distance between real and generated image distributions. LPIPS measures the perceptual similarity between two images.
Low-pass filters
本质上是一系列上下采样的滤波器
We used bicubic down-sampling and up-sampling with
correctly implemented function.
https://zhuanlan.zhihu.com/p/436720853
https://blog.csdn.net/search_129_hr/article/details/127629904
Zero-Shot Learning
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models
Control unconditional DDPM
Motivate controlling of two factors: downsampling factors and conditioning range
想要生成的样本和reference image共享语义信息，语义信息具体层次由缩放因子N决定。
Specifically, we refine each unconditional transition with a down-sampled reference image.
Free-training
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models
从不同的下采样因子中生成，我们可以看到随着N的加大，生成图像与参考图像的相似性逐渐减少，这是因为在较低的分辨率空间中，算法可以从更广泛的图像集中采样，生成的图像更具多样性，换句话说就是，较高的因子N可以共享粗略的特征，使生成器发挥的空间更大，而较低的因子N共享更精细的特征，约束条件更强。
在不同的采样步数下生成，我们可以看到更狭隘，更短的采样距离，生成图像更具多样性。采样步骤越多，生成的图像与参考图像越相似。与下采样因子的作用相比，步数体现在细粒度上。
B🡪A
高分辨率图像更加精细，因为它们具有更多的像素，可以提供更多的细节信息。而低分辨率图像虽然可能不太清晰，但它们可以提供更广泛的语义信息，因为它们可以呈现更广泛的场景或对象。
DeS3: Attention-driven Self and Soft Shadow Removal using ViT Similarity and Color Convergence
可借鉴：
Structure Preservation Loss
结合了一个预训练的网络 + 自相似性的思想
We extract keys (deep features) from the pre-trained DINO-ViT.
High-Resolution Image Synthesis with Latent Diffusion Models
As image size grows, so does the computational power required to work with those images. This is especially pronounced in an operation called self-attention, where the amount of operations grows quadratically with the number of inputs. A 128px square image has 4x as many pixels as a 64px square image, and so requires 16x (i.e. 42) the memory and compute in a self-attention layer. This is a problem for anyone who'd like to generate high-resolution images!
Latent diffusion helps to mitigate this issue by using a separate model called a Variational Auto-Encoder (VAE) to compress images to a smaller spatial dimension. The rationale behind this is that images tend to contain a large amount of redundant information - given enough training data, a VAE can hopefully learn to produce a much smaller representation of an input image and then reconstruct the image based on this small latent representation with a high degree of fidelity. The VAE used in SD takes in 3-channel images and produces a 4-channel latent representation with a reduction factor of 8 for each spatial dimension. That is, a 512px square input image will be compressed down to a 4x64x64 latent.
优点：
This innovation dramatically lowers the cost to train and run these models.
Particularly, it pre-trains an autoencoding model (i.e., an encoder-decoder architecture) to learn the perceptual space, which is able to preserve the perceptual quality of the reconstructed image while reducing the computational complexity.
Diffusion Models for Image Restoration and Enhancement – A Comprehensive Survey(具体可以参考笔记)
损失函数
Some pioneering IR studies pinpoint that previous pixel-wise losses, e.g., MSE loss, and L1 loss are susceptible to the blurring textures
分类：
Supervised DM-based: SR3
Zero-shot DM-based: ILVR(可参考其描述)
Standard: Whether the diffusion models (DMs) are Training-free
Supervised: Training the diffusion model from scratch relies on a large quantity of paired training data; Collecting paired distorted/clean images in the real world are challenging.
Zero-shot DM-based: Offer an appealing alternative by requiring only distorted images and dispensing with the need for retraining diffusion models. Instead of acquiring the restoration capability from the training datasets of IR, it excavates and exploits the structure and texture priors from the pre-trained diffusion models for image restoration. The core idea stems from the intuition that the pre-trained generative models can be viewed as the structure and texture repository, constructed using amounts of real-world datasets,
Improvements with Diffusion Model
IDDPM introduces a cosine noise schedule to obviate the negative effects of rapid noise accumulation at the early perturbation stages.
Model Architecture. Diffusion models predominantly employ two kinds of architectures, i.e., CNN-based U-Net, and Transformer-based model.
Diffusion Models for Image Restoration and Enhancement – A Comprehensive Survey(具体可以参考笔记)
Pre-trained模型的优缺点
i)The pre-trained diffusion models are devoted to preserving the data distribution instead of pixelwise data consistency. ii) how to excavate the perceptual knowledge aligned with low-quality images, which imposes higher requirements on the design of the condition.
Zero-shot DM-based
To mitigate the primary challenges in zero-shot DM-based IR, the projection-based method has been introduced in some studies. This approach aims to extract inherent structures/textures from low-quality images as complementary to generated images at each step, which can ensure data consistency.
Evaluation Metrics----参考价值极大
The objective and subjective metrics play a vital role in measuring and comparing the performances between different algorithms of DM-based IR. In this section, we clarify the commonly-used metrics in image restoration in detail,
i.e., PSNR, SSIM [283], LPIPS [284], DISTS [285], FID [286], KID [287], NIQE [288] and PI [289].
To compare the computation cost and network complexity, we also measure the running time, parameters, and flops for diffusion model-based IR methods.
An intuitive solution is that we can generate each part of the whole image, and then stitch them into one image. However, it will cause severe mismatching issues and inconsistencies at the edges of each part, stemming from the inherent randomness within the diffusion model.
🡪提出了overlapping-patch的思想
ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal
Remodel the shadow removal as a joint task to pursue a shadow-free image and refined mask, in which mask refinement would be an auxiliary task of the diffusion generator to progressively refine the shadow mask along with shadow-free image generation.
Degradation Prior
Mask prediction head
Unrolling