Nighttime Combustion Dataset from NASA’s Black Marble Product Suite                                                                                Srija Chakraborty

vnp46a1_thermal_anomaly consists of the Black Marble (VNP46A1) (Román et al. 2018) based per-pixel detections of combustion (fires and gas flaring) using anomalous thermal and light emission over land. The models used are Siamese Networks (triplet, contrastive loss), fully-connected neural network, outlier from thermal, thermal+light bands. The primary dataset is located in vnp46a1_thermal_anomaly/classification. The initial labels are generated using anomaly detection (Chakraborty et al. 2023) which are shared in vnp46a1_thermal_anomaly/anomaly_det. The training, test sample paths are described below and followed by a detailed breakdown of the dataset.

REGIONAL TRAINING DATA (h05v05, h20v10, h08v06, h08v06_w):  

vnp46a1_thermal_anomaly/classification/regional/hHHvVV/training_iter<#>/training_reg; where # is 1 and 2 for iterations

REGIONAL TEST SAMPLES (h05v05, h20v10, h08v06, h08v06_w):

vnp46a1_thermal_anomaly/classification/regional/hHHvVV/ test_<YYYY>_<mon>/label_reg; where <mon> is the month

JOINT TRAINING DATA (based on regional detections from h05v05, h08v06, h08v06_w):

vnp46a1_thermal_anomaly/classification/regional/hHHvVV/training_iter2/training_joint and

vnp46a1_thermal_anomaly/classification/regional/hHHvVV/test_<YYYY>_<mon>/training_joint

JOINT TEST SAMPLES (h05v04, h07v05):

vnp46a1_thermal_anomaly/classification/regional/hHHvVV/ test_<YYYY>_<mon>/label_joint

TABLE 1: VNP46A1 DATA RADIANCE CONVERSIONS:

Fill value=65535

TABLE 2: VNP46A1 CLOUD, CIRRUS, LANDWATER FLAGS:

World Settlement Footprint (WSF): settlement likelihood 0-1, 1-high, 0-none

DESCRIPTION:

vnp46a1_thermal_anomaly

                   /anomaly-det

                   /classification

vnp46a1_thermal_anomaly/classification

                   /joint

                   /regional

/regional/<hHHvVV> where HH and VV are horizontal and vertical tile numbers (h05v05, h20v10, h08v06, h08v06_w; where h08v06 and hv08v06_w are flaring detectors in summer and winter respectively)

TRAINING SAMPLES: training_iter1/training_reg and training_iter2/training_reg (20% split for validation)

                   /WSF_<hHHvVV>_clipped.tif – World settlement footprint clipped to tile ROI.

                   /training_iter1 (first training iteration where the classifier is trained on anomaly detections)

/dataVNP -original VNP46 as single band GeoTIFF. VNP46A1_<band>_<YYYYDOY>.tif where band is M-10, M-11, M-12, M-13, M-15, M-16, DNB.

/dataVNPmask -original VNP46 masks as single band GeoTIFF. VNP46A1_<mask>_<YYYYDOY>.tif  where mask is Cloud, Cirrus, LandWater VNP layers

/training_reg- markers or labels for regionally training background (1) and detections (2) corresponding to dataVNP and saved as training_im_<YYYYDOY>.tif. The detections are derived from anomaly detection. All detections (2) are used for training, while the background is randomly sampled to select the training pixels (1), remainder (background samples) are marked as 0. The labels are derived from anomaly detection by combining the autoencoder, DNB and M-11 detections.

/training_iter2 (second training iteration where the classifier is trained on training samples from training_iter1 (anomaly detection samples) and detections made by the classifier from previous iteration (iter1) on new VNP46 instances in 2018 to create the fully-trained regional classifier)

                                   /dataVNP

                                   /dataVNPmask

/training_reg – markers or labels for regionally training background (1) and detections (2)        for dataVNP saved as training_im_<YYYYDOY>.tif The detections are derived by applying previous iteration1 detector. All detections (2) are used for training, while the background is randomly sampled to select the training samples (1), remainder (background samples) are marked as 0.

/labels_reg – labels or detections of fully-trained regional detector on dataVNP. det_<model>_<YYYYDOY>.tif where models include (cont) contrastive, (trip) triplet, fcnn (fully connected net), t10 (top 10 percentile in M11*4), t10l (top 10 percentile in M11*4 and DNB: min(98 percentile,15)), hconf (high confidence with detections from at least two models from cont, trip, t10l). Here (1) is a detection and (0) is background. For cont, trip, fcnn pixels with prediction probability >0.98 are selected as detections.

det_<model>_filtered_<YYYYDOY>.tif where the model detections from cont, trip, fcnn, and hconf are cleaned to minimize potential contamination.

Filtering non-urban pixels (wsf< 0.05): if difference with background in M-10>= 0.025, or M-11>= 0.015, thresholds visually determined from training_iter2  

Filtering urban pixels (wsf>= 0.05): if difference with background in M-10>= 0.04, or M-11> =0.03, thresholds visually determined from training_iter2

/training_joint – training samples from dataVNP used for jointly training the model. training_im_<hHHvHH>_<YYYYDOY>.tif. Detections are (2), background (1). All detections are used, randomly selected background samples are used (unused background samples are 0). Detections are selected when at least two models out of cont, trip and t10 agree. Cont and trip and selected when there is also a light emission signal checked from exceeding 98% percentile of DNB radiance.

/test_<YYYY>_<mon> (test phase VNP46A1 observations on which the regionally trained models are applied). Follows the same structure as training_iteration2

                                   /dataVNP

                                   /dataVNPmask

                                   /labels_reg: similar structure and thresholds as listed in training_iter2

/training_joint (optional; only for regional test months in 2018 that are used in training the joint model)

/joint/<hHHvVV> where HH and VV are horizontal and vertical tile numbers (h05v04, h07v05, for generalization to fire and flare respectively). Regional models h05v05, h08v06, h08v06_w are used.

TRAINING SAMPLES: vnp46a1_thermal_anomaly/classification/regional/hHHvVV/training_iter2/training_joint and

vnp46a1_thermal_anomaly/classification/regional/hHHvVV/test_<YYYY>_<mon>/training_joint (20% split for validation) : h05v05 (2018 aug, sep), h08v06 (2018 aug, sep), h08v06_w (2019 dec)

/WSF_<hHHvVV>_clipped.tif – World settlement footprint clipped to tile ROI.

/test_<YYYY>_<mon> (test phase VNP46A1 observations for joint model testing).

                                   /dataVNP

                                   /dataVNPmask

/labels_joint – labels or detections of jointly trained joint detector on dataVNP. Follows same structure as labels_reg

vnp46a1_thermal_anomaly/anomaly_det

(regional anomaly detections used for generating initial detections of training data )

                   /regional/hHHvVV

/labels_ae : det_ae_<YYYYDOY>.tif autoencoder detections: background (0), detection(>0)

/labels_vae : det_vae_<YYYYDOY>.tif VAE detections: background (0), detection(>0)

/labels_dnb : det_dnb_<YYYYDOY>.tif DNB detections: background(0),  detection (1)

/labels_m11 : det_m11_<YYYYDOY>.tif M-11 detections: background (0), detection(1)

dataVNP and dataVNPmask for h08v06, h08v06_w and h20v10 are the same as vnp46a1_thermal_anomaly/classification/regional/hHHvVV/training_iter1/dataVNP and vnp46a1_thermal_anomaly/classification/regional/hHHvVV/training_iter1/dataVNPmask

/dataVNP (only for h05v05 as anomaly detector was run across the tile)

/dataVNPmask (only for h05v05 as anomaly detector was run across the tile)

Regional training bounds and days

References:

Román, M.O., Wang, Z., Sun, Q., Kalb, V., Miller, S.D., Molthan, A., Schultz, L., Bell, J., Stokes, E.C., Pandey, B. and Seto, K.C., 2018. NASA's Black Marble nighttime lights product suite. Remote Sensing of Environment, 210, pp.113-143.

Román, M.O., Wang, Z., Shrestha, R., Yao, T. and Kalb, V., 2019. Black marble user guide version 1.0. NASA: Washington, DC, USA.

Chakraborty, S., Oda, T., Kalb, V.L., Wang, Z. and Román, M.O., 2023. Potentially underestimated gas flaring activities—a new approach to detect combustion using machine learning and NASA’s Black Marble product suite. Environmental Research Letters, 18(3), p.035001.