Proposed data format

	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S	T	U	V	W	X	Y	Z	AA	AB	AC
1	`	metadata						spike time related				voltage-related							quality metrics					Layer-related		ground truth / predicted labels
2		lab_id	line	dataset_id	neuron_relative_id	neuron_id	sampling_rate	sane_periods	spike_indices	optostims	sane_spikes	mean_waveform_preprocessed (across channels, 200 um radius)	channel_ids	channelmap	raw_waveforms	scaling_factor	raw_voltage_snippet	voltage_snippet_start_index	channel_noise_std	peakchan_SNR	acrosschan_SNR	amplitudes	fn_fp_filtered_spikes	phyllum_layer	human_layer	expert_label	ground_truth_label	ground_truth_source	mli_cluster

3	dtype	str	str	str	int	int	int	array of [t1,t2] values	array of uint32	array of float64	boolean array (0 or 1, uint8)	array of float32	array of ints	array of floats	array of int16s	float	array of int16	int	array of floats	float	float	array	boolean array (0 or 1, uint8, length of spike indices) (or null)	string	string	string	string	string	string
4	shape	1	1	1	1	1	1	"all" or (n_periods, 2)	(n_spikes,)	(n_stims, 2) or None	(n_spikes,)	(n_channels, n_samples,)	n_channels	(n_channels,2)	(n_waveforms, n_channels, n_samples)	1	(7, 30*sampling_rate)	1	(n_channels,)	1	1	(n_spikes,)	(n_spikes,)	1	1	1	1	1	1
5	unit	alphanumeric	alphanumeric	alphanumeric	indices	indices	Hertz (samples/second)	seconds	indices	seconds	boolean mask	microvolts (uV)	indices	micrometers (um)	bits	uV/bits	bits	indices	MAD	ratio	ratio	arbitrary units (kilosort)	boolean mask	alphanumeric	alphanumeric	alphanumeric	alphanumeric	alphanumeric	alphanumeric
6	comments	"hausser" "medina" "hull" "lisberger"	"monkey" "mouse_pcp2" "mouse_thy1" "mouse_math1" "mouse_nos1" "mouse_glyt2" "mouse_ckit"	subject/session/etc. (lab specific)	unit relative id within its dataset, for mouse recordings.	unit absolute id across datasets (within lab database)	typically 30000 for Neuropixels, 40000 for plexon	list of time windows in seconds, containing spikes to consider (typically, periods with low drift/high quality)	samples since start of recording	1st column: stim onset, 2nd column: stim offset	By default, all True for spikes in 'periods'. Shove-it-all boolean mask for spikes to consider 'good'.	n_channels = number of channels to reach a radius of 200um (centered on the neuron's peak channel) n_samples = 6ms (centered) * sampling_rate	channel_ids (numbers, 0 indexed)	n_channels to match mean_waveform_preprocessed (200 um) 2 columns correspond to 1. xPos, 2. yPos. Positions are in um.	n_waveforms = min(total_n_spikes, 1000) n_channels to match mean_waveform_preprocessed (200 um) n_samples = 6ms (centered) * sampling_rate	bit_uV_conv_factor	a snippet of raw voltage centered on a period with plenty of spikes (use fp_fn_giltered_spikes)	starting time point of the voltage snippet	std of noise on each of n_channels in mean_waveform_preprocessed. A robust estimator of the std is in the meeting minutes	SNR on peak channel, as the waveform amplitude (max - min) divided by 1x the std on this channel (as defined in channel_noise_std)	SNR across 7 channels centered on the peak channel, computed as follow:	Will allow to compute FN rate	spikes occuring in periods which have BOTH a low enough false positive and false negative rate	["GCL", "PCL", "ML", "unknown", "not_cortex", ""]	["GrL", "PcL", "MoL", "unknown", "out of cortex"]	["PkC_ss", "PkC_cs", "MLI", "MFB", "GoC", "GrC", "unlabelled"]	["PkC_ss", "PkC_cs", "MLI", "MFB", "GoC", "GrC", "unlabelled"]	["optotagged", "CSxSS"]	["MLI_1", "MLI_2"]
7	comments				in the mouse database, neurons can be accessed at root/{lab_id}_neuron_{unit_absolute_id} or at root/datasets/{dataset_id}/{neuron_relative_id} in the monkey database, only at root/{lab_id}_neuron_{unit_absolute_id}							Taken from 300Hz High pass filtered binary file (non-whitened), 1st order Butterworth, causal. Aligned (drift-shift-matched) with Maxime's code.		Edge effects: just don't include, no need to nan-populate	a sample of maximum 10_000 raw waveforms from this unit, to allow inspection of the quality of single waveforms.	scaling factor to convert voltage data into uV (multiply raw_waveforms, in bits, by this value, in uV/bit, to get uV)	7 channels centered on the peak channel		Explanation here (relationship between MAD and std: https://en.wikipedia.org/wiki/Median_absolute_deviation		Take 1000 waveforms over 7 channels, average them. Compute the 1000 dot products between the mean waveform and the individual waveforms (S), and 1000 dot products between the mean waveform and 1000 random snippets of voltage on the same channel set (N). Compute the ratio (mean(S) - mean(N)) / std(N).	Scaling factor of template onto spike across channels (amplitudes.npy file)	Optional field. If this should be computed post-hoc, this field should be set to "None"/NULL
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100