# Please cite: Guo, F., Jiang, W., 2014. Single Particle Cryo-electron Microscopy and 3-D Reconstruction of Viruses, in: Kuo, J. (Ed.), Electron Microscopy, Methods in Molecular Biology. Humana Press[a][b], Totowa, NJ, pp. 401–443. doi:10.1007/978-1-62703-776-1

# You will need jspr.2017-7-20.tar.gz for the following examples.

# example JSPR commands for a ribosome dataset (EMPIAR-10107, 3.25Å) dataset originally processed using Relion software.

# the lines in blue are actual commands. You would need to adapt to your own data (sampling, mask sizes, symmetry)

# this shared Google Doc file allows anyone to add comments (using Insert -> Comment). If you cannot see the comments or Google Doc does not allow you to add comments, make sure that Suggesting instead of Viewing mode is used. Please ask question by adding your own comments at the right places.  However, please do NOT edit the content of document. More general questions can be asked/answered through the jspr Google group.

# download data from EMPIAR into ../data directory

mkdir ../data

wget -c -r -nH --cut-dirs=6 --directory-prefix ../data ftp://ftp.ebi.ac.uk/pub/databases/empiar/archive/10107/data[c][d]

# fix the filename path in the provided star file[e]

images2star.py ../data/shiny_new.star[f] ../data/particles.star --newdir ../data/shinyparticles

# split the half sets using the same Relion halves (based on the rlnRandomSubset parameter)

images2star.py ../data/particles.star particles.1.star --select rlnRandomSubset 1

images2star.py ../data/particles.star particles.2.star --select rlnRandomSubset 2

# convert Relion star file to lst file used by jspr. If the structure has icosahedral symmetry, you should use I4 sym in Relion if you want reuse the euler angles in jspr

# Euler, Center, and CTF parameters are converted and included in the output lst file

# jspr can directly use any binary image format supported by EMAN2 including hdf5, mrcs, etc.

images2lst.py particles.1.star particles.1.lst

# 170309 particles

images2lst.py particles.2.star particles.2.lst

# 170714 particles

# total 341023 particles

# NOTE: reverse direction conversion (lst -> star) can also be done easily

# it will convert Euler, Center, and CTF parameters

# it will also convert particle image format if necessary (for example, hdf files used in the input lst file → mrcs files in the output star file)

# this conversion requires CTF parameters that are either in the lst file or stored in the image header of the particle files

# this reverse direction conversion shown here is information only. It is not needed for this example.

images2star.py input.lst output.star --verbose 1

# make sure the .mparm file (required by EMAN1 runpar program) and mpi_nodefile file (in OpenMPI hostfile format, required by jspr j3dr program) are provided in current folder[g][h][i][j]

# verify conversion using jspr to build 3D maps without refinement

jspr.py particles.1.lst particles.2.lst  --iters 0 --iter0 0 --sym c1 --initEuler current --eotest 1 --apix 1.34 --phaseCorrected 0 --mergeDataSets 0[k]

--reconstructor j3dr --reconstructorOption "--preprocess normalize.mask.circlemean:mask=1:radius=165[l]:masksoft=-2 --postprocess normalize.mask.circlemean:mask=1:radius=165:masksoft=-2 --reconstructor fourier:mode=gauss_2:ctfweight=1" --normHand 0 --post3dOption "--cpu 4 --process mask.auto3d[m]:nshells=5:nshellsgauss=5:nmaxseed=240:threshold=2[n]" --verbose 3 --cpus 128 --make3dcpus 8[o] --parallelMethod runpar

#                 Map 1: threed.0.set-0.mrc[p]

#                 Map 2: threed.0.set-1.mrc

#                 Resolution at fsc=0.5:   6.04 Angstrom (fit: 6.07 Angstrom)

#                 Resolution at fsc=0.333: 5.4 Angstrom (fit: 5.47 Angstrom)

#                 Resolution at fsc=0.25: 5.16 Angstrom (fit: 5.18 Angstrom)

#                 Resolution at fsc=0.143: 4.81 Angstrom (fit: 4.74 Angstrom)

#                 Map 1: threed.0a.set-0.mrc[q][r]

#                 Map 2: threed.0a.set-1.mrc

#                 Resolution at fsc=0.5:   4.85 Angstrom (fit: 4.85 Angstrom)

#                 Resolution at fsc=0.333: 4.64 Angstrom (fit: 4.42 Angstrom)

#                 Resolution at fsc=0.25: 4.53 Angstrom (fit: 4.18 Angstrom)

#                 Resolution at fsc=0.143: 4.34 Angstrom (fit: 3.81 Angstrom)

# a dip at 4.2A and then a peak after that (what caused the dip?)

# refine using jspr[s][t]

jspr.py particles.1.lst particles.2.lst[u]  --iters 3 --iter0 0 --sym c1 --batchSize 50 [v][w]--initEuler current --eotest 5[x][y] --apix 1.34  --mergeDataSets 1 --phaseCorrected 0 --aligner jalign --alignerOption "--oversample 0 --batchsize 1 --aligner refineMicrographDefocus:batchsize=1:useOrigDefocus=0:precision=0.001:stepdefocus=0.02:defocusrange=0.2:maskradius=125:masksoft=-2:discardMap=0 --aligner refineAstigmatism:batchsize=1:dfdiffrange=0.1:stepdfdiff=0.02:precision=0.001:discardMap=0:maskradius=125:masksoft=-2:reset=0 --aligner refineScale:batchsize=1:flipphase=1:maskradius=125:masksoft=-2:precision=0.0001:scalerange=0.01:stepscale=0.001 --aligner refineAnisoScale:batchsize=1:asamprange=0.01:stepasamp=0.002:flipphase=1:precision=0.0001:maskradius=125:masksoft=-2 --aligner refineOrientationCenter:flipphase=1:maskradius=125[z]:masksoft=-2:steprot=1:stepxy=1:precision=0.01 --preprocess normalize.mask.circlemean:mask=0:radius=165 --cmp phaseResRange:ampweight=0:ctfampweight=1:lowRes=30:highRes=3:abs=0:weightFile=fsc.%litera.set-0-1.txt.fit" --reconstructor j3dr --reconstructorOption "--preprocess normalize.mask.circlemean:mask=1:radius=165:masksoft=-2 --postprocess normalize.mask.circlemean:mask=1:radius=165:masksoft=-2 --reconstructor fourier:mode=gauss_2:ctfweight=1 --weight scoreSigma:w=2" --normHand 0 --post3dOption "--cpu 4 --process mask.auto3d:nshells=5:nshellsgauss=5:nmaxseed=240:threshold=2" --verbose 3 --cpus 128[aa] --make3dcpus 8 --parallelMethod runpar

#                 Map 1: threed.1.set-0.mrc

#                 Map 2: threed.1.set-1.mrc

#                 Resolution at fsc=0.5:   4.52 Angstrom (fit: 4.77 Angstrom)

#                 Resolution at fsc=0.333: 4.18 Angstrom (fit: 4.27 Angstrom)

#                 Resolution at fsc=0.25: 4.02 Angstrom (fit: 4.02 Angstrom)

#                 Resolution at fsc=0.143: 3.79 Angstrom (fit: 3.66 Angstrom)

#                 Map 1: threed.1a.set-0.mrc

#                 Map 2: threed.1a.set-1.mrc

#                 Resolution at fsc=0.5:   3.85 Angstrom (fit: 3.89 Angstrom)

#                 Resolution at fsc=0.333: 3.63 Angstrom (fit: 3.64 Angstrom)

#                 Resolution at fsc=0.25: 3.52 Angstrom (fit: 3.51 Angstrom)

#                 Resolution at fsc=0.143: 3.33 Angstrom (fit: 3.31 Angstrom)

#                 Map 1: threed.2.set-0.mrc

#                 Map 2: threed.2.set-1.mrc

#                 Resolution at fsc=0.5:   4.11 Angstrom (fit: 4.29 Angstrom)

#                 Resolution at fsc=0.333: 3.77 Angstrom (fit: 3.82 Angstrom)

#                 Resolution at fsc=0.25: 3.61 Angstrom (fit: 3.59 Angstrom)

#                 Resolution at fsc=0.143: 3.35 Angstrom (fit: 3.26 Angstrom)

#                 Map 1: threed.2a.set-0.mrc

#                 Map 2: threed.2a.set-1.mrc

#                 Resolution at fsc=0.5:   3.46 Angstrom (fit: 3.48 Angstrom)

#                 Resolution at fsc=0.333: 3.22 Angstrom (fit: 3.23 Angstrom)

#                 Resolution at fsc=0.25: 3.14 Angstrom (fit: 3.1 Angstrom)

#                 Resolution at fsc=0.143: 3.04 Angstrom (fit: 2.9 Angstrom)

#                 Map 1: threed.3.set-0.mrc

#                 Map 2: threed.3.set-1.mrc

#                 Resolution at fsc=0.5:   4 Angstrom (fit: 4.18 Angstrom)

#                 Resolution at fsc=0.333: 3.66 Angstrom (fit: 3.7 Angstrom)

#                 Resolution at fsc=0.25: 3.48 Angstrom (fit: 3.47 Angstrom)

#                 Resolution at fsc=0.143: 3.24 Angstrom (fit: 3.13 Angstrom)

#                 Map 1: threed.3a.set-0.mrc

#                 Map 2: threed.3a.set-1.mrc

#                 Resolution at fsc=0.5:   3.35 Angstrom (fit: 3.41 Angstrom)

#                 Resolution at fsc=0.333: 3.16 Angstrom (fit: 3.17 Angstrom)

#                 Resolution at fsc=0.25: 3.1 Angstrom (fit: 3.04 Angstrom)

#                 Resolution at fsc=0.143: 3 Angstrom (fit: 2.85 Angstrom)

# much improved maps and FSC curves

# map built from all particles: threed.3.all-sets.mrc (unmasked, unsharpened)

# sharpen the final map for structrual analysis: first use an inverse B-factor to sharpen the map to a “perfect” map (i.e. the power spectra beyond 10A are approximately constant), then use the FSC curve to dampen the power spectra to reduce high frequency noises

# the sharpened map is then background normalizied: set background mean to zero and sigma to one

e2proc3d.py[ab][ac] threed.3.all-sets.mrc threed.3a.all-sets.mrc --process filter.fourier:bfactor=-120:cutoffres=3 --calcsf x.sf[ad] --process filter.file:file=fsc.3a.set-0-1.txt.fit[ae] --process normalize.mask.circlemean[af]:mask=1:radius=165:masksoft=-2

FSC curves of unmasked maps refined using jspr

FSC curves of unmasked half maps by CryoSPARC, Relion, and jspr

#####################################################################################################################

#####################################################################################################################

# Relion results:

# computer FSC curve of unmasked maps

relion_reconstruct --i particles.1.star --o relion.1.mrc --ctf  --sym c1 --j 8

relion_reconstruct --i particles.2.star --o relion.2.mrc --ctf  --sym c1 --j 8

e2proc3d.py relion.1.mrc relion.fsc --apix 1.34 --calcfsc relion.2.mrc

#                 Map 1: relion.1.mrc

#                 Map 2: relion.2.mrc

#                 Resolution at fsc=0.5:   4.7 Angstrom (fit: 4.95 Angstrom)

#                 Resolution at fsc=0.333: 4.23 Angstrom (fit: 4.39 Angstrom)

#                 Resolution at fsc=0.25: 4.05 Angstrom (fit: 4.11 Angstrom)

#                 Resolution at fsc=0.143: 3.78 Angstrom (fit: 3.72 Angstrom)

#####################################################################################################################

#####################################################################################################################

# cryosparc results: experiment-331

#        particles.star as input; ab initio model; refine

#         reported resolution: 3.15A (after tight masking)

# compute FSC curve of maps with corner masked (spherical mask)

e2proc3d.py cryosparc.experiment-331.e.mrc cryosparc.experiment-331.fsc --apix 1.34 --calcfsc cryosparc.experiment-331.o.mrc

#                 Map 1: cryosparc.experiment-331.e.mrc

#                 Map 2: cryosparc.experiment-331.o.mrc

#                 Resolution at fsc=0.5:   4.52 Angstrom (fit: 4.79 Angstrom)

#                 Resolution at fsc=0.333: 4.14 Angstrom (fit: 4.26 Angstrom)

#                 Resolution at fsc=0.25: 3.99 Angstrom (fit: 4.01 Angstrom)

#                 Resolution at fsc=0.143: 3.74 Angstrom (fit: 3.64 Angstrom)

[a]Different from the paper, in this example, reconstructor does not include filtrt.ctf:type=1. Any special reason? is it default to be added so was not included in script?

[b]jspr will automatically add it if it is not already included. It will not add it if it is already added by the user.

[c]Do you still have this dataset in your folder?

[d]No. You can download it using globus

[e]If the particle has icosahedral symmetry, make sure the symmetry used in Relion is I4 which is consistent with jspr/EMAN2 icos symmetry (5-fold along Z axis and 2-fold along Y axis).

[f]the input star file should be a particle star file with "rlnImageName" column and other columns for CTF parameters

[g]Our hpc clusters use slurm job management system. How can I make jspr.py work with slurm queue? Thanks!

[h]I don't have access to slurm so I don't know how to do it at this moment.

If slum provides the PBS_NODEFILE environment variable as PBS does, then you can just include the jspr command in a slum job file and submit it. jspr checks for the PBS_NODEFILE variable and uses it automatically if the PBS_NODEFILE variable is found

[i]I have access to slurm. All you need do is to output all the commands into a single file. You may have to modify jspr.py a little bit. Then follow the tutorial of srun.

[j]SLURM is now supported by the new version that can be downloaded from our website. No change to the jspr.py options as jspr.py will automatically detect the queue and submit the jobs accordingly

[k]0 (default)

1: will merge datasets and build a map from the merged dataset. Use this at the end of refinement to get a map as final result for structural analysis

[l]Unit: pixel

Here, 165 is half of the box size (330x330 pixels). This is probably larger than necessary. One can use a radius:

(max radius in Angstrom + 50 Angstrom)/apix

[m]this processor will mask the maps with an adaptive mask instead of a spherical mask. See this EMAN2 FAQ for more details:

http://blake.bcm.edu/emanwiki/EMAN2/FAQ/AutoMask

FSC curve of the automasked maps is in general used in publication to report the final resolution. However, depending on how tight the mask is, the FSC curve (and thus the resolution number) will be different.

The adaptive masking plays similar role as that of Relion postprocessing and CryoSPARC tightmasking.

Note that sharpening (normally done in Fourier space) only changes the visual space visual details of the maps. It does not affect FSC curve.

See the example command below for sharpening the final map built from all particles.

[n]threshold for automasking.

use a smaller value if the structure is cut

use a larger value if the background is not fully removed

[o]# of cores used for 3D reconstruction. The cores can be on different nodes specified in the mpi_nodefile file. Only cores on the local computer will be used if mpi_nodefile file is not provided

[p]unmasked, unsharpened map generated by the reconstruction program.

an example command for sharpening is shown in next page

[q]automasked, unsharpened map generated by the "--post3dOption" option.

add savemask=mask.mrc to the mask.auto3d processor to save the mask to a mrc file

[r]you can use your favorite graphic program to view the saved mask map

[s]Does this need an initial model? If so, is it no need to put in the script? Or, it likes EMAN that has a default name like threed.0a.mrc, no need to add into the script?

[t]It will look for threed.*a.mrc and use them if found. Otherwise, it will generate threed.0a.mrc using the random model approach

[u]you can include any number of datasets (n>=1) here and the datasets will be independently refined without mixing.

The results of one iteration (i.e. orientation-center.*.lst files) will be used as input for next iteration

[v]How to choose this parameter? Thank you!

[w]The exact value is not critical. You might increase it if your particle image size is small and each 2D alignment task only takes very short time.

[x]And this?

[y]generate FSC curves for the last n iterations

[z]Unit: pixel

This value should be about the same as the maximal radius of the particle.

In jspr, a moving mask is used: the mask is always centered around the current particle center instead of image box center

[aa]# of cores used for 2D alignment. The cores can be on different nodes specified by using the EMAN runpar method. Only cores on the local computer will be used if the mparm file is not properly setup for runpar

[ab]Can one also use a MTF profile to take the DQE into account similar to Relion 2?

[ac]Not directly. You can create a file containing 1/MTF and then use filter.file processor.

[ad]x.sf will be a text that stores the power spectra of the sharpened map. Plot this file and check if the power spectra at high resolutions (for example, 10A and higher) are approximately horizontal.

[ae]the fsc curves are generated by jspr when --eotest option is set

[af]if an inner mask is needed, use the following processor:

--process mask.inner:radius=<n>:masksoft=<n>