Lecture 8
Importing the data into Galaxy
Click the new-history icon at the top of the history panel.
Click on the icon (History options) on the top of the history panel
Select the option Create New from the menu
Renaming a history
Click on (Edit) next to the history name (which by default is “Unnamed history”)
Type the new name
Click on Save
Import the sample FASTQ files to your history
https://zenodo.org/record/800651/files/F3D0_R1.fastq
https://zenodo.org/record/800651/files/F3D0_R2.fastq
https://zenodo.org/record/800651/files/F3D141_R1.fastq
https://zenodo.org/record/800651/files/F3D141_R2.fastq
https://zenodo.org/record/800651/files/F3D142_R1.fastq
https://zenodo.org/record/800651/files/F3D142_R2.fastq
https://zenodo.org/record/800651/files/F3D143_R1.fastq
https://zenodo.org/record/800651/files/F3D143_R2.fastq
https://zenodo.org/record/800651/files/F3D144_R1.fastq
https://zenodo.org/record/800651/files/F3D144_R2.fastq
https://zenodo.org/record/800651/files/F3D145_R1.fastq
https://zenodo.org/record/800651/files/F3D145_R2.fastq
https://zenodo.org/record/800651/files/F3D146_R1.fastq
https://zenodo.org/record/800651/files/F3D146_R2.fastq
https://zenodo.org/record/800651/files/F3D147_R1.fastq
https://zenodo.org/record/800651/files/F3D147_R2.fastq
https://zenodo.org/record/800651/files/F3D148_R1.fastq
https://zenodo.org/record/800651/files/F3D148_R2.fastq
https://zenodo.org/record/800651/files/F3D149_R1.fastq
https://zenodo.org/record/800651/files/F3D149_R2.fastq
https://zenodo.org/record/800651/files/F3D150_R1.fastq
https://zenodo.org/record/800651/files/F3D150_R2.fastq
Import Reference DataImport the following reference datasets
https://zenodo.org/record/800651/files/HMP_MOCK.v35.fasta
https://zenodo.org/record/800651/files/silva.v4.fasta
https://zenodo.org/record/800651/files/trainset9_032012.pds.fasta
https://zenodo.org/record/800651/files/trainset9_032012.pds.tax
https://zenodo.org/record/800651/files/mouse.dpw.metadata
Organizing our data into a paired collection
Click on the checkmark icon param-check at top of your history.
Select all the FASTQ files
Tip: type fastq in the search bar at the top of your history to filter only the FASTQ files;
Click on All selected
Select Build List of Dataset Pairs from the dropdown menu
Click on Auto-pair to create the suggested pairs.
Or click on “Pair these datasets” manually for every pair that looks correct.
Name your collection at the bottom right of the screen
You can pick whatever name makes sense to you
Click the Create Collection button.
A new dataset collection item will now appear in your history
Quality Control
Make.contigs with the following parameters
“Way to provide files”: Multiple pairs - Combo mode
“Fastq pairs”: the collection you just created
Leave all other parameters to the default settings
Data Cleaning
As the next step, we want to improve the quality of our data.
Summary.seqs with the following parameters
“fasta”: the trim.contigs.fasta file created by Make.contigs tool
“Output logfile?”: yes
Start End NBases Ambigs Polymer NumSeqs
Minimum: 1 249 249 0 3 1
2.5%-tile: 1 252 252 0 3 624
25%-tile: 1 252 252 0 4 6235
Median: 1 252 252 0 4 12470
75%-tile: 1 253 253 0 5 18705
97.5%-tile: 1 253 253 6 6 24316
Maximum: 1 502 502 247 243 24939
Mean: 1 252.74 252.74 0.72128 4.46445
# of Seqs: 24939
The Start and End columns tell us that the majority of reads aligned between positions 1 and 252, which is what we expect to find given the reference file we used. However, some reads align to very different positions, which could indicate insertions or deletions at the terminal ends of the alignments or other complicating factors
We do this data cleaning using the Screen.seqs tool, which removes
sequences with ambiguous bases (maxambig) and
contigs longer than a given threshold (maxlength).
Screen.seqs with the following parameters
“fasta”: the trim.contigs.fasta file created by Make.contigs tool
“group”: the group file created in the Make.contigs tool step
“maxlength”: 275
“maxambig”: 0
Optimize files for computation
Microbiome samples typically contain a large numbers of the same organism, and therefore we expect to find many identical sequences in our data. In order to speed up computation, we first determine the unique reads, and then record how many times each of these different reads was observed in the original dataset. We do this by using the Unique.seqs tool.
Unique.seqs with the following parameters
“fasta”: the good.fasta output from Screen.seqs tool
“output format”: Name File
This step has greatly reduced the size of our sequence file; not only will this
speed up further computational steps, it will also greatly reduce the amount
of disk space (and your Galaxy quota) needed to store all the intermediate files generated during this analysis. This Unique.seqs tool created two files, one is a FASTA file containing only the unique sequences, and the second is a so-called names file. This names file consists of two columns, the first contains the sequence names for each of the unique sequences, and the second column contains all other sequence names that are identical to the representative sequence in the first column.
To recap, we now have the following files:
To further reduce file sizes and streamline analysis, we can use the Count.seqs tool to combine the group file and the names file into a single count table.
Count.seqs with the following parameters
“name”: the names output from Unique.seqs tool
“Use a Group file”: yes
“group”: the group file we created using the Screen.seqs
Count.seqs tool summarizes the number of times each unique sequence was observed across each of the samples
From now on, we will only work with the set of unique sequences, but it’s important to remember that these represent a larger number of total sequences, which we keep track of in the count table.
In the following we will use the unique sequences together with the count table as input to tools instead of the complete set of sequences. If this is done for the Summary.seqs tool tool it will report both the number of unique representative sequences as well as the total sequences they represent.
Sequence Alignment
Alignments are a powerful way to compare related DNA or protein sequences.
They can be used to capture various facts about the sequences aligned, such as
common evolutionary descent or common structural function. We take the general view that the alignment of letters from two or multiple sequences represents the hypothesis that they are descended from a common ancestral sequence.
Sequence comparison is a crucial aspect of bioinformatics analysis that involves comparing newly determined biological sequences with previously known sequences stored in databases.
Sequence alignment is considered the most essential step in comparing biological sequences. Sequence alignment arranges two or more nucleotide or amino acid sequences to identify regions of similarity between the sequences. These regions of similarity are helpful in understanding the functional, structural, and evolutionary relationships between the sequences.
Two commonly used sequence alignment algorithms are global alignment and local alignment.
Global alignment: Global alignment is a method of comparing two sequences, which aligns the entire length of the sequences by maximizing the overall similarity. This method is used when comparing sequences that are of the same length.
Local alignment: In local alignment, instead of attempting to align the entire length of the sequences, only the regions with the highest density of matches are aligned. This is useful for identifying short conserved regions in protein or nucleotide sequences.
Pairwise Alignment
Multiple Sequence Alignment
Align.seqs with the following parameters
“fasta”: the fasta output from Unique.seqs tool
“reference”: silva.v4.fasta reference file from your history
There are different alignment characters in this output:
.: terminal gap character (before the first or after the last base in our query sequence)
-: gap character within the query sequence
Summary.seqs with the following parameters:
“fasta”: the align output from Align.seqs tool
“count”: count_table output from Count.seqs tool
“Output logfile?”: yes
More Data Cleaning
1. Remove any reads not overlapping the region V4 region (position 1 to 252) using Screen.seqs tool.
2. Remove any overhang on either end of the V4 region to ensure our sequences overlap only the V4 region, using Filter.seqs tool.
3. Clean our alignment file by removing any columns that have a gap character (-, or . for terminal gaps) at that position in every sequence (also using Filter.seqs tool).
4. Remove redundancy in the aligned sequences that might have been introduced by filtering columns by running Unique.seqs once more.
5. Group near-identical sequences together with Pre.cluster tool. Sequences that only differ by one or two bases at this point are likely to represent sequencing errors rather than true biological variation, so we will cluster such sequences together.
Screen.seqs with the following parameters
“fasta”: the aligned fasta file from Align.seqs tool
“start”: 1
“end”: 252
“maxhomop”: 8
“count”: the count table file from Count.seqs
Filter.seqs with the following parameters
“fasta”: good.fasta output from the latest Screen.seqs tool
“vertical”: yes
“trump”: .
“Output logfile”: yes
Unique.seqs with the following parameters
“fasta”: the filtered fasta output from Filter.seqs tool
“name file or count table”: the count table from the last Screen.seqs
Import the sample FASTQ files to your history
https://zenodo.org/record/815875/files/SRR531818_pampa.fasta
https://zenodo.org/record/815875/files/SRR651839_anguil.fasta
https://zenodo.org/record/800651/files/silva.v4.fasta
https://zenodo.org/record/815875/files/trainset16_022016.pds.fasta
https://zenodo.org/record/815875/files/trainset16_022016.pds.tax
Merge.files tool with the following parameters
Make.group tool with the following parameters
Unique.seqs tool with the following parameters
“fasta” to the merged fasta file
“output format” to
Count.seqs tool with the following parameters
“name” to the name file from Unique.seqs
“Use a group file” to yes
“group” to the group file from Make.group
Summary.seqs tool with the following parameters
“fasta” parameter to the fasta from Unique.seqs
“count” to count table from Count.seqs
“output logfile?” to yes
Screen.seqs tool with the following parameters
“fasta” to the fasta file from Unique.seqs
“minlength” parameter to 225
“maxlength” parameter to 275
“maxambig” parameter to 0
“maxhomop” parameter to 8
• “count” to the count file from Count.seqs
Using 1 processors.
Start End NBases Ambigs Polymer NumSeqs
Minimum: 1 80 80 0 3 1
2.5%-tile: 1 104 104 0 3 501
25%-tile: 1 242 242 0 4 5001
Median: 1 245 245 0 4 10001
75%-tile: 1 245 245 0 4 15001
97.5%-tile: 1 247 247 0 6 19501
Maximum: 1 275 275 2 31 20000
Mean: Align.seqs 1 237.519 237.519 0.00495 4.24965
# of unique seqs: 19502
total # of seqs: 20000
Align.seqs tool with the following parameters
“fasta” to the good.fasta output from Screen.seqs
“Select Reference Template from” to Your history
“reference” to the silva.v4.fasta reference file
“flip” to Yes
Summary.seqs tool with the following parameters
“fasta” parameter to the aligned output from Align.seqs
“count” parameter to count_table output from Screen.seqs
Screen.seqs tool with the following parameters