Ucsbgalaxy - User contributions [en]

Talk:Newbler parameter '-a' (minimum contig length) not working

2011-11-23T00:56:23Z

Pankey:

11/22/11: Ran Newbler via command line now to see if -a works outside of Galaxy interface:

*Method
#copied Newbler command from Galaxy run, just changed the filenames for local use, deleted _mcf, -vs, -fe and -fi options
runAssembly -o uro-assembled -cpu 6 -rip -e 0 -mi 90 -ml 40 -minlen 20 -info -qo -ss 12 -sl 16 -sc 1 -ais 2 -a 100 -vt ../../trimfileforUrPHO.fas -cdna -ig 500 -it 100 -icc 100 -icl 3 ../UrEy.sff ../PHO_F5BN4WZ03.sff

*Results from command line run
454Allcontigs.fna has many more contigs (from 3700 with Galaxy's Newbler to 4094).
752/4094 contigs are less than 100 in length, so this -a problem must be a Newbler-wide bug and not due to the wrapper.

Talk:Newbler parameter '-a' (minimum contig length) not working

2011-11-23T00:55:56Z

Pankey:

11/22/11: Am trying Newbler via command line now to see if -a works outside of Galaxy interface:

*Method
#copied Newbler command from Galaxy run, just changed the filenames for local use, deleted _mcf, -vs, -fe and -fi options
runAssembly -o uro-assembled -cpu 6 -rip -e 0 -mi 90 -ml 40 -minlen 20 -info -qo -ss 12 -sl 16 -sc 1 -ais 2 -a 100 -vt ../../trimfileforUrPHO.fas -cdna -ig 500 -it 100 -icc 100 -icl 3 ../UrEy.sff ../PHO_F5BN4WZ03.sff

*Results from command line run
454Allcontigs.fna has many more contigs (from 3700 with Galaxy's Newbler to 4094).
752/4094 contigs are less than 100 in length, so this -a problem must be a Newbler-wide bug and not due to the wrapper.

Talk:Newbler parameter '-a' (minimum contig length) not working

2011-11-23T00:42:41Z

Pankey:

11/22/11: Am trying Newbler via command line now to see if -a works outside of Galaxy interface:

# copied Newbler command from Galaxy run, just changed the filenames for local use, deleted _mcf, -vs, -fe and -fi options
runAssembly -o uro-assembled -cpu 6 -rip -e 0 -mi 90 -ml 40 -minlen 20 -info -qo -ss 12 -sl 16 -sc 1 -ais 2 -a 100 -vt ../../trimfileforUrPHO.fas -cdna -ig 500 -it 100 -icc 100 -icl 3 ../UrEy.sff ../PHO_F5BN4WZ03.sff

Talk:Newbler parameter '-a' (minimum contig length) not working

2011-11-22T23:54:51Z

Pankey: Created page with "11/22/11: Am trying Newbler via command line now to see if -a works outside of Galaxy interface..."

11/22/11: Am trying Newbler via command line now to see if -a works outside of Galaxy interface...

Newbler parameter '-a' (minimum contig length) not working

2011-11-22T23:53:56Z

Pankey: Created page with "Newbler2.6 implemented in Galaxy offers the optional parameter (-a) to restrict contigs below a minimum length from showing up in AllContigs.fna. Even when set to 100, I find ma…"

Newbler2.6 implemented in Galaxy offers the optional parameter (-a) to restrict contigs below a minimum length from showing up in AllContigs.fna. Even when set to 100, I find many contigs smaller, including many contigs with length=1.

Galaxy Problems to address

2011-11-22T23:51:07Z

Pankey:

==Outstanding==
* [[Upgrade FastQC Tool]]
* [['Memory leak' type behavior]]
* [[Install cron jobs to clear deleted datasets]]
* [[Barcode splitter output]]
* [[blast2go not working]]
* [[FTP connection refused]]
* [[Version tracking]]
* [[ Newbler tool doesn't assemble FASTQ files as it claims to]]
* [[Newbler parameter '-a' (minimum contig length) not working]]
==Resolved==

Restarting Galaxy

2011-11-21T18:53:55Z

Pankey:

# [[Connect to knot using the galaxy user login]]
# Find the galaxy processs that are running by typing:
ps aux | grep galaxy
# Identify the process numbers for 3 Galaxy processes, which will change every time Galaxy is restarted.

For example this line:
galaxy 11638 0.0 0.0 63856 988 ? S Nov10 0:00 /bin/sh ./run.sh --reload

indicates the process has a number of 11638.

The 3 processes look like:
## /bin/sh ./run.sh --reload
## python ./scripts/paster.py serve universe_wsgi.ini --reload
## /home/galaxy/bin/python ./scripts/paster.py serve universe_wsgi.ini --reload

Next, kill the three processes with this command:
kill -9 <ID1> <ID2> <ID3>
Then change to the galaxy-dist directory:
cd /home/galaxy/galaxy-dist
Then restart Galaxy by executing the run.sh script:
nohup ./run.sh --reload &

Alternate method to restart Galaxy:
./run.sh --daemon --reload

Restarting Galaxy

2011-11-21T18:53:21Z

Pankey:

Restarting Galaxy

2011-11-21T18:52:20Z

Pankey:

# [[Connect to knot using the galaxy user login]]
# Find the galaxy processs that are running by typing:
ps aux | grep galaxy
# Identify the process numbers for 3 Galaxy processes, which will change every time Galaxy is restarted.

For example this line:
galaxy 11638 0.0 0.0 63856 988 ? S Nov10 0:00 /bin/sh ./run.sh --reload

indicates the process has a number of 11638.

The 3 processes look like:
## /bin/sh ./run.sh --reload
## python ./scripts/paster.py serve universe_wsgi.ini --reload
## /home/galaxy/bin/python ./scripts/paster.py serve universe_wsgi.ini --reload

# Next, kill the three processes with this command:
kill -9 <ID1> <ID2> <ID3>
#Then change to the galaxy-dist directory:
cd /home/galaxy/galaxy-dist
#Then restart Galaxy by executing the run.sh script:
nohup ./run.sh --reload &

#Alternate method to restart Galaxy:
./run.sh --daemon --reload

Newbler tool doesn't assemble FASTQ files as it claims to

2011-11-17T22:39:29Z

Pankey: Created page with "Newbler2.6 is meant to be able to assemble runs from multiple platforms (FASTA from ESTs, FASTQsanger from Illumina, SFF from 454) but so far, such 'hybrid' assemblies have not f…"

Newbler2.6 is meant to be able to assemble runs from multiple platforms (FASTA from ESTs, FASTQsanger from Illumina, SFF from 454) but so far, such 'hybrid' assemblies have not finished running after several days.

Galaxy Problems to address

2011-11-17T22:37:32Z

Pankey:

Implement NCBI's tools from SRA Toolkit

2011-11-16T01:09:50Z

Pankey: Created page with "NCBI's Trace Archive data is only available in SRA format now. NCBI offers downloadable tools to convert these files to FASTQ and other formats on local machines, but it would…"

NCBI's Trace Archive data is only available in SRA format now.

NCBI offers downloadable tools to convert these files to FASTQ and other formats on local machines, but it would be ideal if the galaxy would accept SRA input and we had the tools implemented from NCBI's SRA Toolkit:
[http://www.ncbi.nlm.nih.gov/books/NBK47540/#SRA_Download_Guid_B.5_Converting_SRA_for NCBI SRA Handbook]

High priority tools: fastq-dump, sff-dump

[http://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=software SRA Toolkit Software]

UCSB Galaxy Documentation

2011-11-16T01:05:26Z

Pankey:

== '''Galaxy Topics''' ==
* [[The very basics]]
* [[Uploading data]]
* [[Restarting Galaxy]]
* [[Using Trinity Assembler]]
* [[Adding a tool from tool shed]]
*
*[[Galaxy Development Priorities]]
**[[Regular Expression Search and replace -- Assigned to Karl]]
**[[Implement EvolMap -- Assigned to Roger]]
**[[Implement Hammstr - an external pipeline of tools]]
**[[Implement Davis' Illumina Scripts]]
**[[Implement RAxML]]
**[[Implement Beast]]
**[[Implement r8s]]
**[[Implement sequence alignment tools, MUSCLE MAFFT PROALIGN]]
**[[Implement phylogenetics scripts like format converters and concatenation tools]]
**[[Implement TransAbyss pipeline]]
**[[Implement NCBI's tools from SRA Toolkit]]
* [[Galaxy Problems to address]]
** [['Memory leak' type behavior]]
** [[Install cron jobs to clear deleted datasets]]
** [[Barcode splitter output]]
** [[blast2go not working]]
** [[FTP connection refused]]

Newbler tool doesn't recognize FASTQ files as it claims to

2011-11-16T01:04:35Z

Pankey: Blanked the page

Newbler tool doesn't recognize FASTQ files as it claims to

2011-11-16T01:03:39Z

Pankey: Created page with "Both Newbler assembly tools appear to accept SFF format as well as FASTA/FASTQ formats. So far, only SFF have worked. Newber could be useful for assembling datasets from multi…"

Both Newbler assembly tools appear to accept SFF format as well as FASTA/FASTQ formats. So far, only SFF have worked. Newber could be useful for assembling datasets from multiple sources that may only be available as FASTQ.

UCSB Galaxy Documentation

2011-11-16T00:59:32Z

Pankey:

== '''Galaxy Topics''' ==
* [[The very basics]]
* [[Uploading data]]
* [[Restarting Galaxy]]
* [[Using Trinity Assembler]]
* [[Adding a tool from tool shed]]
*
*[[Galaxy Development Priorities]]
**[[Regular Expression Search and replace -- Assigned to Karl]]
**[[Implement EvolMap -- Assigned to Roger]]
**[[Implement Hammstr - an external pipeline of tools]]
**[[Implement Davis' Illumina Scripts]]
**[[Implement RAxML]]
**[[Implement Beast]]
**[[Implement r8s]]
**[[Implement sequence alignment tools, MUSCLE MAFFT PROALIGN]]
**[[Implement phylogenetics scripts like format converters and concatenation tools]]
**[[Implement TransAbyss pipeline]]
**[[Create conversion tool for SRA files to SFF or FASTQ formats]]
* [[Galaxy Problems to address]]
** [['Memory leak' type behavior]]
** [[Install cron jobs to clear deleted datasets]]
** [[Barcode splitter output]]
** [[blast2go not working]]
** [[FTP connection refused]]
** [[ Newbler tool doesn't recognize FASTQ files as it claims to]]

Create conversion tool for SRA files to SFF or FASTQ formats

2011-11-16T00:57:47Z

Pankey:

NCBI's Trace Archive data is only available in SRA format now. They offer tools to convert these files to FASTQ on local machines, but it would be ideal if the galaxy would accept SRA input and we had the fastq-dump tool implemented from NCBI's SRA Toolkit:
[http://www.ncbi.nlm.nih.gov/books/NBK47540/#SRA_Download_Guid_B.5_Converting_SRA_for NCBI SRA Handbook]

5.1 Program Name: fastq-dump

5.2 Synopsis:

../bin64/fastq-dump <SRA archive file>

5.3 Example:

toolkit installed in: ../sra/bin64/

downloaded archive in: ../sra/download/SRR000299/

command to convert the SRA object into fastq data would be:

../sra/bin64/fastq-dump ../sra/download/SRR000299/SRR000299.sra
Output will be ‘SRR000299.fastq’ in current directory

This will also work for SRA archives that have been extracted out into their directory structure (contains sub-directories 'col' and files md5, meta, sealed... ) using the SRR###### directory as the location.

../sra/bin64/fastq-dump ../sra/download/SRR000299/

Create conversion tool for SRA files to SFF or FASTQ formats

2011-11-16T00:25:22Z

Pankey: Created page with "NCBI's Trace Archive data is only available in SRA format now. They offer tools to convert these files to FASTQ on local machines, but these "FASTQ" files are not recognized as …"

NCBI's Trace Archive data is only available in SRA format now. They offer tools to convert these files to FASTQ on local machines, but these "FASTQ" files are not recognized as FASTQ by the Galaxy assemblers.
[http://www.ncbi.nlm.nih.gov/books/NBK47540/#SRA_Download_Guid_B.5_Converting_SRA_for NCBI SRA Handbook]

5.1 Program Name: fastq-dump

5.2 Synopsis:

../bin64/fastq-dump <SRA archive file>

5.3 Example:

toolkit installed in: ../sra/bin64/

downloaded archive in: ../sra/download/SRR000299/

command to convert the SRA object into fastq data would be:

../sra/bin64/fastq-dump ../sra/download/SRR000299/SRR000299.sra
Output will be ‘SRR000299.fastq’ in current directory

This will also work for SRA archives that have been extracted out into their directory structure (contains sub-directories 'col' and files md5, meta, sealed... ) using the SRR###### directory as the location.

../sra/bin64/fastq-dump ../sra/download/SRR000299/

UCSB Galaxy Documentation

2011-11-16T00:13:20Z

Pankey:

== '''Galaxy Topics''' ==
* [[The very basics]]
* [[Uploading data]]
* [[Restarting Galaxy]]
* [[Using Trinity Assembler]]
* [[Adding a tool from tool shed]]
*
*[[Galaxy Development Priorities]]
**[[Regular Expression Search and replace -- Assigned to Karl]]
**[[Implement EvolMap -- Assigned to Roger]]
**[[Implement Hammstr - an external pipeline of tools]]
**[[Implement Davis' Illumina Scripts]]
**[[Implement RAxML]]
**[[Implement Beast]]
**[[Implement r8s]]
**[[Implement sequence alignment tools, MUSCLE MAFFT PROALIGN]]
**[[Implement phylogenetics scripts like format converters and concatenation tools]]
**[[Implement TransAbyss pipeline]]
**[[Create conversion tool for SRA files to SFF or FASTQ formats]]
* [[Galaxy Problems to address]]
** [['Memory leak' type behavior]]
** [[Install cron jobs to clear deleted datasets]]
** [[Barcode splitter output]]
** [[blast2go not working]]
** [[FTP connection refused]]

Macroevolution URL upload

2011-11-15T21:55:02Z

Pankey: Created page with "Use URL links to upload data from Macroevolution server to Galaxy on knot. The Macroevolution directory IlluminaData and its contents can be accessed (read-only) at http://macro…"

Use URL links to upload data from Macroevolution server to Galaxy on knot. The Macroevolution directory IlluminaData and its contents can be accessed (read-only) at http://macroevolution.lifesci.ucsb.edu/IlluminaData/

1. Access macroevolution through your unix terminal OR oakley-lab through your network connection window. Copy or move data files from the original location to the folder '''IlluminaData''' (The full path is: /labdata/nfs/lab/IlluminaData ). Use your own folder within IlluminaData to keep the directory organized.

2. From you web browser, go to [http://macroevolution.lifesci.ucsb.edu/IlluminaData/ macroevolution.lifesci.ucsb.edu/IlluminaData/]. Navigate to your data file.

3. Right-click on the data file and select "Copy Link Address" or similar. (If the file is very large, do not click to view the file, or the browser may crash.)

4. Once logged into Galaxy from your browser, select the '''Get Data''' tool called 'Upload File from your computer'.

5. Paste the copied url link into the text field under 'URL/Text' and Execute.

Uploading data

2011-11-15T21:54:08Z

Pankey:

# [[scp upload]]
# [[ftp upload]]
# [[macroevolution URL upload]]

* Potential Problems during Uploads
Sometimes if permissions are set incorrectly, you could get the following error:

Forbidden
You don't have permission to access /IlluminaData/YourFile

In that case, you must set permissions of the file. The user must be lab and group must be illumina. To fix this, you must [[Change ownership of linux files]].

Macroevolution url upload

2011-11-15T21:53:06Z

Ftp upload

2011-11-15T21:35:59Z

Pankey: Created page with "Use '''ftp''' (file transfer protocol) to copy data files to knot. 1. In your unix terminal, navigate to the directory containing your data file. 2. '''ftp''' into knot. Provi…"

Use '''ftp''' (file transfer protocol) to copy data files to knot.

1. In your unix terminal, navigate to the directory containing your data file.

2. '''ftp''' into knot. Provide your own Galaxy user login credentials when prompted.

-bash-3.2$ ftp
ftp> open knot-head.cnsi.ucsb.edu
ftp>

3. use '''put''' to copy file.

ftp> put datafile.fasta

* FTP connection refused by knot-head as of 11/15/2011.

Scp upload

2011-11-15T21:20:37Z

Pankey:

Use '''scp''' (secure copy) to transfer datafiles to the FTP directory within Galaxy on knot. Once the files are transferred, you will see them listed under the 'Files uploaded via FTP:' when you select Get Data:Upload File tool.

1. In your unix terminal, navigate to the macroevolution directory which contains your file(s). Use '''scp''' to copy the file(s) to the appropriate directory on knot.
For instance, to transfer the file 'datafile.fasta' to the Galaxy user 'yourname@lifesci.ucsb.edu':

-bash-3.2$ scp datafile.fasta galaxy@knot.cnsi.ucsb.edu:/home/galaxy/galaxy-dist/database/ftp/yourname\@lifesci.ucsb.edu/

(Note that the @ character requires the preceeding escape character \ in the latter instance because this email address must be understood as the directory rather than an address.)
You will be prompted to enter the galaxy password.

2. Using Galaxy from your browser, select the tool 'Upload File from your computer' under '''Get Data'''. You will see your scp'd files listed. Select the files you would like to upload and 'Execute'.

Scp upload

2011-11-15T21:18:03Z

Pankey: Created page with "Use '''scp''' (secure copy) to transfer datafiles to the FTP directory within Galaxy on knot. Once the files are transferred, you will see them listed under the 'Files uploaded …"

Use '''scp''' (secure copy) to transfer datafiles to the FTP directory within Galaxy on knot. Once the files are transferred, you will see them listed under the 'Files uploaded via FTP:' when you select Get Data:Upload File tool.

1. From the macroevolution directory which contains your file(s), use '''scp'''.
For instance, to transfer the file 'datafile.fasta' to the Galaxy user 'yourname@lifesci.ucsb.edu':

-bash-3.2$ scp datafile.fasta galaxy@knot.cnsi.ucsb.edu:/home/galaxy/galaxy-dist/database/ftp/yourname\@lifesci.ucsb.edu/

(Note that the @ character requires the preceeding escape character \ in the latter instance because this email address must be understood as the directory rather than an address.)

2. Using Galaxy from your browser, select the tool 'Upload File from your computer' under '''Get Data'''. You will see your scp'd files listed. Select the files you would like to upload and 'Execute'.

Uploading data

2011-11-15T20:56:21Z

Pankey:

# [[scp upload]]
# [[ftp upload]]
# [[macroevolution url upload]]

* Potential Problems during Uploads
Sometimes if permissions are set incorrectly, you could get the following error:

Forbidden
You don't have permission to access /IlluminaData/YourFile

In that case, you must set permissions of the file. The user must be lab and group must be illumina. To fix this, you must [[Change ownership of linux files]].

Uploading data

2011-11-15T20:55:11Z

Pankey:

# [[scp upload]]
# [[ftp upload]]
# [[macroevolution url upload]]
* Potential Problems
Sometimes if permissions are set incorrectly, you could get the following error:

Forbidden
You don't have permission to access /IlluminaData/YourFile

In that case, you must set permissions of the file. The user must be lab and group must be illumina. To fix this, you must [[Change ownership of linux files]].

Change ownership of linux files

2011-11-15T20:53:41Z

Pankey:

The '''chown''' command (abbreviation for '''ch'''ange '''own'''er) is used in Unix to change the owner of a file. Usually, it can only be executed by the superuser. To get superuser privileges, see Todd.

Unprivileged (regular) users who wish to change the group of a file that they own may use '''chgrp'''.

To change the ownership permissions, first look at the current permissions with '''ls -l''':

-bash-3.2$ ls -l
-rw-r----- 1 lab lab 324742643 Oct 10 17:28 file.txt

The output indicates that 'lab' is both the owing user and the owning group.

To change the owing group to 'illumina' (as required for uploading data to knot), use '''chgrp'''

-bash-3.2$ chgrp illumina file.txt

Confirm the group ownership has changed with '''ls -l''':

-bash-3.2$ ls -l
-rw-r----- 1 lab illumina 324742643 Oct 10 17:28 file.txt

Change ownership of linux files

2011-11-15T20:52:48Z

Pankey: Created page with "The '''chown''' command (abbreviation for '''ch'''ange '''own'''er) is used in Unix to change the owner of a file. Usually, it can only be executed by the superuser. To get sup…"

The '''chown''' command (abbreviation for '''ch'''ange '''own'''er) is used in Unix to change the owner of a file. Usually, it can only be executed by the superuser. To get superuser privileges, see Todd.
Unprivileged (regular) users who wish to change the group of a file that they own may use '''chgrp'''.

To change the ownership permissions, first look at the current permissions with '''ls -l''':

-bash-3.2$ ls -l
-rw-r----- 1 lab lab 324742643 Oct 10 17:28 file.txt

The output indicates that 'lab' is both the owing user and the owning group.

To change the owing group to 'illumina' (as required for uploading data to knot), use '''chgrp'''

-bash-3.2$ chgrp illumina file.txt

Confirm the group ownership has changed with '''ls -l''':

-bash-3.2$ ls -l
-rw-r----- 1 lab illumina 324742643 Oct 10 17:28 file.txt