Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use relative paths for inputs to make the training portable #438

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Use relative paths for inputs to make the training portable #438

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cc1cec9
Use relative paths for inputs to make the training portable
adamrtalbot Nov 6, 2024
2e9e2d4
Use slightly simpler map statement
adamrtalbot Nov 6, 2024
e7957de
Change reference to changing into the absolute path
adamrtalbot Nov 6, 2024
e83a38f
Use file method to convert string to path
adamrtalbot Nov 6, 2024
08428f0
Add results_genomics to gitignore
adamrtalbot Nov 7, 2024
603e7ac
Use splitCsv to parse the samplesheet
adamrtalbot Nov 7, 2024
389e560
Merge branch 'master' into use_relative_paths_for_inputs
pinin4fjords Nov 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/hello_nextflow/03_hello_containers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,9 +115,11 @@ One way to do this is to **mount** a **volume** from the host system into the co
Prior to working on the next task, confirm that you are in the `hello-nextflow` directory.

```bash
cd /workspace/gitpod/hello-nextflow
pwd
```

This should show `/workspaces/training/hello-nextflow`. The important point is the `hello-nextflow` is the final path.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This should show `/workspaces/training/hello-nextflow`. The important point is the `hello-nextflow` is the final path.
This should show `/workspaces/hello-nextflow`. The important point is the `hello-nextflow` is the final path.


Then run:

```bash
Expand Down
21 changes: 11 additions & 10 deletions docs/hello_nextflow/04_hello_genomics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ The tools we need (Samtools and GATK) are not installed in the Gitpod environmen
!!! note

Make sure you're in the correct working directory:
`cd /workspace/gitpod/hello-nextflow`
`pwd` should return a path ending in `hello-nextflow`

### 0.1. Index a BAM input file with Samtools

Expand Down Expand Up @@ -561,9 +561,9 @@ This error will not reproduce consistently because it is dependent on some varia
This is what the output of the two `.view` calls we added looks like for a failed run:

```console title="Output"
/workspace/gitpod/hello-nextflow/data/bam/reads_mother.bam
/workspace/gitpod/hello-nextflow/data/bam/reads_father.bam
/workspace/gitpod/hello-nextflow/data/bam/reads_son.bam
./data/bam/reads_mother.bam
./data/bam/reads_father.bam
./data/bam/reads_son.bam
/workspace/gitpod/hello-nextflow/work/9c/53492e3518447b75363e1cd951be4b/reads_father.bam.bai
/workspace/gitpod/hello-nextflow/work/cc/37894fffdf6cc84c3b0b47f9b536b7/reads_son.bam.bai
/workspace/gitpod/hello-nextflow/work/4d/dff681a3d137ba7d9866e3d9307bd0/reads_mother.bam.bai
Expand Down Expand Up @@ -717,9 +717,9 @@ Here we are going to show you how to do the simple case.
We already made a text file listing the input file paths, called `sample_bams.txt`, which you can find in the `data/` directory.

```txt title="sample_bams.txt"
/workspace/gitpod/hello-nextflow/data/bam/reads_mother.bam
/workspace/gitpod/hello-nextflow/data/bam/reads_father.bam
/workspace/gitpod/hello-nextflow/data/bam/reads_son.bam
/data/bam/reads_mother.bam
/data/bam/reads_father.bam
/data/bam/reads_son.bam
Comment on lines +721 to +723
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/data/bam/reads_mother.bam
/data/bam/reads_father.bam
/data/bam/reads_son.bam
./data/bam/reads_mother.bam
./data/bam/reads_father.bam
./data/bam/reads_son.bam

```

As you can see, we listed one file path per line, and they are absolute paths.
Expand Down Expand Up @@ -768,9 +768,10 @@ reads_ch = Channel.fromPath(params.reads_bam)

_After:_

```groovy title="hello-genomics.nf" linenums="68"
````groovy title="hello-genomics.nf" linenums="68"
// Create input channel from a text file listing input file paths
reads_ch = Channel.fromPath(params.reads_bam).splitText()
reads_ch = Channel.fromPath(params.reads_bam)
.splitText() { bamPath -> file(bamPath.trim()) }
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does anyone know if there's a better way of doing this? Without the map nothing works 😠

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

splitCsv?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the equivalent splitCsv():

reads_ch = Channel.fromPath(params.reads_bam)
                .splitCsv()
                .map { bamPath -> file(bamPath[0]) }

I think it looks nicer

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. I like that version a lot. It's a lot easier to follow.

```

!!! tip
Expand All @@ -783,7 +784,7 @@ Let's run the workflow one more time.

```bash
nextflow run hello-genomics.nf -resume
```
````

This should produce the same result as before, right?

Expand Down
6 changes: 3 additions & 3 deletions hello-nextflow/data/sample_bams.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
/workspace/gitpod/hello-nextflow/data/bam/reads_mother.bam
/workspace/gitpod/hello-nextflow/data/bam/reads_father.bam
/workspace/gitpod/hello-nextflow/data/bam/reads_son.bam
data/bam/reads_mother.bam
data/bam/reads_father.bam
data/bam/reads_son.bam
3 changes: 2 additions & 1 deletion hello-nextflow/hello-config/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,8 @@ process GATK_JOINTGENOTYPING {
workflow {

// Create input channel from a text file listing input file paths
reads_ch = Channel.fromPath(params.reads_bam).splitText()
reads_ch = Channel.fromPath(params.reads_bam)
.splitText() { bamPath -> file(bamPath.trim()) }

// Load the file paths for the accessory files (reference and intervals)
ref_file = file(params.reference)
Expand Down
3 changes: 2 additions & 1 deletion hello-nextflow/hello-modules/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,8 @@ process GATK_JOINTGENOTYPING {
workflow {

// Create input channel from a text file listing input file paths
reads_ch = Channel.fromPath(params.reads_bam).splitText()
reads_ch = Channel.fromPath(params.reads_bam)
.splitText() { bamPath -> file(bamPath.trim()) }

// Load the file paths for the accessory files (reference and intervals)
ref_file = file(params.reference)
Expand Down
3 changes: 2 additions & 1 deletion hello-nextflow/hello-nf-test/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ include { GATK_JOINTGENOTYPING } from './modules/local/gatk/jointgenotyping/main
workflow {

// Create input channel from a text file listing input file paths
reads_ch = Channel.fromPath(params.reads_bam).splitText()
reads_ch = Channel.fromPath(params.reads_bam)
.splitText() { bamPath -> file(bamPath.trim()) }

// Load the file paths for the accessory files (reference and intervals)
ref_file = file(params.reference)
Expand Down
3 changes: 2 additions & 1 deletion hello-nextflow/hello-operators.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,8 @@ process GATK_HAPLOTYPECALLER {
workflow {

// Create input channel from a text file listing input file paths
reads_ch = Channel.fromPath(params.reads_bam).splitText()
reads_ch = Channel.fromPath(params.reads_bam)
.splitText() { bamPath -> file(bamPath.trim()) }

// Load the file paths for the accessory files (reference and intervals)
ref_file = file(params.reference)
Expand Down
5 changes: 3 additions & 2 deletions hello-nextflow/solutions/hello-config/final-main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ process SAMTOOLS_INDEX {

output:
tuple path(input_bam), path("${input_bam}.bai")

script:
"""
samtools index '$input_bam'
Expand Down Expand Up @@ -96,7 +96,8 @@ process GATK_JOINTGENOTYPING {
workflow {

// Create input channel from a text file listing input file paths
reads_ch = Channel.fromPath(params.reads_bam).splitText()
reads_ch = Channel.fromPath(params.reads_bam)
.splitText() { bamPath -> file(bamPath.trim()) }

// Load the file paths for the accessory files (reference and intervals)
ref_file = file(params.reference)
Expand Down
5 changes: 3 additions & 2 deletions hello-nextflow/solutions/hello-genomics/hello-genomics-4.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ process GATK_HAPLOTYPECALLER {

output:
path "${input_bam}.vcf" , emit: vcf
path "${input_bam}.vcf.idx" , emit: idx
path "${input_bam}.vcf.idx" , emit: idx

script:
"""
Expand All @@ -67,7 +67,8 @@ process GATK_HAPLOTYPECALLER {
workflow {

// Create input channel from a text file listing input file paths
reads_ch = Channel.fromPath(params.reads_bam).splitText()
reads_ch = Channel.fromPath(params.reads_bam)
.splitText() { bamPath -> file(bamPath.trim()) }

// Load the file paths for the accessory files (reference and intervals)
ref_file = file(params.reference)
Expand Down
3 changes: 2 additions & 1 deletion hello-nextflow/solutions/hello-modules/final-main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ include { GATK_JOINTGENOTYPING } from './modules/local/gatk/jointgenotyping/main
workflow {

// Create input channel from a text file listing input file paths
reads_ch = Channel.fromPath(params.reads_bam).splitText()
reads_ch = Channel.fromPath(params.reads_bam)
.splitText() { bamPath -> file(bamPath.trim()) }

// Load the file paths for the accessory files (reference and intervals)
ref_file = file(params.reference)
Expand Down
5 changes: 3 additions & 2 deletions hello-nextflow/solutions/hello-operators/hello-operators-1.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ process GATK_HAPLOTYPECALLER {

output:
path "${input_bam}.g.vcf" , emit: vcf
path "${input_bam}.g.vcf.idx" , emit: idx
path "${input_bam}.g.vcf.idx" , emit: idx

script:
"""
Expand All @@ -68,7 +68,8 @@ process GATK_HAPLOTYPECALLER {
workflow {

// Create input channel from a text file listing input file paths
reads_ch = Channel.fromPath(params.reads_bam).splitText()
reads_ch = Channel.fromPath(params.reads_bam)
.splitText() { bamPath -> file(bamPath.trim()) }

// Load the file paths for the accessory files (reference and intervals)
ref_file = file(params.reference)
Expand Down
5 changes: 3 additions & 2 deletions hello-nextflow/solutions/hello-operators/hello-operators-2.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ process GATK_HAPLOTYPECALLER {

output:
path "${input_bam}.g.vcf" , emit: vcf
path "${input_bam}.g.vcf.idx" , emit: idx
path "${input_bam}.g.vcf.idx" , emit: idx

script:
"""
Expand Down Expand Up @@ -99,7 +99,8 @@ process GATK_GENOMICSDB {
workflow {

// Create input channel from a text file listing input file paths
reads_ch = Channel.fromPath(params.reads_bam).splitText()
reads_ch = Channel.fromPath(params.reads_bam)
.splitText() { bamPath -> file(bamPath.trim()) }

// Load the file paths for the accessory files (reference and intervals)
ref_file = file(params.reference)
Expand Down
5 changes: 3 additions & 2 deletions hello-nextflow/solutions/hello-operators/hello-operators-3.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ process GATK_HAPLOTYPECALLER {

output:
path "${input_bam}.g.vcf" , emit: vcf
path "${input_bam}.g.vcf.idx" , emit: idx
path "${input_bam}.g.vcf.idx" , emit: idx

script:
"""
Expand Down Expand Up @@ -109,7 +109,8 @@ process GATK_JOINTGENOTYPING {
workflow {

// Create input channel from a text file listing input file paths
reads_ch = Channel.fromPath(params.reads_bam).splitText()
reads_ch = Channel.fromPath(params.reads_bam)
.splitText() { bamPath -> file(bamPath.trim()) }

// Load the file paths for the accessory files (reference and intervals)
ref_file = file(params.reference)
Expand Down
Loading