I am reviewing a workflow. The workflow is marked as Failed (see attached PDF).
The workflow ran 3 tasks. Two are done (with green check marks) and one is marked "Running". Of the one that is marked "Running" if I ask gcloud alpha genomics operations describe about it, then it says done: true.
wm8b1-75c:~ esalinas$ gcloud alpha genomics operations describe operations/EJOTlISoKxjaw-zrh9eokKkBIN6K28SDHCoPcHJvZHVjdGlvblF1ZXVl
done: true
metadata:
'@type': type.googleapis.com/google.genomics.v1.OperationMetadata
clientId: ''
createTime: '2017-02-27T17:01:26Z'
endTime: '2017-02-27T17:27:07Z'
events:
- description: start
startTime: '2017-02-27T17:02:10.496569246Z'
- description: pulling-image
startTime: '2017-02-27T17:02:12.154200674Z'
- description: localizing-files
startTime: '2017-02-27T17:03:26.833744916Z'
- description: running-docker
startTime: '2017-02-27T17:11:43.078594022Z'
- description: delocalizing-files
startTime: '2017-02-27T17:26:58.038856218Z'
- description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task-rc.txt"
startTime: '2017-02-27T17:26:59.216433922Z'
- description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/wigs.coverage.tgz"
startTime: '2017-02-27T17:27:00.452218832Z'
- description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/dstat.log.txt"
startTime: '2017-02-27T17:27:01.672082122Z'
- description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/df.log.txt"
startTime: '2017-02-27T17:27:02.928070326Z'
- description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/call_stats.tgz"
startTime: '2017-02-27T17:27:04.333975179Z'
- description: copied 1 file(s) to "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/power.coverage.tgz"
startTime: '2017-02-27T17:27:05.908141602Z'
- description: ok
startTime: '2017-02-27T17:27:07.217623964Z'
labels: {}
projectId: broad-firecloud-benchmark
request:
'@type': type.googleapis.com/google.genomics.v1alpha2.RunPipelineRequest
ephemeralPipeline:
docker:
cmd: /bin/bash /cromwell_root/exec.sh
imageName: broadinstitute/broadmutationcalling_beta:benchmark_3
inputParameters:
- name: __extra_config_gcs_path
- localCopy:
disk: local-disk
path: 5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-01A-11W-A019-09.bam.bai
name: Mutect1_Task.tumorBamIdx-0
- localCopy:
disk: local-disk
path: 5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-10A-01W-A021-09.bam.bai
name: Mutect1_Task.normalBamIdx-0
- localCopy:
disk: local-disk
path: firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta.fai
name: Mutect1_Task.refFastaIdx-0
- localCopy:
disk: local-disk
path: firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.dict
name: Mutect1_Task.refFastaDict-0
- localCopy:
disk: local-disk
path: firecloud-tcga-open-access/tutorial/reference/refseq_exome_10bp_hg19_300_1kg_normal_panel.vcf
name: Mutect1_Task.normalPanel-0
- localCopy:
disk: local-disk
path: firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf
name: Mutect1_Task.cosmicVCF-0
- localCopy:
disk: local-disk
path: firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf
name: Mutect1_Task.dbSNPVCF-0
- localCopy:
disk: local-disk
path: fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-CallSomaticMutations_131_Prepare_Task/split_intervals.tar.gz
name: Mutect1_Task.mutectIntervals-0
- localCopy:
disk: local-disk
path: firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt
name: Mutect1_Task.readGroupBlackList-0
- localCopy:
disk: local-disk
path: 5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-10A-01W-A021-09.bam
name: Mutect1_Task.normalBam-0
- localCopy:
disk: local-disk
path: exec.sh
name: exec
- localCopy:
disk: local-disk
path: firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta
name: Mutect1_Task.refFasta-0
- localCopy:
disk: local-disk
path: 5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-01A-11W-A019-09.bam
name: Mutect1_Task.tumorBam-0
name: CallingGroup_Workflow
outputParameters:
- localCopy:
disk: local-disk
path: Mutect1_Task-rc.txt
name: Mutect1_Task-rc.txt
- localCopy:
disk: local-disk
path: wigs.coverage.tgz
name: wigs.coverage.tgz
- localCopy:
disk: local-disk
path: dstat.log.txt
name: dstat.log.txt
- localCopy:
disk: local-disk
path: df.log.txt
name: df.log.txt
- localCopy:
disk: local-disk
path: call_stats.tgz
name: call_stats.tgz
- localCopy:
disk: local-disk
path: power.coverage.tgz
name: power.coverage.tgz
projectId: broad-firecloud-benchmark
resources:
bootDiskSizeGb: 10
disks:
- mountPoint: /cromwell_root
name: local-disk
sizeGb: 103
type: PERSISTENT_HDD
minimumCpuCores: 32
minimumRamGb: 100
preemptible: true
zones:
- us-central1-b
- us-central1-c
- us-central1-f
pipelineArgs:
clientId: ''
inputs:
Mutect1_Task.cosmicVCF-0: gs://firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf
Mutect1_Task.dbSNPVCF-0: gs://firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf
Mutect1_Task.mutectIntervals-0: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-CallSomaticMutations_131_Prepare_Task/split_intervals.tar.gz
Mutect1_Task.normalBam-0: gs://5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-10A-01W-A021-09.bam
Mutect1_Task.normalBamIdx-0: gs://5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-10A-01W-A021-09.bam.bai
Mutect1_Task.normalPanel-0: gs://firecloud-tcga-open-access/tutorial/reference/refseq_exome_10bp_hg19_300_1kg_normal_panel.vcf
Mutect1_Task.readGroupBlackList-0: gs://firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt
Mutect1_Task.refFasta-0: gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta
Mutect1_Task.refFastaDict-0: gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.dict
Mutect1_Task.refFastaIdx-0: gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta.fai
Mutect1_Task.tumorBam-0: gs://5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-01A-11W-A019-09.bam
Mutect1_Task.tumorBamIdx-0: gs://5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/WGA_RepliG/WXS/BI/ILLUMINA/TCGA_MC3.TCGA-A8-A06U-01A-11W-A019-09.bam.bai
__extra_config_gcs_path: gs://cromwell-auth-broad-firecloud-benchmark/52641b4a-006f-4421-81a9-77443b420970_auth.json
exec: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/exec.sh
labels: {}
logging:
gcsPath: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task.log
outputs:
Mutect1_Task-rc.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task-rc.txt
call_stats.tgz: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/call_stats.tgz
df.log.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/df.log.txt
dstat.log.txt: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/dstat.log.txt
power.coverage.tgz: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/power.coverage.tgz
wigs.coverage.tgz: gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/wigs.coverage.tgz
projectId: broad-firecloud-benchmark
resources:
bootDiskSizeGb: 10
disks:
- autoDelete: false
mountPoint: ''
name: local-disk
sizeGb: 103
source: ''
type: PERSISTENT_HDD
minimumCpuCores: 32
minimumRamGb: 100
noAddress: false
preemptible: true
zones:
- us-central1-b
- us-central1-c
- us-central1-f
serviceAccount:
email: default
scopes:
- https://www.googleapis.com/auth/genomics
- https://www.googleapis.com/auth/compute
runtimeMetadata:
'@type': type.googleapis.com/google.genomics.v1alpha2.RuntimeMetadata
computeEngine:
diskNames:
- local-disk-12186919504927465946
instanceName: ggp-12186919504927465946
machineType: us-central1-b/n1-standard-32
zone: us-central1-b
startTime: '2017-02-27T17:01:32Z'
name: operations/EJOTlISoKxjaw-zrh9eokKkBIN6K28SDHCoPcHJvZHVjdGlvblF1ZXVl
wm8b1-75c:~ esalinas$ gsutil gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/
CommandException: Invalid command "gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/".
wm8b1-75c:~ esalinas$ gsutil ls gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task-rc.txt
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task-stderr.log
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task-stdout.log
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task.log
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/call_stats.tgz
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/df.log.txt
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/dstat.log.txt
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/exec.sh
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/power.coverage.tgz
gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/wigs.coverage.tgz
Now, when I review the return code for the "Running" task and I see zero:
wm8b1-75c:~ esalinas$ gsutil cat gs://fc-6e92fceb-ef44-4648-a5b0-71cacc0248eb/b81303cf-a79c-41d6-bea5-1de19d97732c/CallingGroup_Workflow/52641b4a-006f-4421-81a9-77443b420970/call-Mutect1_Task/Mutect1_Task-rc.txt
0
My question is what causes a workflow to be marked as Failed? In the case here, I would presume that the non-done task (without a green-checkmark and marked as "Running" would be marked "Failed" or have a non-zero return code. I reviewed the JES log and saw the all files were successfully delocalized.