My workflow (attached) ran and has a job stuck in the running state. The first task, bsmap ran and succeeded (the job outputs are in the appropriate google buckets) and status is properly updated/reflected in FireCloud/Cromwell. The next task samtools_sort has finished running successfully (the output has been delocalized and there's a proper return code) but it's status is still "Running". The last line in the logs for this workflow states 2017-07-19 17:05:23,472 cromwell-system-akka.dispatchers.backend-dispatcher-139 INFO - JesAsyncBackendJobExecutionActor [UUID(4f1968d0)methpipeindv.samtools_sort:NA:1]: Status change from - to Success but there's no status update.
Workflow logs:
2017-07-19 17:05:23,472 cromwell-system-akka.dispatchers.backend-dispatcher-139 INFO - JesAsyncBackendJobExecutionActor [UUID(4f1968d0)methpipeindv.samtools_sort:NA:1]: Status change from - to Success
2017-07-19 16:55:35,151 cromwell-system-akka.dispatchers.backend-dispatcher-172 INFO - JesAsyncBackendJobExecutionActor [UUID(4f1968d0)methpipeindv.samtools_sort:NA:1]: job id: operations/EILnmt7VKxjsyqf28KnJ2J0BILaftM_3CSoPcHJvZHVjdGlvblF1ZXVl
2017-07-19 16:45:11,316 cromwell-system-akka.dispatchers.backend-dispatcher-153 INFO - JesAsyncBackendJobExecutionActor [UUID(4f1968d0)methpipeindv.bsmap:NA:1]: Status change from Running to Success
2017-07-19 15:51:41,874 cromwell-system-akka.dispatchers.backend-dispatcher-125 INFO - JesAsyncBackendJobExecutionActor [UUID(4f1968d0)methpipeindv.bsmap:NA:1]: `bsmap -a /cromwell_root/fc-dceaadae-be69-41ab-a230-0b735c0556c1/sc-RRBS-zygote_01_R1.fastq.gz -b /cromwell_root/fc-dceaadae-be69-41ab-a230-0b735c0556c1/sc-RRBS-zygote_01_R2.fastq.gz -d /cromwell_root/fc-dceaadae-be69-41ab-a230-0b735c0556c1/Mus_musculus_assembly10.fasta -p 4 -v 0.05 -s 16 -r 0 -u -S 1 -R -o sc-RRBS-zygote_01_raw_bs.bam`
2017-07-19 15:51:41,873 cromwell-system-akka.dispatchers.backend-dispatcher-125 WARN - JesAsyncBackendJobExecutionActor [UUID(4f1968d0)methpipeindv.bsmap:NA:1]: Unrecognized runtime attribute keys: defaultDisks
2017-07-19 15:51:41,142 cromwell-system-akka.dispatchers.engine-dispatcher-39 INFO - WorkflowExecutionActor-4f1968d0-5060-4a85-aaec-a325d459d46e [UUID(4f1968d0)]: Starting calls: methpipeindv.bsmap:NA:1
gs://cromwell-auth-broad-firecloud-methylation/4f1968d0-5060-4a85-aaec-a325d459d46e_auth.json
2017-07-19 15:51:38,610 cromwell-system-akka.dispatchers.backend-dispatcher-178 INFO - JES [UUID(4f1968d0)]: Creating authentication file for workflow 4f1968d0-5060-4a85-aaec-a325d459d46e at
2017-07-19 15:51:38,574 cromwell-system-akka.dispatchers.backend-dispatcher-178 WARN - JES [UUID(4f1968d0)]: Key/s [defaultDisks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2017-07-19 15:51:38,574 cromwell-system-akka.dispatchers.backend-dispatcher-178 WARN - JES [UUID(4f1968d0)]: Key/s [defaultDisks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2017-07-19 15:51:38,574 cromwell-system-akka.dispatchers.backend-dispatcher-178 WARN - JES [UUID(4f1968d0)]: Key/s [defaultDisks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2017-07-19 15:51:38,574 cromwell-system-akka.dispatchers.backend-dispatcher-178 WARN - JES [UUID(4f1968d0)]: Key/s [defaultDisks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2017-07-19 15:51:38,574 cromwell-system-akka.dispatchers.backend-dispatcher-178 WARN - JES [UUID(4f1968d0)]: Key/s [defaultDisks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2017-07-19 15:51:38,573 cromwell-system-akka.dispatchers.engine-dispatcher-63 INFO - MaterializeWorkflowDescriptorActor [UUID(4f1968d0)]: Call-to-Backend assignments: methpipeindv.samtools_read_metrics -> JES, methpipeindv.MethylDackel -> JES, methpipeindv.bsmap -> JES, methpipeindv.samtools_sort -> JES, methpipeindv.MethylDackel_CHH -> JES, methpipeindv.create_rda -> JES, methpipeindv.bs_conversion_rate -> JES
2017-07-19 15:51:38,525 cromwell-system-akka.dispatchers.engine-dispatcher-90 INFO - WorkflowManagerActor Successfully started WorkflowActor-4f1968d0-5060-4a85-aaec-a325d459d46e
2017-07-19 15:51:38,525 cromwell-system-akka.dispatchers.engine-dispatcher-90 INFO - WorkflowManagerActor Starting workflow UUID(4f1968d0-5060-4a85-aaec-a325d459d46e)
2017-07-19 15:51:37,424 cromwell-system-akka.dispatchers.api-dispatcher-904 INFO - Workflows 4f1968d0-5060-4a85-aaec-a325d459d46e submitted.