We observed this in a benchmarking run of the best practice QC workflow on 1000 BRCA tumor/normal pairs. Of the 1000 workflows, 991 succeeded, 5 failed, and 4 are still flagged as running. This is an issue around one of the workflows that is reported as still running.
The workflow includes a scatter step, where a region coverage tool is scattered across 24 shards. All but one of the scattered calls is reported as being done (and successful). One of the scattered calls (call #10) is reported as still running. However, if you go to the bucket folder associated with that call, you see that the task successfully completed (there is a return code file containing a return code of 0). This "running" task is holding up the completion of the workflow.
The workspace in which we see this is broad-firecloud-broade/BenchmarkWorkspace_template_CloneTestSharing2; this is a controlled access workspace.
The analysis submission in which this stalled workflow was launched is 7afeacc4-d787-4ca6-8e32-7acab85c1d1f
The stalled workflow id is: e6f66e08-449f-4c11-b050-3f04bbf8352d
The operations ID of the scatter call that is being reported as running, but has actually completed is: EJLFsrSWKxjbm-GG3emt6lcg5pjYguIXKg9wcm9kdWN0aW9uUXVldWU