We observed this in a benchmarking run of the best practice QC workflow on 1000 BRCA tumor/normal pairs. Of the 1000 workflows, 991 succeeded, 5 failed, and 4 are still flagged as running. This is an issue with one of the workflows that is reported as still running. The workflow is stalled in a single scatter call (call #15, shard-14). The workflow log is identical to that of the scatter calls that completed successfully. Stderr log is empty. The Stdout log is very different from the stdout log from successful scatter calls. It is unclear what is going on, but this stalled call should have completed 5 hours ago (at around the same time that the other scatter calls completed).
The workspace in which we see this is broad-firecloud-broade/BenchmarkWorkspace_template_CloneTestSharing2; this is a controlled access workspace.
The analysis submission in which this stalled workflow was launched is 7afeacc4-d787-4ca6-8e32-7acab85c1d1f
The workflow id for the stalled workflow is: 431f9ae1-a32f-4b0e-84de-6a38de337cf6
The operations ID associated with call#15 of the scatter task is: EJLj9LmWKxif29vzk7Dl_CQg5pjYguIXKg9wcm9kdWN0aW9uUXVldWU