Hello,
I have experienced many instances of tasks getting stuck while running today. In the first screenshot, I have about 227 out of 387 runs that have been stuck in the aborting phase since 11:40AM today. I had aborted these after I noticed they were taking especially long to run. When I looked at these workflows individually, I saw that these workflows were stuck in two ways
1) The task had been launched but then it did not run (the only item found in the google bucket with respect to this submission ID was the script file, no rc or log files)
2) The task was actually completed (all outputs were in the bucket and the VM was closed) but would not go to "completed" state.
In either case, the VMs seem to be spun down: I saw no API activity a few minutes after the point at which I had hit "abort".
Additionally, I have a workflow that has been stuck in a call cache-d task for 40 minutes, for a task that without call cache-ing runs in three (as seen in the second screenshot).
Is there a reason that my tasks are all getting stuck? The workspace is ebert-fc/exac-maf-aggregation; it is already shared with GROUP_support@firecloud.org. Thank you!
Best,
Matt
