Hi FC Team,
In the last ~6 hours, I've noticed some unusual behavior when rerunning workflows almost entirely from cached results. This new behavior is causing these workflows to go from rerunning entirely from cache in a matter of minutes to hours, despite ~no new compute being performed.
While looking into this discrepancy, I noticed that tasks were appearing to only check the cache in regular ~15 minute intervals. I've attached an example of a workflow timing chart for one of the subworkflows here: it shows that some tasks hit the cache right away, and all others wait 15 minutes at which time some more (but not all) hit the cache, after which there is another 15 minute period of waiting (and so on):
Any thoughts as to what could be causing this type of behavior? The example workspace is talkowski-sv-gnomad-wgs-v2/SV_Talkowski_GNOMAD_WGS-V2 and an example workflow where I noticed this behavior has Submission ID 2ed19be0-ec83-47f2-a020-960098b442ff
Thanks!
Ryan