Chip submitted a job. Some starting tasks ran okay, but a middle task did not start. I suggested to him to inspect the network tab. Upon inspection, we saw the message here:
"failures": [{
"message": "Couldn't resolve all inputs for FilterWorkflow.mergeInIndelsToOBF at index None.: Google credentials are invalid: connect timed out"
}],
Other workflows in the same submission did not have this issue. So this one workflow (of 7 total) had the "Google credentials are invalid" issue.
What is causing this problem? What is the solution? Since other workflows did not have the issue it would seem that this is transient.
Should the job simply be re-launched?
Is there an estimated time of arrival for call-caching/job-caching? with call-caching/job-caching, re-launching won't be as costly because the cache lookups could be cache hits especially for workflows where any failure/issue occurs towards the end or in the middle.