I am running tests using a workflow (somatic mutation calling) which includes the 25-way scattering of a mutect1 job. The workflow is configured to run those scatter jobs on preemptible VMs. We have run four analysis submissions, with each submission running the workflow on 1000 tumor/normal pairs. That has resulted in 4 X 1000 X 25 = 100,000 JES jobs run on preemptible machines (probably closer to 99,000 because for each analysis submission approximately 10 workflows failed). Each of these jobs run between 30 minutes and an hour.
I then used the /api/workspaces/{workspaceNamespace}/{workspaceName}/submissions/{submissionId}/workflows/{workflowId} endpoint to get call-level metadata which, according to the Cromwell team (discussed during office hours), should include information on preemption status. The call-level metadata includes an 'attempt' attribute. For all 100,000 jobs examined, the reported attempt attribute value was '1', with an execution status of 'Done', return code of '0'. I have attached a file containing a sample of the call level metadata.
Is it likely that I am seeing no evidence of preemption in close to 100,000 calls?
Am I looking for preemption data in the wrong place?
Is Cromwell not accurately reporting preemption data?
I have asked @esalinas to look for evidence of preemption in these four submissions by directly querying google JES for call-level metadata using the gcloud alpha genomics operations list and gcloud alpha genomics operations describe commands.