I ran a submission on one pair, which had never been run before, and I did not uncheck the 'use call caching' checkbox. It ran and passed. I then ran a new submission on a 5000-ish wide pairset that included the first pair I ran, again leaving the 'use call caching' box checked. All of the pairs in the set were run, including the first one I ran. I expected the first one I ran to call cache. In the monitor tab, both jobs say Call Caching was disabled. I'm pretty sure I saw this same unexpected-cache-miss behavior on submissions I made on August 20 in the same workspace, but this example is very clean cut, with everything contained in a tight timeframe.
Workspace: nci-gsaksena-bi-org/MC3_mutation_validator
Shared with GROUP_support@firecloud.org, in TCGA authorization domain
Around 9:30am today:
pair entity: UVM-YZ-A985-TP-NB
Initial submission:
2b3f22a8-df52-47bc-a752-e17abb9013b1
initial workflow id
f5ac1eb0-af59-4455-a170-d014fefe52a1
operations id
operations/EIr-vMzhKxjL9r2Fys6ZsnMgobKt79UBKg9wcm9kdWN0aW9uUXVldWU
The job finished in 2 minutes - about the time expected - based on both the JES log and operations log.
Around 11am today:
pairset entity: mc3_2of2 (contains 5459 pairs, one of which is: UVM-YZ-A985-TP-NB)
submission id
314435bf-ddc9-4d95-844e-852e82bcf346
workflow id
95b223ec-a4ae-401e-b761-78f6e2b28f0a
operations id
operations/EOvzoM_hKxjItNfv5L-e4uYBIKGyre_VASoPcHJvZHVjdGlvblF1ZXVl
Based on the operations log, the job finished on the VM in 27 minutes - much longer than expected. It looks like there was an additional 15 minutes of setup and 10 minutes of teardown on the VM that was not present for the single pair submission. The JES log does not reflect this extra time, just showing the 2 minutes needed for the job.
In addition, I notice that the second submission has the wrong submission time listed in the monitor page... I know I submitted it right at 11am, but the page claims I submitted it at 10:14am. The operations log and JES log are consistent with an 11am start.
So there are several issues:
1) Why didn't the second job call-cache?
2) In the second submission, why is there an extra 25 minutes of VM time that seems to be doing nothing useful for me?
3) In the second submission, why is the submission time on the monitor page back-dated from the real submission time?
Thanks,
Gordon