In the LOG I see :
2016/04/29 13:07:43 I: Switching to status: delocalizing-files
2016/04/29 13:07:43 I: Docker file /cromwell_root/CallSomaticMutations_131_Prepare-rc.txt maps to host location /mnt/local-disk/CallSomaticMutations_131_Prepare-rc.txt.
2016/04/29 13:07:43 I: Copying /mnt/local-disk/CallSomaticMutations_131_Prepare-rc.txt to gs://fc-328e1dba-07f7-49e5-9034-33f47fa027cc/239fa60b-b6a3-4832-bd54-29bde10169c1/CallingGroupWorkflow/a261a132-edfb-4f52-aeb1-11c5092dfd8a/call-CallSomaticMutations_131_Prepare/CallSomaticMutations_131_Prepare-rc.txt
2016/04/29 13:07:44 I: Docker file /cromwell_root/interval_split_base.* maps to host location /mnt/local-disk/interval_split_base.*.
2016/04/29 13:07:44 I: Copying /mnt/local-disk/interval_split_base.* to gs://fc-328e1dba-07f7-49e5-9034-33f47fa027cc/239fa60b-b6a3-4832-bd54-29bde10169c1/CallingGroupWorkflow/a261a132-edfb-4f52-aeb1-11c5092dfd8a/call-CallSomaticMutations_131_Prepare/glob-efd083f9b00eb987f01181c24e49dc7c
2016/04/29 13:07:46 E: command failed: AccessDeniedException: 403 Forbidden
CommandException: 1 file/object could not be transferred.
(exit status 1)
2016/04/29 13:07:46 W: cp failed: gsutil -o Credentials:gs_oauth2_refresh_token=1/xxxxxxxxx -o OAuth2:client_id=972537320227-dipecl4g8mftumf2edelamfrf76tolok.apps.googleusercontent.com -o OAuth2:client_secret=xxxxxxxxx -q -m cp /mnt/local-disk/interval_split_base.* gs://fc-328e1dba-07f7-49e5-9034-33f47fa027cc/239fa60b-b6a3-4832-bd54-29bde10169c1/CallingGroupWorkflow/a261a132-edfb-4f52-aeb1-11c5092dfd8a/call-CallSomaticMutations_131_Prepare/glob-efd083f9b00eb987f01181c24e49dc7c, command failed: AccessDeniedException: 403 Forbidden
CommandException: 1 file/object could not be transferred.
2016/04/29 13:07:49 E: command failed: AccessDeniedException: 403 Forbidden
AccessDeniedException: 403 Forbidden
CommandException: 2 files/objects could not be transferred.
(exit status 1)
2016/04/29 13:07:49 W: cp failed: gsutil -o Credentials:gs_oauth2_refresh_token=1/xxxxxxxxx -o OAuth2:client_id=972537320227-dipecl4g8mftumf2edelamfrf76tolok.apps.googleusercontent.com -o OAuth2:client_secret=xxxxxxxxx-H2lP -q -m cp /mnt/local-disk/interval_split_base.* gs://fc-328e1dba-07f7-49e5-9034-33f47fa027cc/239fa60b-b6a3-4832-bd54-29bde10169c1/CallingGroupWorkflow/a261a132-edfb-4f52-aeb1-11c5092dfd8a/call-CallSomaticMutations_131_Prepare/glob-efd083f9b00eb987f01181c24e49dc7c, command failed: AccessDeniedException: 403 Forbidden
AccessDeniedException: 403 Forbidden
CommandException: 2 files/objects could not be transferred.
2016/04/29 13:07:54 I: Done copying files.
2016/04/29 13:07:54 I: Copying /var/log/google-genomics/*.log to gs://fc-328e1dba-07f7-49e5-9034-33f47fa027cc/239fa60b-b6a3-4832-bd54-29bde10169c1/CallingGroupWorkflow/a261a132-edfb-4f52-aeb1-11c5092dfd8a/call-CallSomaticMutations_131_Prepare/
So it says there was an error de-localizing files....BUT in the bucket, I see the files!
Buckets/fc-328e1dba-07f7-49e5-9034-33f47fa027cc/239fa60b-b6a3-4832-bd54-29bde10169c1/CallingGroupWorkflow/a261a132-edfb-4f52-aeb1-11c5092dfd8a/call-CallSomaticMutations_131_Prepare/glob-efd083f9b00eb987f01181c24e49dc7c
Filter by prefix...
Name Size Type Last modified Share publicly
interval_split_base.1.interval_list 697.46 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.10.interval_list 301.4 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.11.interval_list 403.13 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.12.interval_list 415 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.13.interval_list 143.25 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.14.interval_list 248.94 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.15.interval_list 265.14 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.16.interval_list 304.49 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.17.interval_list 430.92 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.18.interval_list 116.38 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.19.interval_list 418.67 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.2.interval_list 534.8 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.20.interval_list 186.61 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.21.interval_list 85.07 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.22.interval_list 166.12 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.3.interval_list 422.83 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.4.interval_list 279.31 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.5.interval_list 317.54 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.6.interval_list 355.1 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.7.interval_list 340.78 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.8.interval_list 251.76 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.9.interval_list 293.9 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.X.interval_list 262.28 KB application/octet-stream 4/29/16, 9:07 AM
interval_split_base.Y.interval_list 34.05 KB application/octet-stream 4/29/16, 9:07 AM
I came back in the morning from leaving at the end of the day yesterday and submitted the job. Though I was able to submit the job this morning, the error was that the refresh token seemingly didn't allow permission to delocalize. Did the token on the back-end not get updated? After all, I was able to submit the job! Given that the files in the bucket are there, the copy must have take place seemingly on the 3rd try....but if that's true then why did the task fail?
CallingGroupWorkflow.CallSomaticMutations_131_PrepareHide
Call #1:
ID:operations/EPWc_JDGKhj65Z_FquCK_jAg8J7JsoIWKg9wcm9kdWN0aW9uUXVldWU
Status:Failed
Started:04/29/2016 at 9:04:43 AM (21 minutes ago)
Ended:04/29/2016 at 9:08:07 AM (18 minutes ago)
stdout:239fa60b-b6a3-4832-bd54-29bde10169c1/CallingGroupWorkflow/a261a132-edfb-4f52-aeb1-11c5092dfd8a/call-CallSomaticMutations_131_Prepare/CallSomaticMutations_131_Prepare-stdout.log
stderr:239fa60b-b6a3-4832-bd54-29bde10169c1/CallingGroupWorkflow/a261a132-edfb-4f52-aeb1-11c5092dfd8a/call-CallSomaticMutations_131_Prepare/CallSomaticMutations_131_Prepare-stderr.log
Inputs:Show
Outputs:None
The return code is zero!
wm8b1-75c:~ esalinas$ gsutil cat gs://fc-328e1dba-07f7-49e5-9034-33f47fa027cc/239fa60b-b6a3-4832-bd54-29bde10169c1/CallingGroupWorkflow/a261a132-edfb-4f52-aeb1-11c5092dfd8a/call-CallSomaticMutations_131_Prepare/CallSomaticMutations_131_Prepare-rc.txt
0
The split files (interval_split_base.*.interval_list) are the only outputs and they got written back to the bucket and they were written.