my dbGAP access has seemingly been whimsical. Going out unexpectedly.
See also http://gatkforums.broadinstitute.org/firecloud/discussion/8894/cannot-access-controlled-access-workspaces-on-firecloud#latest
I started workflows in FC in a controlled access workspace. Then, in the middle of the day, my dbGAP link was lost. It was maybe an hour or two or so ago. After I link my dbGAP back (by going to my profile page and several times trying to link) my linkage is restored I find a workflow failed!
Two scatter-tasks of a 25-way scatter failed. In the terminal transcript below you can see how they both started having permission denied errors at around the same time 18:04!
wm8b1-75c:dm esalinas$ for LOG in `gsutil ls gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/**|ggrep -Pi 'log$'|ggrep -Piv 'std' `; do echo $LOG ; gsutil cat $LOG | egrep -ic 'forbidden' ; gsutil cat $LOG | egrep -C 5 -m 3 -i 'forbidden' ; done ;
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-0/Mutect1_Task-0.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-1/Mutect1_Task-1.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-10/Mutect1_Task-10.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-11/Mutect1_Task-11.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-12/Mutect1_Task-12.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-13/Mutect1_Task-13.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-14/Mutect1_Task-14.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-15/Mutect1_Task-15.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-16/Mutect1_Task-16.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-17/Mutect1_Task-17.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-18/Mutect1_Task-18.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-19/Mutect1_Task-19.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-2/Mutect1_Task-2.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-20/Mutect1_Task-20.log
632
2017/01/26 17:54:57 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-20/
2017/01/26 17:59:57 I: Copying /var/log/google-genomics/*.log to gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-20/
2017/01/26 17:59:57 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-20/
2017/01/26 18:04:57 I: Copying /var/log/google-genomics/*.log to gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-20/
2017/01/26 18:04:57 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-20/
2017/01/26 18:04:58 E: command failed: AccessDeniedException: 403 Forbidden
AccessDeniedException: 403 Forbidden
AccessDeniedException: 403 Forbidden
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-21/Mutect1_Task-21.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-22/Mutect1_Task-22.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-23/Mutect1_Task-23.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/Mutect1_Task-24.log
476
2017/01/26 17:54:46 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/
2017/01/26 17:59:46 I: Copying /var/log/google-genomics/*.log to gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/
2017/01/26 17:59:46 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/
2017/01/26 18:04:46 I: Copying /var/log/google-genomics/*.log to gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/
2017/01/26 18:04:46 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/
2017/01/26 18:04:47 E: command failed: AccessDeniedException: 403 Forbidden
AccessDeniedException: 403 Forbidden
AccessDeniedException: 403 Forbidden
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/out.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-3/Mutect1_Task-3.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-4/Mutect1_Task-4.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-5/Mutect1_Task-5.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-6/Mutect1_Task-6.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-7/Mutect1_Task-7.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-8/Mutect1_Task-8.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-9/Mutect1_Task-9.log
0
wm8b1-75c:dm esalinas$
I noticed previous workflows (see http://gatkforums.broadinstitute.org/firecloud/discussion/8907/workflow-wf-running-in-controlled-access-ca-workspace-fail-if-ca-turned-off-during-wf-execution#latest ) also seemingly have permission issue at the same time-of-day (near 18:00) but two days before (on the 24, Tuesday, not 26th, today).
What's going on? The seemingly whimsical linkage makes workflow success harder to achieve