Tuesday had I relinked my FC account with my dbGAP access.
After the successful link I was able to enter some controlled access workspaces and launch a workflow.
For example, I entered a workspace and launched submission 8503c306-28d7-4e01-9c41-0cb1417f0d6d
The submission was launched Tuesday at/around 12:17:33 PM
The submission failed because a single scatter job was marked with failure. The other scatter jobs ("tasks") are marked "running" but they are complete as I see a "0"
When I investigate the reason for failure I see a of "Forbidden" error. In log segment below, there is some successful copying before 18:00 (6PM) but then around 18:00 I see "AccessDeniedException: 403 Forbidden"
2017/01/24 17:59:01 I: Copying gs://firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf to /mnt/local-disk/firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf
2017/01/24 17:59:01 I: Running command: sudo gsutil -q -m cp gs://firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf /mnt/local-disk/firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf
2017/01/24 18:00:19 I: Docker file /cromwell_root/firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf maps to host location /mnt/local-disk/firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf.
2017/01/24 18:00:19 I: Copying gs://firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf to /mnt/local-disk/firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf
2017/01/24 18:00:19 I: Running command: sudo gsutil -q -m cp gs://firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf /mnt/local-disk/firecloud-tcga-open-access/tutorial/reference/hg19_cosmic_v54_120711.vcf
2017/01/24 18:00:20 I: Docker file /cromwell_root/5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/DNA/WGS/BI/ILLUMINA/G92908.TCGA-A7-A26J-10A-01D-A46W-08.2.bam.bai maps to host location /mnt/local-disk/5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/DNA/WGS/BI/ILLUMINA/G92908.TCGA-A7-A26J-10A-01D-A46W-08.2.bam.bai.
2017/01/24 18:00:20 I: Copying gs://5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/DNA/WGS/BI/ILLUMINA/G92908.TCGA-A7-A26J-10A-01D-A46W-08.2.bam.bai to /mnt/local-disk/5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/DNA/WGS/BI/ILLUMINA/G92908.TCGA-A7-A26J-10A-01D-A46W-08.2.bam.bai
2017/01/24 18:00:20 I: Running command: sudo gsutil -q -m cp gs://5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/DNA/WGS/BI/ILLUMINA/G92908.TCGA-A7-A26J-10A-01D-A46W-08.2.bam.bai /mnt/local-disk/5aa919de-0aa0-43ec-9ec3-288481102b6d/tcga/BRCA/DNA/WGS/BI/ILLUMINA/G92908.TCGA-A7-A26J-10A-01D-A46W-08.2.bam.bai
2017/01/24 18:00:21 E: command failed: AccessDeniedException: 403 Forbidden
CommandException: 1 file/object could not be transferred.
(exit status 1)
In addition, some log scanning reports forbidden string seemingly between 18:00ish and 20:00ish
wm8b1-75c:dm esalinas$ gsutil ls gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/**|ggrep -Pi '\.log$'|ggrep -Piv 'std'|ggrep -Piv 'df.log'|ggrep -Piv 'dstat' | xargs -P 100 -I {} gsutil cat {}|ggrep -C 3 -Pi 'forbidden'|ggrep -Pio '^2017................'|sort|uniq -c|head
3 2017/01/24 17:23:33
3 2017/01/24 18:00:20
2 2017/01/24 18:00:21
1 2017/01/24 18:00:26
2 2017/01/24 18:00:27
1 2017/01/24 18:00:34
2 2017/01/24 18:00:35
1 2017/01/24 18:00:46
2 2017/01/24 18:00:47
1 2017/01/24 18:01:06
wm8b1-75c:dm esalinas$ gsutil ls gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/**|ggrep -Pi '\.log$'|ggrep -Piv 'std'|ggrep -Piv 'df.log'|ggrep -Piv 'dstat' | xargs -P 100 -I {} gsutil cat {}|ggrep -C 3 -Pi 'forbidden'|ggrep -Pio '^2017................'|sort|uniq -c|tail
3 2017/01/24 19:30:32
3 2017/01/24 19:45:40
3 2017/01/24 19:46:48
3 2017/01/24 20:10:34
1 2017/01/24 20:17:26
2 2017/01/24 20:17:27
3 2017/01/24 20:38:13
1 2017/01/24 20:40:23
2 2017/01/24 20:40:24
3 2017/01/24 20:47:56
After the submission my access/link to dbGap became unlinked and I had to re-link. If the linkage becomes unlinked (meaning that I cannot enter a workspace) is it true that if the linkage unlinks during a workflow that the workflow cannot read for localization? What about de-localization? It would seem that losing access also causes problems in de-localization. For example in the following log, there can be seen a "window" of permission denied:
wm8b1-75c:dm esalinas$
wm8b1-75c:dm esalinas$ gsutil cat gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/Mutect1_Task-3.log|ggrep -Pin '.'
1:2017/01/24 17:23:15 I: Switching to status: pulling-image
2:2017/01/24 17:23:15 I: Calling SetOperationStatus(pulling-image): &{ EKrtgIydKxi21tWMq-SNwt8BIN6K28SDHCoPcHJvZHVjdGlvblF1ZXVl [0xc4203e6780 0xc420404500] 2961810260898001727 [] []}
.......
#snipped for brevity
.......
69:2017/01/24 18:03:13 I: Copying /var/log/google-genomics/*.log to gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
70:2017/01/24 18:03:13 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
71:2017/01/24 18:08:13 I: Copying /var/log/google-genomics/*.log to gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
72:2017/01/24 18:08:13 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
73:2017/01/24 18:08:14 E: command failed: AccessDeniedException: 403 Forbidden
.......
#snipped for brevity
.......
235:AccessDeniedException: 403 Forbidden
236:CommandException: 3 files/objects could not be transferred.
238:2017/01/24 18:14:35 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
239:2017/01/24 18:14:37 E: command failed: AccessDeniedException: 403 Forbidden
240:AccessDeniedException: 403 Forbidden
241:AccessDeniedException: 403 Forbidden
242:CommandException: 3 files/objects could not be transferred.
243: (exit status 1)
244:2017/01/24 18:14:37 W: cp failed: gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/, command failed: AccessDeniedException: 403 Forbidden
245:AccessDeniedException: 403 Forbidden
246:AccessDeniedException: 403 Forbidden
.......
#snipped for brevity
.......
469:CommandException: 3 files/objects could not be transferred.
471:2017/01/24 18:25:25 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
472:2017/01/24 18:25:27 E: command failed: AccessDeniedException: 403 Forbidden
473:AccessDeniedException: 403 Forbidden
474:AccessDeniedException: 403 Forbidden
475:CommandException: 3 files/objects could not be transferred.
476: (exit status 1)
477:2017/01/24 18:25:27 W: cp failed: gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/, command failed: AccessDeniedException: 403 Forbidden
478:AccessDeniedException: 403 Forbidden
.......
#snipped for brevity
.......
689:AccessDeniedException: 403 Forbidden
690:AccessDeniedException: 403 Forbidden
691:CommandException: 3 files/objects could not be transferred.
693:2017/01/24 18:35:38 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
694:2017/01/24 18:35:39 E: command failed: AccessDeniedException: 403 Forbidden
695:AccessDeniedException: 403 Forbidden
696:AccessDeniedException: 403 Forbidden
697:CommandException: 3 files/objects could not be transferred.
698: (exit status 1)
699:2017/01/24 18:35:39 W: cp failed: gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/, command failed: AccessDeniedException: 403 Forbidden
700:AccessDeniedException: 403 Forbidden
.......
#snipped for brevity
.......
804:2017/01/24 18:40:12 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
805:2017/01/24 18:40:13 E: command failed: AccessDeniedException: 403 Forbidden
806:AccessDeniedException: 403 Forbidden
807:AccessDeniedException: 403 Forbidden
808:CommandException: 3 files/objects could not be transferred.
809: (exit status 1)
810:2017/01/24 18:40:13 W: cp failed: gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/, command failed: AccessDeniedException: 403 Forbidden
.......
#snipped for brevity
.......
885:AccessDeniedException: 403 Forbidden
886:CommandException: 3 files/objects could not be transferred.
887: (exit status 1)
888:2017/01/24 18:44:05 W: cp failed: gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/, command failed: AccessDeniedException: 403 Forbidden
889:AccessDeniedException: 403 Forbidden
890:AccessDeniedException: 403 Forbidden
891:CommandException: 3 files/objects could not be transferred.
893:2017/01/24 18:44:35 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
894:2017/01/24 18:44:36 E: command failed: AccessDeniedException: 403 Forbidden
895:AccessDeniedException: 403 Forbidden
.......
#snipped for brevity
.......
945:AccessDeniedException: 403 Forbidden
946:CommandException: 3 files/objects could not be transferred.
948:2017/01/24 18:47:12 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
949:2017/01/24 18:47:13 E: command failed: AccessDeniedException: 403 Forbidden
950:AccessDeniedException: 403 Forbidden
951:AccessDeniedException: 403 Forbidden
952:CommandException: 3 files/objects could not be transferred.
.......
#snipped for brevity
.......xception: 403 Forbidden
984:AccessDeniedException: 403 Forbidden
985:AccessDeniedException: 403 Forbidden
986:CommandException: 3 files/objects could not be transferred.
987: (exit status 1)
988:2017/01/24 18:48:36 W: cp failed: gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/, command failed: AccessDeniedException: 403 Forbidden
989:AccessDeniedException: 403 Forbidden
990:AccessDeniedException: 403 Forbidden
991:CommandException: 3 files/objects could not be transferred.
993:2017/01/24 18:48:48 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
994:2017/01/24 18:48:49 E: command failed: AccessDeniedException: 403 Forbidden
995:AccessDeniedException: 403 Forbidden
.......
#snipped for brevity
.......
1033:AccessDeniedException: 403 Forbidden
1034:AccessDeniedException: 403 Forbidden
1035:CommandException: 3 files/objects could not be transferred.
1037:2017/01/24 18:50:31 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
1038:2017/01/24 18:53:13 I: Copying /var/log/google-genomics/*.log to gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
1039:2017/01/24 18:53:13 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
.......
#snipped for brevity
.......
1095:2017/01/24 19:05:06 I: Done copying files.
1096:2017/01/24 19:05:06 I: Copying /var/log/google-genomics/*.log to gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
1097:2017/01/24 19:05:06 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/8503c306-28d7-4e01-9c41-0cb1417f0d6d/CallingGroup_Workflow/10ac2ad9-cbb3-4a6d-8447-c955b5dbe7b5/call-Mutect1_Task/shard-3/
wm8b1-75c:dm esalinas$
Can it be confirmed if linkage loss led to these "Forbidden" errors?
-eddie