Hi-
I have a wdl task that uses FISS python bindings to access the workspace data model. While this works fine locally, I receive a credential error when running in Firecloud:
Traceback (most recent call last):
File "/cromwell_root/fc-adaae650-a458-4c56-8a55-d96fa463a5c6/fdfd7400-ad04-4c26-81ed-7b2d6cf65723/w/2c60c47a-86f1-4a70-bcac-fef65af20fbe/call-getScript/cohortToSamples.py", line 12, in <module>
samples = fiss.fapi.get_entities('topmed-shared','topmed-shared', 'sample').json()
File "/usr/local/lib/python2.7/site-packages/firecloud/api.py", line 193, in get_entities
headers = _fiss_access_headers()
File "/usr/local/lib/python2.7/site-packages/firecloud/api.py", line 34, in _fiss_access_headers
credentials = GoogleCredentials.get_application_default()
File "/usr/local/lib/python2.7/site-packages/oauth2client/client.py", line 1288, in get_application_default
return GoogleCredentials._get_implicit_credentials()
File "/usr/local/lib/python2.7/site-packages/oauth2client/client.py", line 1278, in _get_implicit_credentials
raise ApplicationDefaultCredentialsError(ADC_HELP_MSG)
oauth2client.client.ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
Here's the python script and wdl:
import argparse
parser = argparse.ArgumentParser(description='Get a list of sample IDs for all input cohorts within a firecloud data model and print to a text file, one sample id per line',formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('--cohorts', nargs=1, default='', help='common separated list of cohorts that you would like the sample ids from')
parser.add_argument('--outfile_pref', nargs=1, default='', help='label for output file of sample ids')
args = parser.parse_args()
args.cohorts = args.cohorts[0].split(',')
out_file = args.outfile_pref[0]+'.txt'
from firecloud import fiss
samples = fiss.fapi.get_entities('topmed-shared','topmed-shared', 'sample').json()
sample_study_gen = (s['attributes']['participant']['entityName'] for s in samples if s['attributes']['study'] in cohorts)
with open(out_file, 'w') as f:
for p in sample_study_gen:
f.write(p+'\n')
task getScript {
command {
wget "https://raw.githubusercontent.com/manning-lab/topmed-t2d-glycemia-public/dev/methods/dataModel/cohortToSamples.py"
}
runtime {
docker: "tmajarian/alpine_wget@sha256:f3402d7cb7c5ea864044b91cfbdea20ebe98fc1536292be657e05056dbe5e3a4"
}
output {
File outscript = "cohortToSamples.py"
}
}
task runScript {
String cohorts
String label
File script
command {
python ${script} --cohorts ${cohorts} --outfile_pref ${label}
}
runtime {
docker: "broadgdac/fiss@sha256:a65324c8cf1edc769bee3195c798468defacefece3a3d551143706cd412e4c39"
disks: "local-disk 10 SSD"
memory: "2G"
}
output {
File out_file = "${label}.txt"
}
}
workflow w {
String these_cohorts
String this_label
call getScript
call runScript {
input: cohorts=this_cohorts, label=this_label, script=getScript.outscript
}
output {
File sample_list = runScript.out_file
}
}
I get why this error exists as far as authentication within each compute node goes but am wondering if there is a known workaround for this problem. It seems reasonable that a job launched from a given workspace should inherit the credentials necessary to access the workspace metadata. Depending on the answer, this might be a feature request.
Thanks