Using gsutil -m: how do I specify multiple sources and multiple destinations
I want to download 15,000 files from my FireCloud bucket, all of which have the same filename but different paths, eg gs://fc-mybucket/MyTool/*/MySubtask/output.vcf It's too slow to make 15,000 calls...
View ArticleIssues with pairsets in absolute
Hello I'm running absolute and having issues with running it on the pairset. I can run it successfully on individual pairs but it fails when it runs on a pairset using this.pairs. I am able to use the...
View ArticleCreate directory command in WDL
Hi! One of the required inputs of my tool is output directory (--outdir), so I'm trying to add a command to create a new directory within my WDL script (mkdir ${outdir}) but it doesn't seem to work....
View ArticleDefining multiple outputs
Hi! My command creates multiple output files and I'm not sure how to properly define them in output section of my WDL script. Here is the script: Rscript ${pcn_extdata}/Coverage.R \ --outdir ${outdir}...
View ArticleQuestions about the very basics, please help
Hi there, I've been trying my best to work through the tutorials, but it's been hard for me to follow. I'd like to do a very simple task just to make sure I understand the parts correctly. Let's say...
View ArticleOauth Authentication for FireCloud via API calls
Hi There, We are integrating our system to fireCloud via APIs(https://api.firecloud.org/). Now to run APIs we need to provide OAuth Token. While testing it via swagger it allows me to log in using...
View ArticleNo machines available error
Hi, I'm trying to interpret the following error: The job was stopped before the command finished. PAPI error code 5. no machines available Does this mean no machines were available from the cloud, or...
View ArticleDeleting Project Data
Hello, I attempted to delete a project and move the data to archival storage-- however my google cloud console activity reports "Failed: Update Bucket" over the many days since I submitted the request....
View ArticleDoes FireCloud always allocate VMs from google's us-central1 region
We want to move some data off of our workspaces' multiregional cloud storage to regional storage in order to reduce our storage costs. In order to be able to continue running analyses on the data...
View ArticleBUG: Groups addUserToGroup API response does not conform to swagger docs.
The swagger documentation indicates that a 404 response indicates a User not found error. However entering an unknown user elicits a 400 response. A 404 response is given when the group does not exist.
View ArticleHow do I do a bulk download of multiple methods?
I am using the firecloud online portal and I need to download all the methods from my search result. Is there a way to do this either from the online portal or from the command line?
View ArticleWDL cannot find parameter file that exists in same google bucket
I'm trying to run a simple WDL that runs "samtools mpileup", where one WDL parameter is the name of an intervals file. From FireCloud I enter the intervals filename as follows "...
View ArticleDone in the VM, but no output on firecloud
Hi FC team, From my stderr file, I learned that my task was finished for a couple hours. But now, the status is stucked by "submitted". my workspace:...
View Articlelimit swap space when running WDL using Terra
I've encountered a case where if a user doesn't specify enough RAM to the STAR aligner and STAR has to use swap space, the task will get stuck running indefinitely. Is there a way to limit swap? Thanks.
View ArticleSamtool 'non-existent file' stops the the...
Hello, I am trying to run a version of the joint-discovery-gatk4-local workflow slightly adjusted to run with a SLURM backend (I am running with gatk 4.0.12.0; the json and wdl files are both based on...
View ArticleClik here to view.
Issue with CPTAC2 Authorization
According to dbGaP, I'm authorized on TCGA, TARGET, CPTAC2, and CPTAC3: The GDC agrees: But FireCloud does not: I'm going to need to do work in FireCloud with CPTAC2 data soon, so it would be good to...
View Articlestorage.objects.list access error for freecredit project
Hi, I ran a modified method 3-Joint-Discovery from workspace help-gatk/Germline-SNPs-Indels-GATK4-hg38. The GVCF input files failed to import. Because it's a trial account, I couldn't modify the IAM...
View Articlemonitoring_image documentation
I see that Cromwell supports specifying a monitoring docker image (https://cromwell.readthedocs.io/en/stable/wf_options/Google/#google-pipelines-api-workflow-options). I set the monitoring_image to...
View ArticleFirecloud jobs stopping unexpectedly - PAPI error code 10. 14
I have submitted a large computation to FC with ~4000 shards across 4 calls, all tasks allowing a few preemptibles. I noticed a number of shards eventually failing with the following code: message:...
View ArticleTasks queued in Cromwell
I started a workflow on March 29th on ~600 samples that completed for several samples but now all the tasks seem to still be stuck in Cromwell (has been the case with almost no movement since...
View ArticleGoogleCloud Project permissions
Good morning, I am trying to add a lab member to my GoogleCloud Project but do not have appropriate permissions. I was reviewing my GoogleCloud billing project 'stoverlabbilling' and noticed that under...
View ArticleNew feature in FC: workflowFailureMode: "ContinueWhilePossible"
Hi, Is it possible to add "workflowFailureMode": "ContinueWhilePossible" as option when lunching analysis? It's particularly helpful in big computations when some jobs fail randomly PAPI error code 10....
View ArticleGenotypeGVCFs GATK 4.1.0.0 error bcf_update_format: Assertion `nps &&...
Hello! I am trying to run GenotypeGVCFs on a set of 12 pooled-seq (50 fish per pool, using a ploidy of 20 in GATK) whole genome samples. Unfortunately I have had recurrent java memory issues ( I...
View ArticleCannot find output when running docker container with cromwell
Hi there, I currently have a method that I now run with Docker and was hoping to put on Firecloud. The way the docker is currently structured, I run a wrapper script within the container and that...
View Articlejava.lang.Exception: The job was aborted from outside Cromwell
Hi Team, We are enabling Cromwell on AWS and using Cromwell version 39. While testing on five-dollar pipeline, we are facing subjected issue intermittently. There are instances wherein some tasks will...
View Articlemessage: Unable to complete JES Api Request; message: Request payload size...
When running a call to combine multiple files into one using a python script I see the following error: • message: Unable to complete JES Api Request • causedBy: • message: Request payload size exceeds...
View ArticleSubworkflow details cannot be expanded
Hi FC Team, As of last night, a few members of our group have noticed that subworkflow details cannot be expanded for running workflows when clicking the Show button next to each subworkflow on the job...
View ArticleSucessfully completed tasks still marked as running
I have several workflows in my workspaces that are marked as still running even though many of them have successfully completed for several hours now (expanding the individual cases reveals that they...
View ArticleNotebook terminal is freezing
My terminal window only stays stable for about 5 minutes. If I close and reopen the window works just fine. Notebook seems to be working just fine as well.
View ArticleProblem transitioning to new Google Billing Account after free trial
I followed the steps here. I get this error when I try to run a workflow: Task helloUncompressGZ.UncompressGZ:NA:1 failed. The job was stopped before the command finished. PAPI error code 7. Access Not...
View Articlefirecloud cant access project resources - PAPI error code 7, access not...
Hi - I'm seeing the following error when launching my workflow; message: Workflow failed causedBy: message: Task trinity_fusion_wf.TRINITY_FUSION_UC_TASK:NA:4 failed. The job was stopped before the...
View Articleproblem with permission to access to the google bucket associated with a...
Hello, When I try to access a google bucket associated with a Firecloud workspace, I get the following message: "You need the storage.objects.list permission to list objects in this bucket. Ask a...
View Articleparallel upload of data to google bucket in Firecloud workspace
Hello, We have a large set of .bam/.bai files to transfer to the google bucket associated with a Firecloud workspace. We plan to do the upload of the files in parallel, meaning that several .bam/.bai...
View Articleis recursive docker possible?
Hi, Is it possible to have a workflow run in firecloud as privileged, so it can call docker within the docker (recursively)? There's an externally developed pipeline that I'd want to run, where that...
View ArticleSwitching cluster runtime does not preserve the cluster configuration
I switched runtimes on my notebook cluster because I needed a larger computer. When I did that all of the software installed on the cluster disappeared. Is there anyway to image the cluster and...
View ArticleDoes the notebook cluster showdown if you logout of Terra?
I ran an ML job on the notebook cluster which was running fine but overnight. When I came back in the morning the cluster had shutdown and my job was killed. Is that the right behavior? Thanks, Ilya
View ArticleIs Terra/Firecloud down?
I can't login. Get a gcp error. "502. That’s an error. The server encountered a temporary error and could not complete your request. Please try again in 30 seconds. That’s all we know."
View ArticleClik here to view.
Error uploading samples table to workspace
I get the following when trying to upload a 7mb table to my workspace. Is it too large?
View ArticleHow to change billing account/billing project initially assigned to a workspace?
Hi, Could you please let me know how can I change a billing account/billing project that I initially assigned during generation of a workspace in Firecloud?
View ArticleFailure while processing large unmapped bam file as input to cromwell
Hi Team, We are running five dollar pipeline by running Cromwell(v39) on AWS. Whenever we are running our pipeline on small files(~300MB) process is proceeding fine with further processing. But,...
View ArticleHow to exit in the middle of a linear chain tasks in wdl
Hi, I have a sequential of tasks. If I found the output of task1 has an error message "No space on the disk", I would like to exit the analysis. Don't continue with the following tasks. Is that...
View ArticlePlink prefix and wdl
Hi, I am trying to create a WDL that runs plink and takes as an input a prefix to the bed/bim/fam file I am using Cromwell and docker to run it for example: plink -bfile prefix --recode --out prefix...
View ArticleImplementing GenotyGVCFs and GenomicsDBImport in WDL
This discussion was created from comments split from: Picard/GATK MergeVcfs throws errors.
View ArticleConfirm I am getting the right runtime and runtime options
How do I confirm that I am getting the right runtime and where are the different WDL runtime parameters/options described?
View ArticleCan I add the hash of docker container to my snapshot?
runtime { docker: "imagename":HASH memory: memGB disks: "local-disk 256 HDD" cpu: nCores } This would be useful for forcing versioning of the docker image in the snapshot.
View ArticleDocker pull error
I am getting the following docker pull error when working in a project that is using my google billing account instead of the free billing account. Task genoml.train:NA:1 failed. The job was stopped...
View ArticleClik here to view.
View VMs associated with running jobs in GCP
I am trying to view my jobs in GCP. For the free trial account, I do not have have any active jobs in my job history but I do have a VM running in GCP. It's been running since 4/17. See screenshot For...
View ArticleWorkFlow getting aborted intermittently without any exception
Hi Team, We are facing yet another issue with the large file(68GB) processing wherein workflow is getting aborted all of sudden without any exception on AWS enabled Cromwell. This behaviour is not...
View ArticleWhat is the plain text for an Array in Workspace Attributes
What is the plain text for an Array in Workspace Attributes? If not exist, how can I upload an array in my workspace attributes? Say I want to copy the "known_indels_array" attribute in the following...
View ArticleRunning costs on FireCloud
Hi, I'm trying to find out about how much it might cost to run a cohort of samples (500, 1000, or 2000) using germline GATK Best Practices Pipeline on FireCloud , and also how long the runs might take....
View Article