I am writing a WDL script that uses the broadinstitute/genomes-in-the-cloud Docker container, and I am encountering the following perplexing error:
[bwt_restore_bwt] Failed to allocate -40 bytes at bwt.c line 452: Cannot allocate memory
This is particularly perplexing because the script that I am running was working perfectly fine, at least past this step (I can't even get past the first step). I appear to have plenty of memory available (+10GB) and Docker itself is allowed to use up to 14G of that. Any ideas?
Full output:
java -jar /Applications/cromwell-0.19.3.jar run SomaticVariantCaller_v1.wdl SV_inputs.json
[2016-08-29 11:00:37,910] [info] Slf4jLogger started
[2016-08-29 11:00:37,930] [info] RUN sub-command
[2016-08-29 11:00:37,931] [info] WDL file: SomaticVariantCaller_v1.wdl
[2016-08-29 11:00:37,931] [info] Inputs: SV_inputs.json
[2016-08-29 11:00:37,959] [info] SingleWorkflowRunnerActor: launching workflow
[2016-08-29 11:00:38,383] [info] Running with database db.url = jdbc:hsqldb:mem:33bc94e8-433a-44e5-913d-eaa29e9d1f24;shutdown=false;hsqldb.tx=mvcc
[2016-08-29 11:00:40,866] [info] WorkflowManagerActor submitWorkflow input id = None, effective id = 1afabde6-7bb7-4abf-bdb1-c1961894f3a6
[2016-08-29 11:00:41,133] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,133] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,134] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,134] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,134] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,135] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,135] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,136] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,136] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,136] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,137] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,137] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,137] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,138] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,138] [warn] Found unsupported keys for backend 'LOCAL': memory
[2016-08-29 11:00:41,139] [warn] Found unsupported keys for backend 'LOCAL': memory
Aug 29, 2016 11:00:42 AM com.google.api.client.googleapis.services.AbstractGoogleClient
WARNING: Application name is not set. Call Builder#setApplicationName.
[2016-08-29 11:00:42,96] [info] Updating WorkflowManager state. New Data: (1afabde6-7bb7-4abf-bdb1-c1961894f3a6,Actor[akka://cromwell-system/user/WorkflowManagerActor/WorkflowActor-1afabde6-7bb7-4abf-bdb1-c1961894f3a6#1160933805])
[2016-08-29 11:00:42,115] [info] WorkflowActor [1afabde6]: Start(Some(Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor#29990427])) message received
[2016-08-29 11:00:42,348] [info] WorkflowActor [1afabde6]: ExecutionStoreCreated(Start(Some(Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor#29990427]))) message received
[2016-08-29 11:00:42,349] [info] SingleWorkflowRunnerActor: workflow ID 1afabde6-7bb7-4abf-bdb1-c1961894f3a6
[2016-08-29 11:00:42,357] [info] WorkflowActor [1afabde6]: Beginning transition from Submitted to Running.
[2016-08-29 11:00:42,358] [info] WorkflowActor [1afabde6]: transitioning from Submitted to Running.
[2016-08-29 11:00:42,364] [info] WorkflowActor [1afabde6]: starting calls: SomaticVariants.AlignNormal, SomaticVariants.AlignTumor
[2016-08-29 11:00:42,365] [info] WorkflowActor [1afabde6]: persisting status of AlignNormal to Starting.
[2016-08-29 11:00:42,383] [warn] Found unsupported keys for backend 'LOCAL': bootDiskSizeGb, cpu, disks, memory, preemptible, zones
[2016-08-29 11:00:42,397] [info] WorkflowActor [1afabde6]: persisting status of AlignTumor to Starting.
[2016-08-29 11:00:42,400] [warn] Found unsupported keys for backend 'LOCAL': bootDiskSizeGb, cpu, disks, memory, preemptible, zones
[2016-08-29 11:00:42,476] [info] WorkflowActor [1afabde6]: inputs for call 'AlignTumor':
Reference_sa -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta.64.sa)
Reference_bwt -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta.64.bwt)
Reference_fai -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta.fai)
Reference_amb -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta.64.amb)
sampleName -> WdlString(Test)
Reference_pac -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta.64.pac)
Input_1 -> WdlSingleFile(sub1t.fastq)
Reference_fa -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta)
Input_2 -> WdlSingleFile(sub2t.fastq)
Reference_ann -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta.64.ann)
type -> WdlString(Tumor)
[2016-08-29 11:00:42,479] [info] WorkflowActor [1afabde6]: created call actor for AlignTumor.
[2016-08-29 11:00:42,480] [info] WorkflowActor [1afabde6]: inputs for call 'AlignNormal':
Reference_sa -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta.64.sa)
Reference_bwt -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta.64.bwt)
Reference_fai -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta.fai)
Reference_amb -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta.64.amb)
sampleName -> WdlString(Test)
Reference_pac -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta.64.pac)
Input_1 -> WdlSingleFile(sub1.fastq)
Reference_fa -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta)
Input_2 -> WdlSingleFile(sub2.fastq)
Reference_ann -> WdlSingleFile(hg19/Homo_sapiens_assembly38.fasta.64.ann)
type -> WdlString(Normal)
[2016-08-29 11:00:42,481] [info] WorkflowActor [1afabde6]: created call actor for AlignNormal.
[2016-08-29 11:00:42,497] [warn] Found unsupported keys for backend 'LOCAL': bootDiskSizeGb, cpu, disks, memory, preemptible, zones
[2016-08-29 11:00:42,497] [warn] Found unsupported keys for backend 'LOCAL': bootDiskSizeGb, cpu, disks, memory, preemptible, zones
[2016-08-29 11:00:42,503] [info] WorkflowActor [1afabde6]: persisting status of AlignTumor to Running.
[2016-08-29 11:00:42,504] [info] WorkflowActor [1afabde6]: persisting status of AlignNormal to Running.
[2016-08-29 11:00:44,641] [info] LocalBackend [1afabde6:AlignTumor]: /bwa.kit/bwa mem -Mt 4 /root/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor/Users/dwells/Documents/Summer2016_OKR/Obj1.3-WDLDockerPipelines/SomaticVariantCaller/hg19/Homo_sapiens_assembly38.fasta /root/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor/Users/dwells/Documents/Summer2016_OKR/Obj1.3-WDLDockerPipelines/SomaticVariantCaller/sub1t.fastq /root/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor/Users/dwells/Documents/Summer2016_OKR/Obj1.3-WDLDockerPipelines/SomaticVariantCaller/sub2t.fastq > output.sam
[2016-08-29 11:00:44,641] [info] LocalBackend [1afabde6:AlignNormal]: /bwa.kit/bwa mem -Mt 4 /root/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal/Users/dwells/Documents/Summer2016_OKR/Obj1.3-WDLDockerPipelines/SomaticVariantCaller/hg19/Homo_sapiens_assembly38.fasta /root/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal/Users/dwells/Documents/Summer2016_OKR/Obj1.3-WDLDockerPipelines/SomaticVariantCaller/sub1.fastq /root/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal/Users/dwells/Documents/Summer2016_OKR/Obj1.3-WDLDockerPipelines/SomaticVariantCaller/sub2.fastq > output.sam
[2016-08-29 11:00:45,751] [info] LocalBackend [1afabde6:AlignTumor]: Return code: 1
[2016-08-29 11:00:45,759] [info] LocalBackend [1afabde6:AlignNormal]: Return code: 1
[2016-08-29 11:00:45,762] [error] CallActor [1afabde6:AlignNormal]: Failing call: Call SomaticVariants.AlignNormal, Workflow 1afabde6-7bb7-4abf-bdb1-c1961894f3a6: return code was 1
Full command was: "/bin/bash" "-c" "cat cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal/script | docker run --rm -v /Users/dwells/Documents/Summer2016_OKR/Obj1.3-WDLDockerPipelines/SomaticVariantCaller/cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal:/root/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal -i broadinstitute/genomes-in-the-cloud /bin/bash <&0"
Contents of cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal/stderr:
[bwt_restore_bwt] Failed to allocate -40 bytes at bwt.c line 452: Cannot allocate memory
java.lang.Exception: Call SomaticVariants.AlignNormal, Workflow 1afabde6-7bb7-4abf-bdb1-c1961894f3a6: return code was 1
Full command was: "/bin/bash" "-c" "cat cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal/script | docker run --rm -v /Users/dwells/Documents/Summer2016_OKR/Obj1.3-WDLDockerPipelines/SomaticVariantCaller/cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal:/root/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal -i broadinstitute/genomes-in-the-cloud /bin/bash <&0"
Contents of cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal/stderr:
[bwt_restore_bwt] Failed to allocate -40 bytes at bwt.c line 452: Cannot allocate memory
at cromwell.engine.backend.local.LocalBackend.cromwell$engine$backend$local$LocalBackend$$runSubprocess(LocalBackend.scala:220)
at cromwell.engine.backend.local.LocalBackend$$anonfun$execute$1.apply(LocalBackend.scala:119)
at cromwell.engine.backend.local.LocalBackend$$anonfun$execute$1.apply(LocalBackend.scala:113)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2016-08-29 11:00:45,763] [error] CallActor [1afabde6:AlignTumor]: Failing call: Call SomaticVariants.AlignTumor, Workflow 1afabde6-7bb7-4abf-bdb1-c1961894f3a6: return code was 1
Full command was: "/bin/bash" "-c" "cat cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor/script | docker run --rm -v /Users/dwells/Documents/Summer2016_OKR/Obj1.3-WDLDockerPipelines/SomaticVariantCaller/cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor:/root/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor -i broadinstitute/genomes-in-the-cloud /bin/bash <&0"
Contents of cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor/stderr:
[bwt_restore_bwt] Failed to allocate -40 bytes at bwt.c line 452: Cannot allocate memory
java.lang.Exception: Call SomaticVariants.AlignTumor, Workflow 1afabde6-7bb7-4abf-bdb1-c1961894f3a6: return code was 1
Full command was: "/bin/bash" "-c" "cat cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor/script | docker run --rm -v /Users/dwells/Documents/Summer2016_OKR/Obj1.3-WDLDockerPipelines/SomaticVariantCaller/cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor:/root/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor -i broadinstitute/genomes-in-the-cloud /bin/bash <&0"
Contents of cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor/stderr:
[bwt_restore_bwt] Failed to allocate -40 bytes at bwt.c line 452: Cannot allocate memory
at cromwell.engine.backend.local.LocalBackend.cromwell$engine$backend$local$LocalBackend$$runSubprocess(LocalBackend.scala:220)
at cromwell.engine.backend.local.LocalBackend$$anonfun$execute$1.apply(LocalBackend.scala:119)
at cromwell.engine.backend.local.LocalBackend$$anonfun$execute$1.apply(LocalBackend.scala:113)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2016-08-29 11:00:45,764] [info] WorkflowActor [1afabde6]: persisting status of AlignNormal to Failed.
[2016-08-29 11:00:45,764] [error] WorkflowActor [1afabde6]: Call SomaticVariants.AlignNormal, Workflow 1afabde6-7bb7-4abf-bdb1-c1961894f3a6: return code was 1
Full command was: "/bin/bash" "-c" "cat cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal/script | docker run --rm -v /Users/dwells/Documents/Summer2016_OKR/Obj1.3-WDLDockerPipelines/SomaticVariantCaller/cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal:/root/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal -i broadinstitute/genomes-in-the-cloud /bin/bash <&0"
Contents of cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignNormal/stderr:
[bwt_restore_bwt] Failed to allocate -40 bytes at bwt.c line 452: Cannot allocate memory
[2016-08-29 11:00:45,793] [info] WorkflowActor [1afabde6]: persisting status of AlignTumor to Failed.
[2016-08-29 11:00:45,794] [error] WorkflowActor [1afabde6]: Call SomaticVariants.AlignTumor, Workflow 1afabde6-7bb7-4abf-bdb1-c1961894f3a6: return code was 1
Full command was: "/bin/bash" "-c" "cat cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor/script | docker run --rm -v /Users/dwells/Documents/Summer2016_OKR/Obj1.3-WDLDockerPipelines/SomaticVariantCaller/cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor:/root/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor -i broadinstitute/genomes-in-the-cloud /bin/bash <&0"
Contents of cromwell-executions/SomaticVariants/1afabde6-7bb7-4abf-bdb1-c1961894f3a6/call-AlignTumor/stderr:
[bwt_restore_bwt] Failed to allocate -40 bytes at bwt.c line 452: Cannot allocate memory
[2016-08-29 11:00:45,806] [info] WorkflowActor [1afabde6]: Beginning transition from Running to Failed.
[2016-08-29 11:00:45,814] [info] WorkflowActor [1afabde6]: transitioning from Running to Failed.
[2016-08-29 11:00:45,815] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2016-08-29 11:00:45,816] [error] Workflow 1afabde6-7bb7-4abf-bdb1-c1961894f3a6 transitioned to state Failed
java.lang.Throwable: Workflow 1afabde6-7bb7-4abf-bdb1-c1961894f3a6 transitioned to state Failed
at cromwell.engine.workflow.SingleWorkflowRunnerActor$RunnerData.addFailure(SingleWorkflowRunnerActor.scala:42)
at cromwell.engine.workflow.SingleWorkflowRunnerActor$$anonfun$2.applyOrElse(SingleWorkflowRunnerActor.scala:89)
at cromwell.engine.workflow.SingleWorkflowRunnerActor$$anonfun$2.applyOrElse(SingleWorkflowRunnerActor.scala:80)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at akka.actor.FSM$class.processEvent(FSM.scala:604)
at cromwell.engine.workflow.SingleWorkflowRunnerActor.akka$actor$LoggingFSM$$super$processEvent(SingleWorkflowRunnerActor.scala:53)
at akka.actor.LoggingFSM$class.processEvent(FSM.scala:734)
at cromwell.engine.workflow.SingleWorkflowRunnerActor.processEvent(SingleWorkflowRunnerActor.scala:53)
at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:598)
at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:592)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at cromwell.engine.workflow.SingleWorkflowRunnerActor.aroundReceive(SingleWorkflowRunnerActor.scala:53)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Workflow 1afabde6-7bb7-4abf-bdb1-c1961894f3a6 transitioned to state Failed