I tried to do "size arithmetic" in WDL for disk size calculation.
The main objective of the size arithmetic is to calculate size of VM disk "manually" using WDL like this:
call some_task {
input:
a=a,
b=b,
....
diskGB=size(a,"G")+size(b,"G")+....+1
}
The WDL pasted here did not run in FC. I found no message by inspection in the network tab of the developer tools when attempting to do so. Neither did I find a workflow log. I attempted to run the WDL locally and got an error. The error is below from the paste of the bash session transcript. The WDL and its inputs are also there. There is an error about the input (which is also the WDL) not being found, but it's there, so this seems confusing. I tried changing the "1" to "1.0" and an error was still observed. I made the change thinking this might be an int/float issue.
wm8b1-75c:arith_explore esalinas$
wm8b1-75c:arith_explore esalinas$ java -jar ~/broadinstitute/cromwell-25.jar run a.wdl
[2017-04-05 13:19:30,20] [info] Slf4jLogger started
[2017-04-05 13:19:30,26] [info] RUN sub-command
[2017-04-05 13:19:30,26] [info] WDL file: /Users/esalinas/arith_explore/a.wdl
[2017-04-05 13:19:30,26] [info] Inputs: /Users/esalinas/arith_explore/a.inputs
[2017-04-05 13:19:30,30] [info] SingleWorkflowRunnerActor: Submitting workflow
[2017-04-05 13:19:30,34] [info] Workflow 62b6463e-0a09-4975-8e00-b1c120613718 submitted.
[2017-04-05 13:19:30,34] [info] SingleWorkflowRunnerActor: Workflow submitted 62b6463e-0a09-4975-8e00-b1c120613718
[2017-04-05 13:19:30,77] [info] Running with database db.url = jdbc:hsqldb:mem:484d2d53-7ca1-4828-8790-22991bc6f3d3;shutdown=false;hsqldb.tx=mvcc
[2017-04-05 13:19:35,46] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2017-04-05 13:19:35,47] [info] [RenameWorkflowOptionsInMetadata] 100%
[2017-04-05 13:19:35,51] [info] Metadata summary refreshing every 2 seconds.
[2017-04-05 13:19:35,97] [info] 1 new workflows fetched
[2017-04-05 13:19:35,98] [info] WorkflowManagerActor Starting workflow 62b6463e-0a09-4975-8e00-b1c120613718
[2017-04-05 13:19:35,98] [info] WorkflowManagerActor Successfully started WorkflowActor-62b6463e-0a09-4975-8e00-b1c120613718
[2017-04-05 13:19:35,98] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2017-04-05 13:19:36,20] [info] MaterializeWorkflowDescriptorActor [62b6463e]: Call-to-Backend assignments: w.bam_task -> Local
[2017-04-05 13:19:36,27] [warn] Local [62b6463e]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2017-04-05 13:19:36,46] [info] WorkflowExecutionActor-62b6463e-0a09-4975-8e00-b1c120613718 [62b6463e]: Starting calls: w.bam_task:NA:1
[2017-04-05 13:19:36,53] [info] Message [cromwell.subworkflowstore.SubWorkflowStoreActor$SubWorkflowStoreCompleteSuccess] from Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/$b#-1909986789] to Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-62b6463e-0a09-4975-8e00-b1c120613718#-2128866478] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2017-04-05 13:19:36,53] [error] WorkflowManagerActor Workflow 62b6463e-0a09-4975-8e00-b1c120613718 failed (during ExecutingWorkflowState): /Users/esalinas/arith_explore/cromwell-executions/w/62b6463e-0a09-4975-8e00-b1c120613718/call-bam_task/execution/a.wdl
java.nio.file.NoSuchFileException: /Users/esalinas/arith_explore/cromwell-executions/w/62b6463e-0a09-4975-8e00-b1c120613718/call-bam_task/execution/a.wdl
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
at java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:72)
at java.nio.file.Files.walk(Files.java:3574)
at better.files.File.walk(File.scala:451)
at better.files.File.size(File.scala:485)
at cromwell.core.path.BetterFileMethods$class.size(BetterFileMethods.scala:313)
at cromwell.core.path.DefaultPath.size(DefaultPathBuilder.scala:53)
at cromwell.backend.wdl.ReadLikeFunctions$$anonfun$fileSize$1$1$$anonfun$apply$4.apply(ReadLikeFunctions.scala:96)
at cromwell.backend.wdl.ReadLikeFunctions$$anonfun$fileSize$1$1$$anonfun$apply$4.apply(ReadLikeFunctions.scala:95)
at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Success.map(Try.scala:237)
at cromwell.backend.wdl.ReadLikeFunctions$$anonfun$fileSize$1$1.apply(ReadLikeFunctions.scala:95)
at cromwell.backend.wdl.ReadLikeFunctions$$anonfun$fileSize$1$1.apply(ReadLikeFunctions.scala:94)
at scala.util.Success.flatMap(Try.scala:231)
at cromwell.backend.wdl.ReadLikeFunctions$class.fileSize$1(ReadLikeFunctions.scala:94)
at cromwell.backend.wdl.ReadLikeFunctions$class.size(ReadLikeFunctions.scala:101)
at cromwell.backend.standard.StandardExpressionFunctions.size(StandardExpressionFunctions.scala:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at wdl4s.expression.WdlFunctions$$anonfun$getFunction$1.apply(WdlFunctions.scala:12)
at wdl4s.expression.WdlFunctions$$anonfun$getFunction$1.apply(WdlFunctions.scala:12)
at wdl4s.expression.ValueEvaluator.evaluate(ValueEvaluator.scala:182)
at wdl4s.expression.ValueEvaluator.evaluate(ValueEvaluator.scala:51)
at wdl4s.expression.ValueEvaluator.evaluate(ValueEvaluator.scala:51)
at wdl4s.WdlExpression$.evaluate(WdlExpression.scala:85)
at wdl4s.WdlExpression.evaluate(WdlExpression.scala:161)
at wdl4s.Call$$anonfun$11$$anonfun$apply$6.apply(Call.scala:156)
at wdl4s.Call$$anonfun$11$$anonfun$apply$6.apply(Call.scala:155)
at scala.util.Success.flatMap(Try.scala:231)
at wdl4s.Call$$anonfun$11.apply(Call.scala:155)
at wdl4s.Call$$anonfun$11.apply(Call.scala:154)
at scala.util.Success.flatMap(Try.scala:231)
at wdl4s.Call.wdl4s$Call$$lookup$2(Call.scala:154)
at wdl4s.Call$$anonfun$lookupFunction$1.apply(Call.scala:199)
at wdl4s.Call$$anonfun$lookupFunction$1.apply(Call.scala:199)
at wdl4s.Call$$anonfun$5.apply(Call.scala:108)
at wdl4s.Call$$anonfun$5.apply(Call.scala:108)
at scala.util.Try$.apply(Try.scala:192)
at wdl4s.Call.wdl4s$Call$$doDeclaration$1(Call.scala:108)
at wdl4s.Call$$anonfun$6.apply(Call.scala:121)
at wdl4s.Call$$anonfun$6.apply(Call.scala:121)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
at wdl4s.Call.evaluateTaskInputs(Call.scala:121)
at cromwell.engine.workflow.lifecycle.execution.preparation.CallPreparation$.resolveAndEvaluateInputs(CallPreparation.scala:36)
at cromwell.engine.workflow.lifecycle.execution.preparation.JobPreparationActor.evaluateInputsAndAttributes(JobPreparationActor.scala:78)
at cromwell.engine.workflow.lifecycle.execution.preparation.JobPreparationActor$$anonfun$1.applyOrElse(JobPreparationActor.scala:48)
at cromwell.engine.workflow.lifecycle.execution.preparation.JobPreparationActor$$anonfun$1.applyOrElse(JobPreparationActor.scala:46)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at akka.actor.FSM$class.processEvent(FSM.scala:663)
at cromwell.engine.workflow.lifecycle.execution.preparation.JobPreparationActor.processEvent(JobPreparationActor.scala:21)
at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:657)
at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:651)
at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
at cromwell.engine.workflow.lifecycle.execution.preparation.JobPreparationActor.aroundReceive(JobPreparationActor.scala:21)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2017-04-05 13:19:36,53] [info] WorkflowManagerActor WorkflowActor-62b6463e-0a09-4975-8e00-b1c120613718 is in a terminal state: WorkflowFailedState
[2017-04-05 13:19:38,15] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
Workflow 62b6463e-0a09-4975-8e00-b1c120613718 transitioned to state Failed
wm8b1-75c:arith_explore esalinas$ cat a.inputs
{
"w.bam": "a.wdl",
"w.bai": "a.inputs"
}
wm8b1-75c:arith_explore esalinas$ head -110000 a.wdl
task bam_task {
File bam
File bai
Float diskGB
command <<<
cp -v ${bam} copy.bam
cp -v ${bai} copy.bai
>>>
output {
File copy_bam="copy.bam"
File copy_bai="copy.bai"
}
runtime {
disks: "local-disk ${diskGB} HDD"
docker : "ubuntu:14.04"
}
}
workflow w
{
File bam
File bai
call bam_task {
input:
bam=bam,
bai=bai,
diskGB=size(bam,"G")+size(bai,"G")+1
}
}
wm8b1-75c:arith_explore esalinas$
PS, I tried to upload a WDL file but got a message "(a.wdl) Uploaded file type is not allowed."