I launched a complex workflow (a version of CGA's production analysis workflow) on 402 sample pairs. 392 of the workflows successfully launched and completed, 10 of the workflows failed to launch, with errors coming from rawls. Below is one of the error reports:
ErrorReport(rawls,http call failed: https://cromwell1.dsde-prod.broadinstitute.org:443/api/workflows/v1/batch: TCP idle-timeout encountered on connection to [cromwell1.dsde-prod.broadinstitute.org:443], no bytes passed in the last 1 minute,Some(500 Internal Server Error),WrappedArray(),WrappedArray(akka.stream.impl.io.TcpIdleTimeout$$anonfun$1.applyOrElse(TcpStages.scala:389), akka.stream.impl.io.TcpIdleTimeout$$anonfun$1.applyOrElse(TcpStages.scala:387), scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34), akka.stream.impl.fusing.MapError$$anon$12.onUpstreamFailure(Ops.scala:285), akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:733), akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:616), akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:471), akka.stream.impl.fusing.GraphInterpreterShell.receive(ActorGraphInterpreter.scala:423), akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:603), akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:618), akka.actor.Actor.aroundReceive(Actor.scala:502), akka.actor.Actor.aroundReceive$(Actor.scala:500), akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:529), akka.actor.ActorCell.receiveMessage(ActorCell.scala:526), akka.actor.ActorCell.invoke(ActorCell.scala:495), akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257), akka.dispatch.Mailbox.run(Mailbox.scala:224), akka.dispatch.Mailbox.exec(Mailbox.scala:234), java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289), java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056), java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692), java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)),None)
I'd like to know whether this is expected behavior when launching submissions of this scale (e.g., hundreds of simultaneous workflows). Is this a problem with the quotas set on my FireCloud billing project? Regardless of whether this is expected behavior or a bug, are there plans to make the system more scalable, and if so, what is the expected timeline for these improvements?
Second, how do you recommend I work around these failures? I can re-run these individual workflows (possibly creating a pairset just containing these failed workflows), or re-launch the entire submission and rely on call caching to only run those jobs that failed to run in the previous submission. My concern about the later approach is that it will still launch all 402 workflows, creating the same load on rawls that triggered the earlier failures. Also, although the majority of the individual jobs will not be run (due to call caching), I believe it still creates an unnecessary load on cromwell.
thanks!