Quantcast
Channel: Ask the FireCloud Team — GATK-Forum
Viewing all articles
Browse latest Browse all 1147

Failure while processing large unmapped bam file as input to cromwell

$
0
0

Hi Team,

We are running five dollar pipeline by running Cromwell(v39) on AWS. Whenever we are running our pipeline on small files(~300MB) process is proceeding fine with further processing. But, whenever we are providing large file ranging 48GB to 68GB then it is failing due to below error.

We observed that "SplitLargeReadGroup.SamSplitter" only triggering for large files but not for smaller files which are ranging in ~MBs.

Do we need to perform any special configuration to handle large files or we are doing some wrong config which is causing failure to this pipeline.

Exception:

AwsBatchAsyncBackendJobExecutionActor [^[[38;5;2m5f712db2^[[0mSplitLargeReadGroup.SamSplitter:NA:1]: ^[[38;5;5mset -e
mkdir output_dir

total_reads=$(samtools view -c /cromwell_root/cromwelleast/references/broad-references/macrogen_NA12878_full.bam)

java -Dsamjdk.compression_level=2 -Xms3000m -jar /usr/gitc/picard.jar SplitSamByNumberOfReads \
  INPUT=/cromwell_root/cromwellbucket/references/broad-references/macrogen_NA12878_full.bam \
  OUTPUT=output_dir \
  SPLIT_TO_N_READS=48000000 \
  TOTAL_READS_IN_INPUT=$total_reads^[[0m
[2019-04-18 20:50:06,31] [^[[38;5;1merror^[[0m] AwsBatchAsyncBackendJobExecutionActor [^[[38;5;2m5f712db2^[[0mSplitLargeReadGroup.SamSplitter:NA:1]: Error attempting to Execute
cromwell.engine.io.IoAttempts$EnhancedCromwellIoException: [Attempted 1 time(s)] - FileSystemException: /tmp/temp-s3-538074772416833219ce_WholeGenomeGermlineSingleSample_91352b21-b271-443c-b332-0a25b27ec894_call-UnmappedBamToAlignedBam_UnmappedBamToAlignedBam_b2858ebe-2463-48b7-bfc8-f83a786e5247_call-SplitRG_shard-0_SplitLargeReadGroup_5f712db2-4f6e-9955-7feeb03af894_call-SamSplitter_script: File name too long
Caused by: java.nio.file.FileSystemException: /tmp/temp-s3-538074772416833219ce_WholeGenomeGermlineSingleSample_91352b21-b271-443c-b332-0a25b27ec894_call-UnmappedBamToAlignedBam_UnmappedBamToAlignedBam_b2858ebe-2463-48b7-bfc8-f83a786e5247_call-SplitRG_shard-0_SplitLargeReadGroup_5f712db2-4f6e-9955-7feeb03af894_call-SamSplitter_script: File name too long
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
        at java.nio.file.Files.newByteChannel(Files.java:361)
        at java.nio.file.Files.createFile(Files.java:632)
        at java.nio.file.TempFileHelper.create(TempFileHelper.java:138)
        at java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:161)
        at java.nio.file.Files.createTempFile(Files.java:897)
        at org.lerch.s3fs.S3SeekableByteChannel.<init>(S3SeekableByteChannel.java:52)
        at org.lerch.s3fs.S3FileSystemProvider.newByteChannel(S3FileSystemProvider.java:360)
        at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
        at java.nio.file.Files.newOutputStream(Files.java:216)
        at java.nio.file.Files.write(Files.java:3292)
        at better.files.File.writeByteArray(File.scala:270)
        at better.files.File.write(File.scala:280)
        at cromwell.core.path.BetterFileMethods.write(BetterFileMethods.scala:179)
        at cromwell.core.path.BetterFileMethods.write$(BetterFileMethods.scala:178)
        at cromwell.filesystems.s3.S3Path.write(S3PathBuilder.scala:158)
        at cromwell.core.path.EvenBetterPathMethods.writeContent(EvenBetterPathMethods.scala:99)
        at cromwell.core.path.EvenBetterPathMethods.writeContent$(EvenBetterPathMethods.scala:97)
        at cromwell.filesystems.s3.S3Path.writeContent(S3PathBuilder.scala:158)
        at cromwell.engine.io.nio.NioFlow.$anonfun$write$1(NioFlow.scala:89)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:87)
        at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:351)
        at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:372)
        at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:312)
        at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)

Viewing all articles
Browse latest Browse all 1147

Trending Articles