Hi Team,
We are running five dollar pipeline by running Cromwell(v39) on AWS. Whenever we are running our pipeline on small files(~300MB) process is proceeding fine with further processing. But, whenever we are providing large file ranging 48GB to 68GB then it is failing due to below error.
We observed that "SplitLargeReadGroup.SamSplitter" only triggering for large files but not for smaller files which are ranging in ~MBs.
Do we need to perform any special configuration to handle large files or we are doing some wrong config which is causing failure to this pipeline.
Exception:
AwsBatchAsyncBackendJobExecutionActor [^[[38;5;2m5f712db2^[[0mSplitLargeReadGroup.SamSplitter:NA:1]: ^[[38;5;5mset -e
mkdir output_dir
total_reads=$(samtools view -c /cromwell_root/cromwelleast/references/broad-references/macrogen_NA12878_full.bam)
java -Dsamjdk.compression_level=2 -Xms3000m -jar /usr/gitc/picard.jar SplitSamByNumberOfReads \
INPUT=/cromwell_root/cromwellbucket/references/broad-references/macrogen_NA12878_full.bam \
OUTPUT=output_dir \
SPLIT_TO_N_READS=48000000 \
TOTAL_READS_IN_INPUT=$total_reads^[[0m
[2019-04-18 20:50:06,31] [^[[38;5;1merror^[[0m] AwsBatchAsyncBackendJobExecutionActor [^[[38;5;2m5f712db2^[[0mSplitLargeReadGroup.SamSplitter:NA:1]: Error attempting to Execute
cromwell.engine.io.IoAttempts$EnhancedCromwellIoException: [Attempted 1 time(s)] - FileSystemException: /tmp/temp-s3-538074772416833219ce_WholeGenomeGermlineSingleSample_91352b21-b271-443c-b332-0a25b27ec894_call-UnmappedBamToAlignedBam_UnmappedBamToAlignedBam_b2858ebe-2463-48b7-bfc8-f83a786e5247_call-SplitRG_shard-0_SplitLargeReadGroup_5f712db2-4f6e-9955-7feeb03af894_call-SamSplitter_script: File name too long
Caused by: java.nio.file.FileSystemException: /tmp/temp-s3-538074772416833219ce_WholeGenomeGermlineSingleSample_91352b21-b271-443c-b332-0a25b27ec894_call-UnmappedBamToAlignedBam_UnmappedBamToAlignedBam_b2858ebe-2463-48b7-bfc8-f83a786e5247_call-SplitRG_shard-0_SplitLargeReadGroup_5f712db2-4f6e-9955-7feeb03af894_call-SamSplitter_script: File name too long
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.createFile(Files.java:632)
at java.nio.file.TempFileHelper.create(TempFileHelper.java:138)
at java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:161)
at java.nio.file.Files.createTempFile(Files.java:897)
at org.lerch.s3fs.S3SeekableByteChannel.<init>(S3SeekableByteChannel.java:52)
at org.lerch.s3fs.S3FileSystemProvider.newByteChannel(S3FileSystemProvider.java:360)
at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
at java.nio.file.Files.newOutputStream(Files.java:216)
at java.nio.file.Files.write(Files.java:3292)
at better.files.File.writeByteArray(File.scala:270)
at better.files.File.write(File.scala:280)
at cromwell.core.path.BetterFileMethods.write(BetterFileMethods.scala:179)
at cromwell.core.path.BetterFileMethods.write$(BetterFileMethods.scala:178)
at cromwell.filesystems.s3.S3Path.write(S3PathBuilder.scala:158)
at cromwell.core.path.EvenBetterPathMethods.writeContent(EvenBetterPathMethods.scala:99)
at cromwell.core.path.EvenBetterPathMethods.writeContent$(EvenBetterPathMethods.scala:97)
at cromwell.filesystems.s3.S3Path.writeContent(S3PathBuilder.scala:158)
at cromwell.engine.io.nio.NioFlow.$anonfun$write$1(NioFlow.scala:89)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:87)
at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:351)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:372)
at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:312)
at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)