Hi, Firecloud team
I've tried Mutect2 for some times recently on my WGS data, but they all failed. I know some of the early failure are associated with some old scripts, but now I change to most updated NIO version, but unfortunately, they still failed with the general error like:
Job Mutect2.M2:16:1 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
The details error are something associated with google compute engine or storages, see below as example
[June 1, 2018 6:48:12 AM UTC] org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2 done. Elapsed time: 501.93 minutes. Runtime.totalMemory()=6155141120 code: 0 message: ComputeEngineCredentials cannot find the metadata server. This is likely because code is not running on Google Compute Engine. reason: null location: null retryable: false com.google.cloud.storage.StorageException: ComputeEngineCredentials cannot find the metadata server. This is likely because code is not running on Google Compute Engine. at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189) at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:515) at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127) at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124) at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:94) at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54) at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:124) at com.google.cloud.storage.contrib.nio.CloudStorageReadChannel.read(CloudStorageReadChannel.java:114) at htsjdk.samtools.reference.IndexedFastaSequenceFile.readFromPosition(IndexedFastaSequenceFile.java:292) at htsjdk.samtools.reference.IndexedFastaSequenceFile.getSubsequenceAt(IndexedFastaSequenceFile.java:244) at org.broadinstitute.hellbender.utils.fasta.CachingIndexedFastaSequenceFile.getSubsequenceAt(CachingIndexedFastaSequenceFile.java:309) at org.broadinstitute.hellbender.engine.ReferenceFileSource.queryAndPrefetch(ReferenceFileSource.java:80) at org.broadinstitute.hellbender.engine.ReferenceDataSource.queryAndPrefetch(ReferenceDataSource.java:64) at org.broadinstitute.hellbender.engine.ReferenceContext.getBases(ReferenceContext.java:166) at org.broadinstitute.hellbender.engine.ReferenceContext.getBase(ReferenceContext.java:367) at org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2Engine.isActive(Mutect2Engine.java:224) at org.broadinstitute.hellbender.engine.AssemblyRegionIterator.loadNextAssemblyRegion(AssemblyRegionIterator.java:159) at org.broadinstitute.hellbender.engine.AssemblyRegionIterator.next(AssemblyRegionIterator.java:135) at org.broadinstitute.hellbender.engine.AssemblyRegionIterator.next(AssemblyRegionIterator.java:34) at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.processReadShard(AssemblyRegionWalker.java:290) at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.traverse(AssemblyRegionWalker.java:271) at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:892) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:134) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:179) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:198) at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160) at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203) at org.broadinstitute.hellbender.Main.main(Main.java:289) Caused by: java.io.IOException: ComputeEngineCredentials cannot find the metadata server. This is likely because code is not running on Google Compute Engine. at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:106) at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:149) at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:135) at shaded.cloud_nio.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96) at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:157) at shaded.cloud_nio.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93) at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300) at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMedia(AbstractGoogleClientRequest.java:380) at shaded.cloud_nio.com.google.api.services.storage.Storage$Objects$Get.executeMedia(Storage.java:6133) at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:494) ... 26 more Caused by: java.net.UnknownHostException: metadata at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) at sun.net.www.http.HttpClient.New(HttpClient.java:339) at sun.net.www.http.HttpClient.New(HttpClient.java:357) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) at shaded.cloud_nio.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93) at shaded.cloud_nio.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972) at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:104) ... 37 more
and
java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.google.cloud.storage.StorageException: www.googleapis.com at org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher.read(SeekableByteChannelPrefetcher.java:309) at htsjdk.samtools.seekablestream.SeekablePathStream.read(SeekablePathStream.java:86) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at htsjdk.samtools.seekablestream.SeekableBufferedStream.read(SeekableBufferedStream.java:104) at htsjdk.samtools.util.BlockCompressedInputStream.readBytes(BlockCompressedInputStream.java:571) at htsjdk.samtools.util.BlockCompressedInputStream.readBytes(BlockCompressedInputStream.java:560) at htsjdk.samtools.util.BlockCompressedInputStream.processNextBlock(BlockCompressedInputStream.java:510) at htsjdk.samtools.util.BlockCompressedInputStream.nextBlock(BlockCompressedInputStream.java:468) at htsjdk.samtools.util.BlockCompressedInputStream.seek(BlockCompressedInputStream.java:380) at htsjdk.tribble.readers.TabixReader$IteratorImpl.next(TabixReader.java:427) at htsjdk.tribble.readers.TabixIteratorLineReader.readLine(TabixIteratorLineReader.java:46) at htsjdk.tribble.TabixFeatureReader$FeatureIterator.readNextRecord(TabixFeatureReader.java:170) at htsjdk.tribble.TabixFeatureReader$FeatureIterator.<init>(TabixFeatureReader.java:159) at htsjdk.tribble.TabixFeatureReader.query(TabixFeatureReader.java:133) at org.broadinstitute.hellbender.engine.FeatureDataSource.refillQueryCache(FeatureDataSource.java:555) at org.broadinstitute.hellbender.engine.FeatureDataSource.queryAndPrefetch(FeatureDataSource.java:524) at org.broadinstitute.hellbender.engine.FeatureManager.getFeatures(FeatureManager.java:308) at org.broadinstitute.hellbender.engine.FeatureContext.getValues(FeatureContext.java:163) at org.broadinstitute.hellbender.engine.FeatureContext.getValues(FeatureContext.java:115) at org.broadinstitute.hellbender.engine.FeatureContext.getValues(FeatureContext.java:230) at org.broadinstitute.hellbender.tools.walkers.mutect.SomaticGenotypingEngine.callMutations(SomaticGenotypingEngine.java:144) at org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2Engine.callRegion(Mutect2Engine.java:182) at org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2.apply(Mutect2.java:183) at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.processReadShard(AssemblyRegionWalker.java:295) at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.traverse(AssemblyRegionWalker.java:271) at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:892) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:134) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:179) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:198) at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160) at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203) at org.broadinstitute.hellbender.Main.main(Main.java:289) Caused by: java.util.concurrent.ExecutionException: com.google.cloud.storage.StorageException: www.googleapis.com at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher$WorkUnit.getBuf(SeekableByteChannelPrefetcher.java:136) at org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher.fetch(SeekableByteChannelPrefetcher.java:255) at org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher.read(SeekableByteChannelPrefetcher.java:300) ... 33 more Caused by: com.google.cloud.storage.StorageException: www.googleapis.com at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189) at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:515) at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127) at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124) at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:94) at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54) at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:124) at com.google.cloud.storage.contrib.nio.CloudStorageReadChannel.read(CloudStorageReadChannel.java:114) at org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher$WorkUnit.call(SeekableByteChannelPrefetcher.java:131) at org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher$WorkUnit.call(SeekableByteChannelPrefetcher.java:104) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.UnknownHostException: www.googleapis.com at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153) at shaded.cloud_nio.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93) at shaded.cloud_nio.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972) at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMedia(AbstractGoogleClientRequest.java:380) at shaded.cloud_nio.com.google.api.services.storage.Storage$Objects$Get.executeMedia(Storage.java:6133) at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:494) ... 12 more
I dont know what's happening here and could you please suggest some solutions.
Thank you very much