I have a workflow that creates a number of large intermediate files that are passed between tasks. These files do not need to be retained at the end of the workflow (and take up a considerable amount of space). Is there a way to specify that they are temporary and should not retained in a GS bucket?
As an example, the workflow below creates a file called random_text.file
that is not required as part of the workflow outputs.
workflow test {
call random_text
call count_lines {input: random_file=random_text.file }
}
task random_text {
command {
cat /dev/random | head -c 10000 > random.txt
}
output {
File file = "random.txt"
}
}
task count_lines {
File random_file
command {
wc -l ${random_file}
}
output {
String num_lines = stdout()
}
}