Because optional outputs have yet to be added to the WDL spec (an outstanding issue for over two years), we've had to implement a work-around using a generic "null" file:
task variable_outputs {
File? opt_input
String null_file = "gs://broad-institute-gdac/GDAC_FC_NULL"
command { ... }
output { File might_exist = "${if defined(opt_input) then 'opt_output.txt' else null_file}" }
}
workflow var_output_wf {
call variable_outputs
}
At some point between our implementation and now, this stopped working and now returns an error:
Failed to delocalize files: failed to copy the following files: "/mnt/local-disk/broad-institute-gdac/GDAC_FC_NULL -> ...
Instead of simply coercing the bucket path String into a File, it seems to now make the assumption the file was localized, and looks for it on the local disk rather than doing a pass-through.
This is an extreme change in behavior, as in the past a File output set to a File input would return the original input, rather than the localized copy of the file, which we discovered when we would zip the results of the first task in a workflow, add to that zip file in the second workflow, and continue passing task-to-task, only to see at the end that only the files in the first task were in the archive. This change seems to imply that if I tried that now, the final output would have all the results in it. I haven't seen any release notes stating this change.
While the above code would probably work if I changed the input type for null_file from String to File, doing so would break our local testing process that involves running the WDLs via local cromwell, which doesn't recognize google bucket paths, and thus would fail when it tries to localize the file. Yes this limits our testing to only be with the optional inputs to prevent a delocalization failure, but we decided that was acceptable.
Any insight into what is happening, or if this can be fixed would be very welcome. Thanks!