Is it possible to move the Docker disk image files off the boot disk?
In Cromwell 19.3, two different disk size parameters need to be set in the runtime block of the wdl, eg via the following:
disks: "local-disk ${{output_disk_gb}} HDD"
bootDiskSizeGb: "${{boot_disk_gb}}"
The first line specifies where the input and output files go, while the second specifies the host machine's boot disk. The Docker images also go there, in the default location of /var/lib/docker. This is fine if the algorithm only writes to the output directory, but many algorithms also write files elsewhere, eg /tmp. We have seen multiple instances where the /tmp directory became too large for the default boot disk, which typically crashes the machine before it has time to generate a useful error message.
Workarounds exist, eg symlinking /tmp to the output directory, or increasing space on the boot disk (requiring extra margin on two disks rather than one). But, a more convenient solution would be moving the docker image to the same disk used for the outputs.
I've gotten this to work on GCE by dropping a symlink in place of /var/lib/docker, though it looks like the same effect can be achieved via dockerd parameters.