see the WDL:
task globber {
command <<<
set -x
mkdir -pv d1 d2
echo "this is one" > d1/out.txt
echo "this is two" > d2/out.txt
head */out.txt
>>>
output {
Array[File] outputs=glob("*/out.txt")
}
runtime {
docker: "ubuntu:14.04"
disks: "local-disk 1 HDD"
}
}
workflow w
{
call globber
}
See how both the out.txt files in the d1 and d2 directories should be retrieved?
The stderr/stdout shows:
wm8b1-75c:execution esalinas$ gsutil cat gs://fc-2e652e1b-d03c-4e00-bd9a-920bd605b1ab/75b4fa60-a2a6-4781-8678-34078302daa6/w/df9ab7d0-511d-4562-85b5-2946e8596ee5/call-globber/globber-stderr.log
+ mkdir -pv d1 d2
+ echo 'this is one'
+ echo 'this is two'
+ head d1/out.txt d2/out.txt
ln: failed to create hard link '/cromwell_root/glob-c23eda6b73b66be4963c1f6554fb3256/out.txt': File exists
ln: failed to create hard link '/cromwell_root/glob-c23eda6b73b66be4963c1f6554fb3256/out.txt': File exists
ln: failed to create hard link '/cromwell_root/glob-c23eda6b73b66be4963c1f6554fb3256/out.txt': File exists
wm8b1-75c:execution esalinas$ gsutil cat gs://fc-2e652e1b-d03c-4e00-bd9a-920bd605b1ab/75b4fa60-a2a6-4781-8678-34078302daa6/w/df9ab7d0-511d-4562-85b5-2946e8596ee5/call-globber/globber-stdout.log
mkdir: created directory 'd1'
mkdir: created directory 'd2'
==> d1/out.txt <==
this is one
==> d2/out.txt <==
this is two
wm8b1-75c:execution esalinas$
The "failed to create hard link" errors seem to be coming from "ln" commands in the exec.sh seemingly added by cromwell or google.
wm8b1-75c:execution esalinas$ gsutil cat gs://fc-2e652e1b-d03c-4e00-bd9a-920bd605b1ab/75b4fa60-a2a6-4781-8678-34078302daa6/w/df9ab7d0-511d-4562-85b5-2946e8596ee5/call-globber/exec.sh|tail
echo $? > /cromwell_root/globber-rc.txt.tmp
(
cd /cromwell_root
mkdir /cromwell_root/glob-c23eda6b73b66be4963c1f6554fb3256
( ln -L */out.txt /cromwell_root/glob-c23eda6b73b66be4963c1f6554fb3256 2> /dev/null ) || ( ln */out.txt /cromwell_root/glob-c23eda6b73b66be4963c1f6554fb3256 )
ls -1 /cromwell_root/glob-c23eda6b73b66be4963c1f6554fb3256 > /cromwell_root/glob-c23eda6b73b66be4963c1f6554fb3256.list
)
sync
mv /cromwell_root/globber-rc.txt.tmp /cromwell_root/globber-rc.txt
wm8b1-75c:execution esalinas$
In addition only one output results in the glob directory:
wm8b1-75c:execution esalinas$ gsutil ls gs://fc-2e652e1b-d03c-4e00-bd9a-920bd605b1ab/75b4fa60-a2a6-4781-8678-34078302daa6/w/df9ab7d0-511d-4562-85b5-2946e8596ee5/call-globber/glob-c23**
gs://fc-2e652e1b-d03c-4e00-bd9a-920bd605b1ab/75b4fa60-a2a6-4781-8678-34078302daa6/w/df9ab7d0-511d-4562-85b5-2946e8596ee5/call-globber/glob-c23eda6b73b66be4963c1f6554fb3256.list
gs://fc-2e652e1b-d03c-4e00-bd9a-920bd605b1ab/75b4fa60-a2a6-4781-8678-34078302daa6/w/df9ab7d0-511d-4562-85b5-2946e8596ee5/call-globber/glob-c23eda6b73b66be4963c1f6554fb3256/out.txt
wm8b1-75c:execution esalinas$ gsutil cat gs://fc-2e652e1b-d03c-4e00-bd9a-920bd605b1ab/75b4fa60-a2a6-4781-8678-34078302daa6/w/df9ab7d0-511d-4562-85b5-2946e8596ee5/call-globber/glob-c23eda6b73b66be4963c1f6554fb3256/out.txt
this is one
wm8b1-75c:execution esalinas$
It would seem that the desired behavior should be to have two directories ("d1" and "d2" in the glob directory and each with its own "out.txt" .