I submitted a wgs_pip_m2_64core job in workspace rebc-oct16/rebc_template on Jan 24th. Three days later the job actually started according to the monitor tab (attached image). No preemption. Why would a job wait three days before starting?
The only job running since 24 Jan appearing on my google cloud compute engine view has a CPU history from the second attached image with a spike at 100% late on the 24th and a long tail of ~1.5% for three days, which usually means that the job ran out of memory and is waiting for a week before google will kill it.
These two interpretations (waiting for three days to start vs. running out of memory and going into hibernation) can't both be right. Should I kill the job and restart or continue to wait and hope for the best?