Single Cell Portal received 403 errors from FireCloud:
E, 2018-08-17T10:45:10.218293 #30059 ERROR – : 1058e905-3df6-4663-b1f0-1bd8966ffc31 2018-08-17 10:45:10 -0400: 403 Forbidden encountered when requesting 'https://api.firecloud.org/api/workspaces/single-cell-portal/development-sync-test-study/importEntities'
During the time of the duration, users of SCP received an accountDisabled error.
The error appears to happen any time we make API calls to either FireCloud or Google. We observed at least 3 reported instances this time: download and delete files (GCS), sync workspace (FC and GCS), and update the workspace entity model (FC). It's hard to verify exactly when the errors happens, because usually when the problem happens, we refresh the tokens so SCP isn't hosed anymore. We think it's possible to run a quick smoke test to find out what exactly is affected in the future.
We have a manual workaround by forcing the FC API client to generate all access tokens: the class that we use in SCP to connect to both FireCloud & Google Cloud Storage uses cached OAuth tokens when making requests. We have the ability to flush all those and generate new ones to use. We generate new tokens automatically when they expire, but sometimes (twice in a month as we've seen) even though the token is supposedly valid and hasn't expired, it still get rejected.
The errors started happening yesterday afternoon ~4:30pm, which may be related to FC deployment schedule; not sure.
We need to find the root cause and at least have an automatic workaround.