The services are still up and running since the last update.
We’re catching up with our ML Usage Analytics events backlog and expect to be done within ~3 hours.
We’ll post a final update by then.
Posted Nov 26, 2020 - 10:10 EST
Update
Our provider has mostly recovered and our services are still up and functional since our last update. We will keep our mitigation measures until tomorrow morning (Eastern time), reassess the situation then and update this incident.
We are monitoring the situation and are in close communications with our provider.
Posted Nov 25, 2020 - 22:37 EST
Monitoring
Our provider is gradually recovering and our workarounds allowed us to stabilize the Analytics events writes.
We are actively monitoring the service.
Posted Nov 25, 2020 - 16:51 EST
Update
We deployed measures to mitigate the impact. This will help prevent the loss of Analytics events. More details will be provided later.
Posted Nov 25, 2020 - 15:54 EST
Update
Indexing is fixed and the backlog has been processed. Some longer sources refreshes are still ongoing.
This service is back to normal and we are actively monitoring it.
Posted Nov 25, 2020 - 15:08 EST
Update
Our infrastructure provider is still working on getting this issue solved but has no ETA to share. You can follow their progression on AWS Service Health Dashboard.
In parallel, we're working on ways to mitigate the outage and will update you with our progress.
Posted Nov 25, 2020 - 13:25 EST
Update
The issue is also impacting indexing so your sources content might not be up to date.
We're still in close communication with our provider.
Posted Nov 25, 2020 - 10:18 EST
Identified
We've identified an issue affecting Usage Analytics Write APIs.
A number of events sent since 8:14am Eastern time might be irrecoverable. We've identified the problem being with our infrastructure provider and we're in communication with them while they investigate.
If you need help or to get in touch with us, please visit our Help Portal
Posted Nov 25, 2020 - 08:36 EST
This incident affected: US (Analytics - Analytics Write API).