Resolved -
This incident has been resolved. All services have remained fully operational since the last update.
Oct 21, 07:29 EDT
Monitoring -
All remaining impacted services have returned to normal operations since the last update.
The Analytics Event Ingestion backlog has been processed, and the Real-Time Machine Learning features that depend on them are now all caught up.
We are continuing to monitor all services closely.
Oct 20, 17:42 EDT
Update -
We are seeing continued delays in Analytics Event Ingestion. No events are lost, at the moment they are being queued for processing.
Real-time Machine Learning features, such as Personalisation as You Go, are also impacted by the delay in Analytics Ingestion, impacting the quality of the results.
The elevated error rates which impacted the Analytics Read API in the last few hours have been mitigated and that service is now back to normal operations.
Oct 20, 16:46 EDT
Update -
We are continuing to see elevated error rates on the Analytics Read API as well as continued delays in Analytics Event Ingestion in the US region. No events are lost, at the moment they are being queued for processing.
Real-time Machine Learning features, such as Personalisation as You Go, are also impacted by the delay in Analytics Ingestion, impacting the quality of the results.
Capacity issues have improved in the US region, and Crawling is back to normal operations.
Oct 20, 14:39 EDT
Update -
We are continuing to see elevated error rates on the Analytics Read API as well as continued delays in Analytics Event Ingestion in the US region. No events are lost, at the moment they are being queued for processing.
Real-time Machine Learning features, such as Personalisation as You Go, are also impacted by the delay in Analytics Ingestion, impacting the quality of the results.
Capacity problems are also slowing down our crawling operations in the US region, introducing some delays for scheduled jobs. Your content for sources crawled by our infrastructure may not be up to date. Push API and Crawling Module sources remain up to date.
Oct 20, 13:37 EDT
Update -
We are continuing to see elevated error rates on the Analytics Read API as well as continued delays in Analytics Event Ingestion in the US region. No events are lost, at the moment they are being queued for processing.
Real-time Machine Learning features, such as Personalisation as You Go, are also impacted by the delay in Analytics Ingestion, impacting the quality of the results.
Oct 20, 12:29 EDT
Update -
We are continuing to see elevated error rates on the Analytics Read API and are now seeing some delays in Analytics Event Ingestion in the US region. No events are lost, at the moment they are being queued for processing.
Oct 20, 11:11 EDT
Update -
All document processing backlogs have been processed, so indexed documents are now up to date for all customers in all regions. We are continuing to see elevated error rates on the Analytics Read service as the issue with our infrastructure provider is ongoing.
Oct 20, 09:50 EDT
Identified -
Due to the ongoing issue with our infrastructure provider, we are observing a degradation of Analytics Read services in the US region. Users of this service are currently seeing elevated error rates. We are also continuing to process the previously mentioned document processing backlog.
Oct 20, 09:11 EDT
Update -
The underlying issues have been resolved by our infrastructure provider. Coveo services have been working normally since the last update and we are working on a document processing backlog that accumulated over the incident in the US region. All other region's document processing is caught up to real time.
Oct 20, 07:27 EDT
Monitoring -
An issue with an infrastructure provider has led to a partial degradation of some Coveo services. Mitigations are already ongoing at the level of infrastructure provider and therefore we are already seeing improvements happening. We will continue to monitor the issue until this is fully resolved.
Oct 20, 05:50 EDT