GroupBy V2 Recommendations - Degraded Performance
Incident Report for GroupBy
Postmortem

Issue:
On July 10th 2020, we experienced a degraded performance issue with the V2 Recommendations where some of our clients experienced a slowness in the API response time or a failure of API calls.

Root Cause:
Our data nodes experienced a period where API calls exceeded data size limits, causing server nodes to restart automatically, with some services within the cluster becoming inaccessible.

Mitigation Steps:
An approach is being investigated to limit the size of individual calls to prevent this from happening in the future.

Posted Jul 15, 2020 - 13:57 EDT

Resolved
We experienced issues where some clients saw slow or non-responsive V2 Recommendation requests. Our engineers have now corrected and resolved the issue. A root cause analysis of the issue should be posted in the next few business days. If you would like further information about this incident please contact us at support@groupbyinc.com
Posted Jul 10, 2020 - 15:05 EDT
Investigating
We are investigating possible degraded performance issues with V2 Recommendations for some clients, where calls are either failing or slow to respond. Our engineers are engaged and working to determine the issue. We will update this incident as soon as we have additional information.
Posted Jul 10, 2020 - 14:16 EDT
This incident affected: Recommendation API (Recommendation API).