Good morning Ovrture users,

This document details the cause and events that occurred during Ovrture’s incident on April 7, 2022 as well as the steps we are taking to mitigate the impact of future incidents like this one in the future.

On April 7, 2022, there was a partial outage impacting the Ovrture platform login page. This outage occurred from 8:40am - 9:50am EST.

When the issue was found, the ODT notified DevOps and the Engagement team of the issue via Slack. ODT, DevOps, and the Engagement team connected via Zoom to diagnose the issue.

Engagement helped investigate the issue and determined that the client using the private production environment and clients using the shared production environment and using mysite.ovrture.com instead of their unique URL were the only tenants being affected. Clients using their own unique URLs and clients using the Canada production environment did not have any issues. No live donor/prospect sites or reports were being affected. Engagement then updated the Ovrture Status Page accordingly. 

ODT and DevOps continued to investigate the issue to determine what could have caused the incident. Through collaborative testing, they were able to pinpoint and fix the issue.

The confirmed root cause was a configuration error that caused the application to malfunction in the way that it handled some requests. We have rolled back the code dependent on this configuration and will build a new solution and test cases that evaluate the configuration prior to production code deployment.

Onward,

The Ovrture Team