Good afternoon Ovrture users,
This document details the cause and events occurring immediately after Ovrture’s outage on May 21, 2019 as well as the steps we are taking to mitigate the impact of future outages like this one in the future.
On May 21, 2019, the login page for Ovrture experienced an outage which affected the ability for users to login to the backend of the platform. This did not affect any live sites or reports. The outage occurred from 12:44 to 1:20 PM.
A client alerted Ovrture that they were having issues logging in. Ovrture investigated the issue and discovered that this was occurring system wide. No live sites or reports were affected. Ovrture alerted the development team and the devops team about the issue and they began to investigate immediately.
The root cause was that the JESSIONID stickiness was not active on the load balancer. These two load balancers were just updated with a new SSL Certificate. As a result of this update, it deactivated the load balancer. That change should have updated the load balancers in-place, leaving the associations with the cookie stickiness policies untouched. We now know to check this in the future.
Things we will improve to make sure issues like this do not happen again in the future:
-When updating SSL Certificates, we will check all load balancers to make sure nothing was affected during the update.
We are very sorry for any inconvenience this outage may have caused on May 21 and we will continue to work hard to make sure something like this doesn’t happen again.
Gideon and the Ovrture Team