On June 14th we experienced an outage for customers using our Single Sign On feature following a routine system update. Making them unable to log in. We were made aware of the issues 08:37 and started working on a solution at 08:40. This outage affected customers that are using our Single Sign On feature and we apologize for the inconvenience it may have caused. The service has now been restored.
Upsales CRM consists of multiple services that communicate with each other. For system updates we are releasing new updates to multiple services at the same time. The update contained changes to functionality that is executed upon logging in to the system which didn’t work for customers using Single Sign On.
At first investigation the problem seemed to be external from Upsales service as there was errors related to one Single Sign On provider in the logs of the system. When more customers with different providers reported similar issues and us being unable to directly find a fix we decided to execute a rollback of the release. However while this rollback seemed to have executed properly it failed to execute across all of our services. Leadning the non-working service still running the latest version of the code. The error rates was quite low after the faulty rollback leading the team to believe that the problem had been mitigated. Afterwards we discovered that the rollback hadn’t completed successfully and did it again which brought back service for the affected customers.
08:30 - The faulty code was released in production
08:37 - The team was first notified that users was unable to log in with Single Sign On
09:34 - A rollback of the release was executed. However leaving the system in a bad state with part of the system running an old version and other parts running the new version
11:53 - A second rollback was executed to revert the remaining parts of the system to an older version
11:59 - Incident fully mitigated