User Impact
Users could not log into their uniFLOW Online tenant. This impacted both admin and user logins, blocking tenant administration and end user functionality. End user functionality such as account admin, print job upload.
Scope of Impact:
EU Deployment
The impact was not across all tenants and only impacted a percentage of the users logging into the service.
Incident Start Date and Time
September 9th 5:22PM UTC
Incident End Date and Time
September 9th 10:50PM UTC
Root Cause:
It was identified that the login session was being revoked prematurely. The login session (cookie) is derived from several information sources, (tokens). It was found one of these sources could change during a scaling event as the underlying Azure hardware changed.
Due to this the resulting cookie was no longer seen as valid and would force the logged in session to be revoked and the user passed back to the login screen.
How did we respond:
On detection of the event the Operations team were able to cycle the web services providing a consistent session token for the cookie creation and allowing login to take place.
Next Steps :
We apologize for the impact on affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):
Monitoring for this event was put in place providing early detection of this specific failing condition.
We will implement a cookie handling architecture independent of the Azure offering. This is being worked on with high priority and will be deployed once our review and Quality Assurance process are complete.
Was this incident related to previous incidents?
Yes, this incident happened twice, once on the 25th of September and the 27th.
Customer Recommendations: