Notification of new incident. Login Issues for end users: US
Incident Report for uniFLOW Online
Postmortem

User Impact  

Users could not log into their uniFLOW Online tenant. This impacted both admin and user logins, blocking tenant administration and end user functionality. End user functionality such as account admin or print job upload. 

 

Scope of Impact: 

US Deployment  

The impact was not across all tenants and only impacted a percentage of the users logging into the service. 

 

Incident Start Date and Time  

September 25th 12:22PM UTC 

Incident End Date and Time  

September 25th 13:20PM UTC 

 

Root Cause: 

It was identified that the login session was being revoked prematurely. The login session (cookie) is derived from several information sources, (tokens). It was found one of these sources could change during a scaling event as the underlying Azure hardware changed. 

Due to this the resulting cookie was no longer seen as valid and would force the logged in session to be revoked and the user passed back to the login screen. 

  

How did we respond: 

On detection of the event the Operations team were able to cycle the web services providing a consistent session token for the cookie creation and allowing login to take place. 

  

Next Steps : 

We apologize for the impact on affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):  

  • Monitoring for this event was put in place providing early detection of this specific failing condition. 

  • We will implement a cookie handling architecture independent of the Azure offering. This is being worked on with high priority and will be deployed once our review and Quality Assurance process are complete. 

 

Was this incident related to previous incidents? 

Yes, this incident happened twice, once on the 25th of September and the 27th.  

 

Customer Recommendations: 

There are no recommendations for this incident type.

Posted Oct 17, 2024 - 22:10 UTC

Resolved
Hello Everyone,

We have moved this incident to resolved. Following a thorough investigation internally we will raise a Postmortem no later than 20 business days from today.

Kind Regards
uniFLOW Online Operations Team.
Posted Sep 25, 2024 - 14:01 UTC
Monitoring
Hello Everyone,

We are moving to the status of this incident to monitoring as we can no longer reproduce the login issue.

Monitoring will continue for another 30 minutes while we ensure full system recovery.

Next update in 30 minutes or as information is available.

Regards,
uniFLOW Online Operations Team
Posted Sep 25, 2024 - 13:34 UTC
Identified
Hello Everyone,

The operations team has confirmed the issue and applying mitigations.

We are already seeing a recovery of services and confirmation from the field.

Next update will be in 30 minutes or as information is available.

uniFLOW Online Operations.
Posted Sep 25, 2024 - 13:23 UTC
Investigating
Incident details:

Investigating: We are investigating reports of login issues for users on our US deployment.

Start Time:
25-09-2024: 13:00 UTC

Incident Scope: 
US Deployment

Description:
End users are seeing an 'Internal server error' on login.

Next Update:
The next update will be in 30 minutes.
Posted Sep 25, 2024 - 13:11 UTC
This incident affected: US Deployment (Identification).