Notification of extended recovery. Delay in displaying user changes in the UI : EU
Incident Report for uniFLOW Online
Postmortem

User Impact  

Customers have experienced issues where user or identity changes are not reflected in the UI immediately. This could be observed by tenant administrators and end users making changes to their own or other user information. 

It is important to note that the changes, in all cases, happened. The issue was the UI did not reflect the implemented change. 

 

Scope of Impact: 

EU Deployment  

The impact of this was observed across all tenants within the deployment. 

 

Incident Start Date and Time  

September 5th , 2024 

Incident End Date and Time  

September 19th , 2024 

 

Root Cause: 

On investigation it was identified that the processing of user information was delayed. It was identified that feature inclusions over 2024.1 and 2024.2 made more use of this process and became less responsive under high load as seen in the EU (our largest deployment). 

 

How did we respond: 

Addressing this issue was a multi-step process during identification and remediation: 

  • From the initial event we investigated both the potential cause of the high load as well as the responsiveness during these periods. 

  • On 4 September 2024 we applied mitigation in an attempt to reduce the high load, but this, unfortunately, did not lead to a large improvement as the issue reoccurred on 5 September 2024. 

  • On 5 September 2024 it was decided to raise an extended recovery status page while we continue to analyze the root cause and provide a fix. 

  • The root cause was found and the processing resolved with a global maintenance update to uniFLOW Online. 

 

Next Steps : 

We apologize for the impact on affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):  

  • The issue has been resolved, (Already deployed globally). 

  • Improved and optimized event processing. 

  • User, Cost Centre, Print job sharing and scan delegation user changes. 

 

Was this incident related to previous incidents? 

Yes, this incident took place over multiple days. Actions were taken during each event to improve the service what a more permanent fix could be deployment. 

  • August 29th (1st occurrence)

  • September 4th (2nd occurrence)

  • September 5th  (This status page)

 

Customer Recommendations: 

There was nothing the end user could do in this case apart from wait for the change to be applied.

Posted Oct 17, 2024 - 22:24 UTC
Resolved
Hello Everyone,

Update: Incident Resolved.

Date/Time: September 19th, 10:50 am UTC

A permanent fix has been implemented for this and monitoring has indicated that this issue is now resolved.

There will be a post-mortem published once we have concluded our investigation but no longer then 20 business days.

We are sorry for the inconvenience this has caused.

Kind Regards
Online Operations Team
Posted Sep 19, 2024 - 10:51 UTC
Update
Hello Everyone,

We have now developed a potential fix and moved it over into Quality Assurance testing. We're aiming to conclude our tests by Thursday 19th, September before moving to production patching.

The next update will be Thursday 19th, September the latest.
Posted Sep 13, 2024 - 18:43 UTC
Update
Hello Everyone,

The investigation is ongoing at this time. Results from internal testing is currently being reviewed and evaluated. When we have conclusive results, we will move to Quality Assurance testing and on to scheduling production patching.

The next update will be Friday 13th, September.
Posted Sep 11, 2024 - 08:49 UTC
Identified
Hello Everyone,

Customers have experienced issues where user or identity changes are not reflected in the UI immediately. The root cause was found to be a processing queue reaching a functional limit, which results in a backlog taking time to process. Our team is working on a resolution however this needs to run through our Development and Review processes before final deployment. Current estimates are to build this into 2024.3.1 service release. There is no set release date, but this status page will be updated next week with further updates if not sooner.

Several mitigations were put in place and while they have provided a slight improvement there are still delays in presenting the user changes in the UI in a timely manner.

This has been identified as impacting the display of changes to users or their identities in the Users > Users section of the uniFLOW Online user interface. There is no printing, scanning or general user functionality impacted at all. Changes / adding / removing of users and identities are processed as normal, but the display of these is currently delayed. This is also only limited to users having access to User\Users menu in the uniFLOW Online Portal, e.g. users with Administrator and User Manager roles.

Recommendations:
UI user changes should be made from lunchtime onwards.
Changes can still be made earlier, but the UI may not show the results for some time (possible hours).

Next Update:
The next update will be Wednesday 11th, September.
Posted Sep 05, 2024 - 10:30 UTC
This incident affected: EU Deployment (Other services).
← Current Status Powered by Atlassian Statuspage