User Impact
Customers have experienced issues where user or identity changes are not reflected in the UI immediately. This could be observed by tenant administrators and end users making changes to their own or other user information.
It is important to note that the changes, in all cases, happened. The issue was the UI did not reflect the implemented change.
Scope of Impact:
EU Deployment
The impact of this was observed across all tenants within the deployment.
Incident Start Date and Time
September 5th , 2024
Incident End Date and Time
September 19th , 2024
Root Cause:
On investigation it was identified that the processing of user information was delayed. It was identified that feature inclusions over 2024.1 and 2024.2 made more use of this process and became less responsive under high load as seen in the EU (our largest deployment).
How did we respond:
Addressing this issue was a multi-step process during identification and remediation:
From the initial event we investigated both the potential cause of the high load as well as the responsiveness during these periods.
On 4 September 2024 we applied mitigation in an attempt to reduce the high load, but this, unfortunately, did not lead to a large improvement as the issue reoccurred on 5 September 2024.
On 5 September 2024 it was decided to raise an extended recovery status page while we continue to analyze the root cause and provide a fix.
The root cause was found and the processing resolved with a global maintenance update to uniFLOW Online.
Next Steps :
We apologize for the impact on affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):
The issue has been resolved, (Already deployed globally).
Improved and optimized event processing.
User, Cost Centre, Print job sharing and scan delegation user changes.
Was this incident related to previous incidents?
Yes, this incident took place over multiple days. Actions were taken during each event to improve the service what a more permanent fix could be deployment.
August 29th (1st occurrence)
September 4th (2nd occurrence)
September 5th (This status page)
Customer Recommendations:
There was nothing the end user could do in this case apart from wait for the change to be applied.