User Impact:
Customers noticed the device UI (User Interface) had changed or gone offline. This was an unintentional action which resulted in the new UI being deployed to devices without customer confirmation. As some devices were not capable or configured for the new UI this would take them offline.
Scope of Impact:
Australian and Singapore Deployments
Incident Start Date and Time:
Sep 6, 2023 - 09:00 UTC
Incident End Date and Time:
Sep 7, 2023 - 07:30 UTC
Root Cause:
NT-ware has identified the cause of the new device UI behaviour being deployed. The intention was that the new UI was only pushed out onto new tenants. The criteria for these decisions were any tenant that had not previously saved the new device UI option. This meant that even if you stayed with the default device UI uniFLOW Online wanted this saved.
As many people did not save the default device UI this left the setting blank and resulted in the misidentification as a new tenant.
For many customers this simply meant the UI changed to the modern look. Unfortunately, this had a negative impact for a number of tenants where the forced update pushed this onto devices with an earlier MEAP version. In this configuration there was a known issue where specific TLS setting on the device would force the device offline.
This issue also impacted uniFLOW Online Express customers in the same way. However, the express tenants do not have the admin options available to revert this change.
Key Event Timeline:
Support tickets from customers impacted by the change were asked to switch the device UI back to classic and save the setting via the tenant admin.
5:30 am UTC 7th Sept a permanent resolution was completed, and testing was finalised.
Confirmation from the field that the corrective actions were taking effect and remaining devices not manually reverted back, (point 2) were returning to the ‘Classic UI.
It should be noted that while the deployment took effect 09:00 UTC on the 6th a UI change can take up to 24 hours to completely propagate to all tenants (customers). Reverting this setting the remaining devices could take the same amount of time. Restarting the device would force a reconnection and this took immediate effect.
Next Steps:
We apologize for the impact to affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):