User Impact: 

Customers noticed the device UI (User Interface) had changed or gone offline. This was an unintentional action which resulted in the new UI being deployed to devices without customer confirmation. As some devices were not capable or configured for the new UI this would take them offline. 

Scope of Impact: 

Australian and Singapore Deployments 

Incident Start Date and Time: 

Sep 6, 2023 - 09:00 UTC 

Incident End Date and Time: 

Sep 7, 2023 - 07:30 UTC 

Root Cause: 

NT-ware has identified the cause of the new device UI behaviour being deployed. The intention was that the new UI was only pushed out onto new tenants. The criteria for these decisions were any tenant that had not previously saved the new device UI option. This meant that even if you stayed with the default device UI uniFLOW Online wanted this saved. 

As many people did not save the default device UI this left the setting blank and resulted in the misidentification as a new tenant. 

For many customers this simply meant the UI changed to the modern look. Unfortunately, this had a negative impact for a number of tenants where the forced update pushed this onto devices with an earlier MEAP version. In this configuration there was a known issue where specific TLS setting on the device would force the device offline. 

This issue also impacted uniFLOW Online Express customers in the same way. However, the express tenants do not have the admin options available to revert this change. 

Key Event Timeline: 

1. Deployment update was started at 9:00 am UTC on September 6th 

2. Support tickets from customers impacted by the change were asked to switch the device UI back to classic and save the setting via the tenant admin.  

   1. This was an in-place mitigation while the root cause was corrected. 

3. 5:30 am UTC 7th Sept a permanent resolution was completed, and testing was finalised. 

   1. Express devices will be reverted to the classic device UI during this process. 
   2. Tenants that have NOT saved the device UI previously will be reverted to ‘Classic UI’. 
   3. Tenants that have saved the device UI will remain unaffected by this rollback. This will include all customers that performed roll back procedures during support calls. 

4. Confirmation from the field that the corrective actions were taking effect and remaining devices not manually reverted back, (point 2) were returning to the ‘Classic UI. 

It should be noted that while the deployment took effect 09:00 UTC on the 6th a UI change can take up to 24 hours to completely propagate to all tenants (customers). Reverting this setting the remaining devices could take the same amount of time. Restarting the device would force a reconnection and this took immediate effect. 

Next Steps: 

We apologize for the impact to affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to): 

• The incident has been reviewed by department heads and senior management. 
• Improvements in our deployment process have been identified and being actioned. 
• We have reviewed our communication plan and will further improve this.