We wanted to provide you with some additional information about the service disruption that occurred for messages sent from the Email2SMS platform on November 19th, 2022.
At 11:00 on November 19th, during a planned migration of our Email and Exchange services we identified that mail routing rules had been impacted, introducing a degradation where certain customer mail2sms messages were identified as spam or dropped prior to sending. This failure resulted in intermittent messages not sending for 19th November 2022 between 10:24 and 17:42.
Normally, our Exchange and mail capabilities have multiple layers of resilience and redundancy to prevent outage and incident. Due to the nature of the planned migration this did not apply and introduced the degradation we identified.
Between 10:24 and 17:42 AEST on November 19th, Email to SMS (E2S) messages were not delivered.
API and web-portal sending was unaffected.
During the deployment the team had received notification that delivery rates had changed outside of expected thresholds which triggered remedial action to identify and resolve the mail routing rules in question and correct. Due to the number of rules and replication needs this contributed to the longer than desired recovery period.
While we have experienced optimal operational performance since our change and routing update, it is apparent we need to enhance this migration approach and resiliency moving forward. Whilst this type of activity is not predicted to occur in the near future, further work to ensure greater resiliency and failover is available within this element of the service.