Temporary SMS delivery issues

Incident Timings

Start time: 23 Feb 07:00 UTC
End time: 24 Feb 14:00 UTC

Incident Summary

Some of our clients reported a decline in SMS delivery rates. After switching to the fallback SMS sending method, delivery remained stable for some time but began declining again after several hours. As a result, a small portion of applicants were unable to complete verification because they did not receive SMS notifications. Once the underlying issue was identified and resolved, our team ensured that no further action was required from clients. Affected applicants can complete the verification process by simply retrying the SMS step.

Root Cause

A significant volume of untracked SMS messages caused by a technical issue. These messages were sent in unusual way, allowing to bypass both our system’s and the SMS provider’s rate-limiting mechanisms. This resulted in an unusually high spike in message traffic, leading to message queues, with some messages timing out and not being delivered.

Action Plan

As a result of this issue, our Team has identified several key areas for improvement:

1. Enhance Monitoring & Visibility: Improve alerting for SMS delivery
2. Prevent Recurrence: Enhance rate limiting and throttling
3. Collaborate with SMS providers: Coordinate with our SMS service providers to improve anomaly detection mechanisms from their side and introduce additional fallback options

Conclusion

If you are still experiencing any unusual SMS delivery rate fluctuations or if you have any questions or concerns please contact our Support Team.