81 lines
7.0 KiB
Markdown
81 lines
7.0 KiB
Markdown
# The complexity of phone networks
|
||
|
||
- **期号**: SRE Weekly Issue #436（2024-08-04）
|
||
- **作者**: Leo Sjöberg — incident.io
|
||
- **链接**: https://incident.io/blog/the-complexity-of-phone-networks
|
||
|
||
## 简介
|
||
|
||
It turns out sending texts and making phone calls automatically is really hard, and many assumptions you might make turn out to be wrong.
|
||
|
||
## 正文
|
||
|
||
July 16, 2024 — 7 min read
|
||
|
||
Arguably the most important part of an [on-call product](https://incident.io/on-call) is knowing that you will be notified when things break, wherever you are. When it comes to SMS and phone call notifications, we have to leave the familiar realm of the internet and JSON responses, and deal with systems that provide limited observability and insight into what’s gone wrong.
|
||
|
||
A lot of things can go wrong when you try to send an SMS or make a phone call. To understand why, it’s useful to understand how an escalation like "*Production is down"* goes from our systems to your phone.
|
||
|
||
When we make a call or send an SMS, we send an API request to a telecom API provider, who will place that call on our behalf.
|
||
|
||
Our provider then forwards that request to one of their telecom network partners. Which partner it’s sent to will depend on which number we’re sending from: for example, it might be AT&T when we call from a US number, and Three when we call from a UK number.
|
||
|
||
Once it reaches the network partner, it mostly works like a regular phone call or SMS: the carrier will connect to whichever network your phone is on, and establish the call or deliver the SMS, and the rest is in the hands of your carrier.
|
||
|
||
But with this many steps, things are bound to go wrong. Especially when, through all this, our telecom API provider tries to give us status updates on the call or SMS. We should be getting information about the call being established or why it failed, and when an SMS enters their systems, is sent from their partner network, and when it’s received by your carrier.
|
||
|
||
Normally, this all works as it should, but telecom is an old business, and sometimes, getting an error code of “the user didn’t pick up”, doesn’t *actually* mean that the user didn’t pick up…
|
||
|
||
So here are some things you might’ve thought were true about phone calls and SMS.
|
||
|
||
`Busy` means the user can’t pick up
|
||
When a user can’t pick up the phone, such as when they’re already on another call, you get the response code `486 Busy Here`. It *should* mean the person you’re trying to reach is busy. But some carriers will reject calls from international numbers with a 486 code, instead of a `400 Bad Request` or `403 Forbidden`. This isn’t something that’s even tied to a specific country. Instead, it’s all decided carrier-by-carrier, and there’s no registry of how different carriers respond. It’s all trial and error.
|
||
|
||
Our primary telecom API partner, Twilio, has a concept of “high-risk numbers”. A high-risk number should be one that’s identified as having a high risk for [*toll fraud*](https://www.twilio.com/docs/glossary/what-is-toll-fraud). So for our trusted users who want to get a phone call when production goes down, we shouldn’t need to worry about high-risk numbers, right? Wrong. High-risk numbers are number *ranges*, and while some numbers in that range might have a high risk of toll fraud, there will likely be plenty that are perfectly legitimate.
|
||
|
||
`Sent` SMS means it’s been received by the target network
|
||
When we send SMS, they go through stages of *Queued* → *Sent* → *Delivered*. *Sent* indicates that the SMS was sent from our API partner’s telecom network partner (e.g. AT&T in the diagram above), and *Delivered* (or *Undelivered* if it’s rejected) indicates that the target carrier (your phone network) accepted the SMS.
|
||
|
||
At least, that’s how it’s supposed to work. Some carriers, like Ireland’s *Eir* network, don’t accept international SMS, but instead of responding with a rejection, we just get *nothing!* To work around this, we now mark outbound SMS as “stuck” if they don’t move to a terminal state quickly enough, and then retry sending from a different number.
|
||
|
||
In both the UK and the US, it’s pretty straightforward to buy an outbound number to send SMS or make calls from. You pick what you want to do with the number (make phone calls and send SMS), fill out a regulatory registration form to detail how you’re going to use the number, and pay some money. If you want a shortcode number for SMS, you fill out a shortcode application, wait a couple of weeks, and pay some money.
|
||
|
||
But try buying a phone number in Latvia - you can’t. Or a shortcode in Ireland? Tough luck, we were told the last time a shortcode became available was about a year ago! What about Germany, one of the biggest economies in the EU? You’ll need a German business address and business registration for that, yikes.
|
||
|
||
Some countries, notably China, apply rules for sending SMS depending on the time of day and the urgency of your SMS 🙃
|
||
|
||
We use shortcode numbers, like *65 956*, to send on-call notifications in the US and UK, to improve our deliverability. However, some UK carriers prevent their customers from sending SMS to a shortcode number (which we rely on for people to acknowledge escalations), and it’s not even configurable!
|
||
|
||
When we don’t have a local number to use for SMS notifications, we usually fall back to one of our UK numbers. Now, if we’re able to send an SMS to a user in another country, and they receive that SMS, surely they can also reply, right? Nope. Even if someone is able to send international SMS, and can send SMS to a different number in the UK (or any other country we send from, we tried Sweden too), their carrier might still be unable to pass the SMS message to our telecom network 😭
|
||
|
||
These are some of the things we’ve learned from our relatively short time working in the world of telecom. If you’ve spent some time banging your head against carrier regulations, SIP codes, and incomprehensible request diagrams (have you ever *seen* a SIP signaling diagram?!), and have a few more things you think we need to be aware of for building a world-class on-call product, we’d love to hear from you.
|
||
|
||
Leo Sjöberg
|
||
|
||
Product Engineer
|
||
|
||
We like to ship it then shout about it, all the time. Building On-call was different.
|
||
|
||
Henry Course
|
||
|
||
July 9, 2024
|
||
|
||
Our customers count on us to sound the alarm when their systems go sideways—so keeping our on-call service up and running isn’t just important; it’s non-negotiable. To nail the reliability our customers need, we lean on some serious observability (or as the cool kids say, o11y) to keep things running smoothly.
|
||
|
||
Martha Lambert
|
||
|
||
August 22, 2024
|
||
|
||
What does it take to build a greenfield mobile app in 2024? When we launched On-call earlier this year, we had to find out.
|
||
|
||
Rory Bain
|
||
|
||
July 15, 2024
|
||
|
||
Ready for modern incident management? Book a call with one of our experts today.
|
||
|
||
- All-in-one incident management
|
||
- Our unmatched speed of deployment
|
||
- Why we’re loved by users and easily adopted
|
||
- How we work for the whole organization
|