On the September 13th Connection Issues

Yesterday Telegram experienced something extraordinary. At 17:45 UTC there was an immense spike in user activity on Telegram that exceeded our peak load by 5 times.

Telegram is different from most other internet services in that we use a distributed server infrastructure, so peak loads cannot make the whole of Telegram go down. However, if one of the Telegram clusters is severely affected, a part of our users can experience difficulties sending and receiving messages.

This is exactly what happened yesterday. Due to the spike, one of the Telegram server clusters went down in part and approximately 15% of the users who were online at the time experienced connection issues from 17:45 to 18:10. The issues were partially fixed at 18:10, but about 11% of online users could still face slow or no performance until 19:00 when the problem was permanently fixed.

The regions that were most affected are Germany, Iraq and the CIS – most notably, Uzbekistan, Russia, Ukraine, Kazakhstan and Belarus. We briefly commented on the situation via Twitter, but I’d like to tell more extensively about it here.

1. First of all, we are extremely sorry for the inconvenience this has caused. The cluster that went down hasn’t had any issues for a few years now, so I can imagine the shock of the users affected by yesterday’s downtime. We understand that you use Telegram constantly for work and leisure, and that we must be online 100% of the time. We take yesterday’s issues very seriously.

2. Secondly, to make sure this doesn’t happen again, we identified the cause of the problem and have eliminated the bottlenecks in our infrastructure that caused the downtime. We think that x5 peak load spikes won’t be a problem any longer. We've also set up a plan to be able to cope with x20 load spikes (however unlikely the x20 scenario may seem) before the end of 2017.

While unfortunately no one can completely rule out the chance of another downtime some time in the future (there’s always a slight probability of some unexpected Black Swan event), I can assure you that we at Telegram are working hard at taking this chance to the lowest in the industry.

We strive to be the winner in every aspect including availability. So far the Telegram uptime stats look good compared to other major communication services – also because Telegram can never go down entirely around the world. But we are not satisfied with just that, and our work continues.

Thanks for your attention, and sorry again to anyone who was affected.

Today, Iran joined the list of countries I can't travel to – Tehran's prosecutor just filed criminal charges against me there. More than 40 million people use Telegram in Iran and we've never blocked a single political channel and gave up exactly zero bytes of data to the government (in Iran and elsewhere).

Russia, where we have about 10 million users, is also rushing to join Iran in filing charges against the Telegram management. They seem to be unhappy because we won't comply with the unconstitutional "Yarovaya laws" and won't give them the encryption keys they wanted. I always publish such demands online, and did the same today with the documents that the FSB has been sending to our London office in the past weeks.

Because of my parents, not being able to visit Russia on occasion is more painful for me than never going to Iran. But Mom and Dad are not too old to travel and the globe is still pretty big.

Our Asian users may have noticed two nasty disruptions in our service in the past week. The Singapore data center we’re using has been causing trouble due to a faulty UPS system. Power losses are something that is never supposed to happen in a data center, so at some point I even thought that this could have been an act of sabotage.

Most likely though, it was “just” a major hardware malfunction. The faulty UPS (Uninterruptible Power Sources) are being replaced by the data center staff as I’m typing this. I hope this will make things right as the absence of power supply is one of the few things we can’t fix by ourselves.

Asia is obviously a huge market for Telegram. However, as you may remember, Telegram is blocked in China – the continent’s largest market – since 2015, when the Chinese human rights activists started using Telegram to communicate.

We didn’t try to get unblocked there by negotiating with the Chinese authorities. It’s pretty obvious that the Chinese government's desire for total control over its population is incompatible with our values. However, Telegram is still available through VPN services, and recently more Chinese users started to join Telegram after their local app WeChat got compromised –

https://www.bloomberg.com/news/articles/2017-09-15/china-s-wechat-crackdown-drives-bitcoin-devotees-to-telegram

For us this is just another indication that sticking to your principles makes more sense than yielding to pressure. In the long run, compromise based on lies and violations of rights gets you nowhere.

Why Charges Against Us in Iran And Russia Don’t Matter

Some users ask me how the charges in Iran and Russia will affect Telegram and me personally. The simple answer to this is: they won’t.

It’s easy for me not to travel to Iran – while the country has a rich history and sounds like a fun place, I have no connections with it and can live with that travel ban. It’s a bit more noticeable in case of Russia since I am an ethnic Russian (although with a St. Kitts passport), and my parents live there. Not being able to occasionally drop by is not great, but a small price to pay considering the matters at stake.

Luckily, we don’t have any legal presence in any of these countries. The last remaining link between Telegram and Russia was cut in July, when we terminated our contract with “Telegraf” – a Saint-Petersburg-based company to which we outsourced fighting spam coming from (surprise) Russia and Iran.

“Telegraf” used to play a larger role in the early days of Telegram when I was living in Russia, losing its importance after I left in 2014 with our core team.

Following the events this June when the Russian authorities threatened to block Telegram, “Telegraf” lost its outsourcing contract from Telegram Messenger, let go all of its employees and changed its owner. By September when the Russian authorities started sending warnings to our London office, they had nobody to target in their jurisdiction, not even Russian spam moderators.

This story highlights something local regulators often tend to ignore: it’s 2017, and the world is open and connected. If you pass archaic laws that limit freedoms, all you’ll end up doing is killing your own economy. In the last few years, Google, Oracle and Microsoft shut down their development offices in Russia, and many smaller companies followed suit.

While the state of affairs in Iran doesn't look much brighter than in Russia, things seem to be going in the right direction there compared with the situation a few years ago. I’m not an expert on Iran, but one thing about the country is clear to me: despite the continuing debate among the Iranian politicians on how to regulate the Internet, Iran is not blocking Telegram, and for the last few years 40 millions Iranians have been able to securely communicate and to get news from independent sources through Telegram channels. Instagram and WhatsApp are also accessible there.

For any European this would sound like the norm, but unfortunately it’s not always like this in the world. The Chinese and the North Koreans, for example, are far less lucky when it comes to such freedoms, and Saudi Arabia had been throttling Telegram’s traffic until recently. The situation in Iran itself used to be very different: almost all major internet services were blocked in the country several years ago (some of them still are).

Don’t get me wrong: there are probably many things that Iran can change for the better (a more IT-friendly prosecutor of Tehran, perhaps?), but overall it seems that the country is moving in the right direction by becoming more open and market-driven.

I hope that one day Iran and Russia get to a point when we (and other IT companies) will be able to set up offices there. Until then, we’ll continue providing secure messaging to users in these markets from places that respect freedom.

We don't care if specific countries press charges against us for defending the privacy of our users. We are always ready to cut all our personal and business links to such places so that they don't have any leverage on us. They can try to block us on their territory, but, as I've shown in my previous post about China, even this won't always help them. Eventually freedom and privacy will prevail, and those who would like to get back to the 1930s will find themselves on the wrong side of history.

You already know Telegram has your back when it comes to privacy. Another human right we feel passionately about is the freedom of speech. In some countries Telegram is the only popular platform for independent sources of information. Take Iran for example, where the BBC is blocked, but its @bbcpersian channel has been active for years and attracted almost a million followers. There are thousands of large Telegram channels that are critical of their governments.

Telegram has never yielded to pressure from officials who wanted us to perform political censorship. Freedom of speech is one of the values we’ve been defending for the last 11 years, first in Russia, and then globally.

Of course, there are rules that admins of Telegram channels should respect, for example, Telegram ToS prohibit publicly promoting violence in any form. This is why this October alone we have blocked over 8,500 channels related to terrorism (more stats in @isiswatch).

Another recent example of a line one shouldn’t cross is an Iranian channel that started to urge its members to throw stones into the windows of public buildings and vehicles (schools, temples, buses) and film it. We got in touch with the channel admins and asked them to stop this vandalism contest; not only did they ignore us, but launched another creepy competition urging their 100K+ users to burn mosques by throwing Molotov cocktails into them and film it. As a result, we were left with no other options but to block their channel.

For us, the line is pretty straightforward. Criticizing local authorities, challenging the status quo and engaging in political debate are OK. Meanwhile, promoting violence and calling for actions that can harm innocent people are not OK. This is a rule of thumb we’re always applying when moderating public channels.