1 of 48

Russia vs. Telegram

technical notes on the battle

Leonid Evdokimov

35c3, Leipzig, 29 Dec 2018

darkk.net.ru/35c3

2 of 48

$ whoami

Internet measurement fanatic

NOT a Telegram team member

One of the millions of Telegram users

3 of 48

Brief history

4 of 48

Pre-blocklist era

2007 May 23: court order for 4 (four) ISP to block access to “extremist” websites

2007 Jul 14: the 1st issue of the “Federal List of Extremist Materials”�by Ministry of Justice

First, attempts to filter Russian internet segment is not a recent development. There were some activities in that area for 11 (eleven) years.

The first documented incident of internet censorship in Russia happened in May of 2007.

Court ordered four ISPs in Novosibirsk academic campus (known as Academgorodoc) to ban some extremist websites.

Those were small internet service provides, so only small fraction of users was affected and the incident have not made loud news story.

The registry that was used to block content in that ages was named “Federal List of Extremist Materials”, but it was quite badly maintained.

For example, one of the entries is exactly following: “Text document named “Terrorism” from the folder named “Orders1”, CD disk number 1”.

So the list was not very useful for the purpose of automatic content blocking.

----

But Russia has several thousands of small ISPs, and those small ISPs still control ~one third of overall broadband market share despite all ongoing centralisation when five largest ISPs control 60% of the market.

5 of 48

Librarian’s resistance

2011 Feb: www.zhurnal.lib.ru is banned

Maksim Moshkow “transfers” domain to the Ministry of Justice (via DNS “A” RR)

Some ISPs block minjust.ru ¯\_(ツ)_/¯

6 of 48

Blocklist awakens

2012 Jul 10: Wikipedia strikes, Yandex & VK protest

2012 Jul 11: the internet restriction bill accepted by Duma (Parliament)

2012 Jul 28: the bill signed

7 of 48

What’s the blocklist like today?

XML file, signed by CN=Roskomnadzor with GOST, fetched by ISPs via SOAP, updated at least hourly.

ISPs control filtering equipment. Roskomnadzor monitors it.

8 of 48

Blocklist 2012 awards

8 Nov: Absurdopedia (Uncyclopedia)

11 Nov: Lurkmore memepedia, lib.rus.ec

17 Nov: Github repo with blocklist leak

18 Nov: Google’s https://….gstatic.com

When the blocklist first appeared in the November of 2012 it immediately delivered lots of fun.

For example, it blocked absurdopedia for Suicide HOWTO.

Of course, the grotesque of blocking of humorous content got some media attention.

And some brave creature decided to leak the blocklist to github to codify that absurd.

Two weeks after the first blocklist release github and sourceforge got a copy of the blocklist.

And that copy was kept up-to-date for years and it’s still maintained.

I don’t know anyone who knows the person maintaining that blocklist mirror, so this talk is the only way for me to thank that maintainer for all the work done.

And, of course, within a few weeks some of Google’s services were also banned… For a day or so… Anyway, who needs jquery libraries from g-static-dot-com.

9 of 48

Blacklisted resources in pre-Telegram era

Web Archive, GitHub, Google, LinkedIn, Pornhub, Reddit, VK, Wikipedia…

Comodo CA CRL & OCSP responders

127.0.0.1 (sic!)

I can spend whole talk time-slot describing funny incidents regarding the blocklist, but it’s not the main topic today.

But I can’t resist, so I’ll name a few more.

First, Pornhub was not only banned, but it was also unbanned.

As far as I know, pornhub was a pioneer that used legal procedure to unblock the website and that legal fight deserves some kudos.

Second, Roskomnadzor loves TCP-dump for sure.

That’s the only sane way to explain how Certificate Revocation Lists and OCSP responders of Comodo Certificate Authority were banned.

That CA ban caused some quite solid collateral damage and hard-to-debug problems.

Third, Roskomnadzor either has some sense of humour or just lacks sanity checks in blacklist maintenance procedures.

But it really banned loopback for a while.

10 of 48

“Revisor” — Roskomnadzor’s monitoring system

The law does not matter. The fine does.

2016 Jan: OpenWRT-based TP-Link MR3020, that was talking with C&C via https API without ca-certificates and via ssh without known_hosts

11 of 48

ValdikSS

12 of 48

“Revisor” — complying with blackbox

No codified monitoring rules, just FAQ

Some ISPs reverse-engineer it

Some ISPs comply at best-effort

Some ISPs place it into a “sandbox”

13 of 48

Logo of Revisor-devoted Telegram chat @i_love_auditor

14 of 48

“Revisor” & DNS

ISPs are forced to comply with the black-box monitoring system

Stale IPs in dump.xml, “Revisor” using DNS… ⇒ ISPs feed A & AAAA from DNS directly to filters

15 of 48

“Revisor”-provoked so called “DNS-attacks”

2017 May 15: block IP from DNS? Bo-om!

Adding /32 from DNS to routing table?

2017 Jun 7: drop IX peers!

2018 Mar 14: routers go on strike!

Those manipulations were called “DNS-attacks”.

There were various sorts of them.

Some of ISPs used IP addresses from DNS to actually block traffic.

Those ISPs brought their users some collateral damage when domain names were pointed to IP addresses of “good” services.

Other ISPs used IP addresses to steer some portion of traffic towards DPI boxes to reduce overall amount of bandwidth going through DPI.

That’s very smart and saves some electricity, but unfortunately it may lead to most-specific routes overriding “ordinary” routing rules when domain names were pointed to IP addresses of “peering” subnets.

Or, in the worst case, routers can go on strike when blacklisted domain names are pointed to a million and a half of distinct IP addresses and overflow router’s TCAM memory.

As you can guess, all of those incidents have already happened.

16 of 48

Telegram:�Policeman Enters The Game

17 of 48

Telegram? Why?

2017 Apr 7: St.Petersburg bombing

2017 Jun 26, FSB: “terrorists used TG”�RKN promises to block, counts days.

2017 Jun 28: Telegram added to the “Information D istributors Registry”

18 of 48

Telegram non-compliance

2017 Dec: Roskomsvoboda starts legal campaign Telegram vs. FSB

2018 Mar 20: court orders Telegram�to pass encryption keys to FSB

2018 Apr 16: RKN attempts to block

19 of 48

Civil cyber-war: leak of BGP-Blackholing letter

Mar 23: Mikhael Klimarev publishes leak

RKN plans ban of 15M IPs: 36 subnets�of Amazon, SoftLayer, … to block Zello.

Keywords: Null0, BGP, redistribute.

But let me give some more bits of context.

Telegram was not the first messenger that terrorists love. Zello was another one.

Zello was also used by protesting truck drivers, but maybe those drivers were meant to be the terrorists… Who knows. Anyway, once again, that’s out of the scope of the talk.

So, attempts to block Zello were ongoing since April 2017 and they were not a complete success.

And there was a leak in the end of March 2018 saying that Roskomnadzor plans to conduct an experiment with new tactics regarding blocking of Zello.

The leak had 36 subnets summing down to 15 million of IP addresses of Amazon and other cloud providers and it had nice keywords like “Null routing”, “BGP”, “redistribute”.

I’m unsure what they actually meant, I don’t know if the leak was genuine. But!

There was a lot of noise in the media regarding the leak.

There were NO reports that could confirm alike experiment happening.

20 of 48

RKN-tan tries�to block 14 million�IP addresses of Amazon hosting half of Internet

– @aquam1ne

21 of 48

Civil cyber-war: April, 16th, TZ=MSK

11:39 RKN bans TG’s ~/19, no effect�17:58 bans Amazon’s ~/13, TG works�18:33 adds missing TG’s /24 ¯\_(ツ)_/¯�20:21 Google’s /12, Amazon’s /15…

1.8 M IPs banned, Telegram is ~fine

22 of 48

Civil cyber-war: that escalated quickly

Apr 16: ~ 1.8 M banned IPs

Apr 17: ~ 16 M

Apr 22: ~ 19 M, local peak

23 of 48

Civil cyber-war: IP space sanity checks? Ha!

Overlapping subnets in blocklist:�52.0/11 ∩ 52.28/15�34.192/10 ∩ 34.240/13�52.192/11 ∩ 52.208/13�…

24 of 48

Civil cyber-war: URL sanity checks? Ha!

Malformed URL in blocklist:

<![CDATA[http:// 46.101.189.65]]>

^ whitespace

Guess, what filter do?

25 of 48

Civil cyber-war: major services by mistake? Ha!

RKN: significant ones are not affected�Affected: ~34 k .ru, .рф, .su services�Affected: vk.com (87.240.129.133)�Affected: Yandex.Metrica (213.180.193.119)�Affected: Yandex ads (77.88.21.90)

During the first days of cyberwar Roskomnadzor has distributed a document with some white-list of socially-significant services that included websites like kremlin.ru, those should not be banned under any circumstances. Probably that was a response to numerous reports regarding collateral damage.

Unfortunately, it has no legal framework for alike actions so many ISPs just ignored that document. But it had caused some media wave as soon as white-list coming from censorship agency was something new for Russia that days.

Anyway, the mantra regarding socially-significant services was repeated over and over again.

�But thousands of dot-ru, dot-su and dot-rf domains were affected according to publicly available statistics from beget hosting provider. �

Also the counter of Yandex.Metrica and Yandex banner system endpoints were banned at midnight of April the 27th together with vk.com, VKontakte, the most popular social network in Russia.

The incident lasted for almost two hours and it clearly showed that:

First, Roskomnadzor works selflessly round the clock.

Second, Roskomnadzor still loves to feed some IP addresses from tcpdump right into blacklist without much analysis as it previously happened with Comodo CA.

26 of 48

Civil cyber-war: fakenews? Sure!

RKN: “Google Play, Google Drive and google.ru IPs were not banned”

Data: dozens IPs of load balancers discovered via EDNS Client Subnet are actually blocklisted

27 of 48

G.DNS

28 of 48

Civil cyber-war: ISP non-compliance

But it was not the only funny visualisation we made.

On April the 19th we took all RIPE Atlas probes in Russian networks that observed filtering, pointed them to randomly selected TLS endpoints in filtered subnets and crafted a live whack-a-mole dashboard. The dashboard showed ISPs practicing “digital resistance” basically.

The reason to create alike dashboard was clear for me: I did not want huge ISPs those are too-big-to-fail to use a possibility of non-compliance as a competitive advantage against smaller independent ISPs who can’t take the risk of paying fines as fine amount is significant for them.

Members of ISP community asked us to hide the dashboard within half an hour for the reason I still don’t quite understand. The cropped dashboard screenshot shows only top-4 ISPs, so I believe it’s okay :-)

And non-compliance story still goes on at least for Rostelecom the largest ISP holding ~30% of B2C broadband market share. Experiments have shown that Rostelecom does not block traffic to blacklisted IPs and subnets completely. It blocks just the http and https TCP ports and, for example, SSH works. And, once again, that’s state-owned ISP holding one third of the market!

29 of 48

Delayed compliance example, RIPE Atlas data

30 of 48

Civil cyber-war: lawful interception?

Sniffers used to hunt proxies?

28 Apr: public “tip”, 30 Apr: private tip

Unsecured SORMs, pumping 20 Gbit/s, leaking rpm repo, clickstream and PII?!

As the civil cyberwar continues, people use proxies and Tor to access telegram and get educated on censorship circumvention in general.

Artists craft parody songs about proxies and mock Roskomnadzor, so people have great share of fun.

But Roskomnadzor hunts proxies as well as Telegram and people wonder, how do they do it.

Other people start suspecting that lawful interception equipment (also known as SORM) is used to hunt proxies and tips come via public and private channels highlighting instances of some sniffers.

Basically, the instances were exposing traffic statistic publicly and that traffic statistics also had a section on “Telegram” traffic that attracted the attention.

We scraped statistics pages and it turned out that all those instances were pumping quite small amount of traffic, just twenty gigabits per second. But the shape of the chart looked like real traffic.

We looked up those instances using Shodan and we have seen that some instances had public FTP repo with SORM binaries.

And, what was significantly worse, some instances were leaking users’ clickstream and some personal information. Of course, all that data was available without any authentication.

31 of 48

SORM: this incident was reported

32 of 48

Protest meetings because of app ban!

33 of 48

Civil cyber-war: Morse prank

D I G I T A L R E S I S T A N C E

But we assumed, that media attention may help to reduce collateral damage caused by massive subnet blocking. And we decided to do some performance.

I’ve previously described “DNS-attacks” that caused a major outage of Transtelecom Internet Service Provider.

One of the outcomes of that incident was a chart, monitoring count of IPs, those are pointed by blacklisted DNS entries.

Basically, if the chart skyrockets, it may be an indicator of ongoing DNS attack of that sort.

So, ISPs were watching that chart closely and it was a nice canvas for some drawing.

I've previously registered some expired blacklisted domains for network measurements, to present the results of the measurements at the roundtable at the Parliament. And those domains were still alive.��So I took some of them and added some dynamic DNS zone script.��The script was updating the zone, so nice Morse code appeared on the chart. And the plotted phrase was "digital resistance".�

34 of 48

Civil cyber-war: Morse prank

Countdown (cheap drama)

35 of 48

Civil cyber-war: Morse prank

“Truly, Popov!” – Radio Day greeting

36 of 48

Civil cyber-war: prank-effects�

Nice amplitude fade-out (thanks, RKN!)

“&.” TLD flash-blocking

15 M → 11 M banned IPs

Expired domains blocklist cleanup

And the fade-out was not the only visible effect that happened during the prank.��First, Roskomnadzor has banned and quickly unbanned dot-ampersand TLD. It's still unclear to me if it was just a mistake or some sign of unity or solidarity.��Second, amount of banned IPs dropped significantly on May the 8th, from 15 to 11 millions.��Third, Roskomnadzor started to remove expired domain from the Registry reducing possible impact of various DNS-attacks.��At first that seems to be hype-driven cleanup, but the proper expired domains maintenance persisted. And it is good for overall network stability.�

Of course, all but the cleanup may be just a coincidence.

------

37 of 48

Civil cyber-war: partial rollback

28 Apr: 19 M → 15 M (protest)�8 May: 15 M → 11 M (prank?)�8 Jun: 11 M → 3.7 M (?)�7 Jul: Open Letter on collateral damage had no effect, still ~3.7 M

Another unblocking happened on the 8th of June and it's unclear what was the trigger of it.��All those clean-ups were followed by people computing the statistics of IP addresses those were remaining banned after subnet unban.��The number was less than 1%. So, lots of people were sharing an opinion that Roskomnadzor basically takes those subnets as “hostages” to push Telegram away from huge computing clouds controlling huge portion of IP address space.�

I was unsure that Roskomnadzor indeed used the terrorists tactics while fighting with terrorists using Telegram. I had an assumption that leftovers of collateral damage could also be explained by incompetence or lack of “big” players in those subnets.

So I’ve spent some days mining Open Data on DNS and I’ve written an Open Letter to Roskomnadzor giving a overview of blocked web resources including, but not limited to, Slack, Jitsi, PowerDNS, Doodle, Python mailing lists, Netlify and, anus-dot-com, the website of American Nihilist Underground Society. The letter was also duly sent to corresponding legal online form.

�Some answer arrived but it looked like pointless legalspeak to me.

So, now I completely understand those people, who say that Roskomnadzor took those subnets as “hostages”.

38 of 48

PCAP or it didn’t happen

39 of 48

Selective protocol throttling

TG speaks Socks5, MTProto, MTproto-dd

~7500 kbps: Socks5, HTTP xor RC4

~22 kbps: MTProto, obfs4, `nc urandom`

Camouflage matters!

One of observations we made during that part of the incident was selective protocol throttling.

Telegram client speaks several protocols to the proxy: generic Socks-five protocol, and two variations of Telegram-specific look-like-nothing M-T-Proto protocol.

And I discovered that one of major mobile broadband ISPs does not treat those protocol equally. Socks5 was working well and MTProto was heavily shaped.

At first I assumed that all look-like-nothing protocols were shaped as obfs4 pluggable transport and bare “netcat u-random“ were throttled as well.

But, thanks to Simone Basso, we’ve conducted an experiment and confirmed that it’s not the case. HTTP protocol xored with RC4 stream was absolutely okay.

So, it’s unclear if the shaper is targeting MTProto or some other protocol, but it’s definitely more complex than I originally assumed.

And, of course, it means that camouflage matters a lot while designing pluggable transports.

40 of 48

Proxy-hunting: MTProto

pkt.len-based hunting was noticed

Rostelecom was part of the experiment

Any IP:Port may be killed by “knocking”

Reuters: “alike experiment happened”

Another proxy servers availability issue was observed in one of Rostelecom subnets in the south of Russia.

MTProto proxies operating without random padding were banned in Rostelecom almost instantly.

It turned out, that the DPI equipment was observing sizes of the packets coming from clients.

And it was trivial to trigger DPI equipment into resetting the TCP stream just sending the packets to the TCP port that was listening.

But there was a thing that really bad in that setup. The IP and Port of the network endpoint was added to dynamic blocklist at that DPI box for a couple of hours.

We assumed that the vulnerability could be possibly exploited from the browser to ban good web resources, so we decided to disclose that shady practice immediately instead of researching it further.

And there was a Reuter’s news article couple of months later, that kinda confirmed that alike experiments on live traffic were actually happening.

It’s unclear to me if alike experiments are legal within current blocking framework, but some people have opinion that they are actually a crime.

41 of 48

Proxy-hunting: Socks5 in Moscow subway

One uses Socks5 in subway
Nmap scans IP:Port
Socks5-scanner tries connect(TG)
IP unreachable via some ISPs
IP officially blocklisted

Another proxy servers availability issue was observed in Moscow.

Thanks to St. Petersburg CTF team, they’ve dissected the story.

First, the user connects to Socks5 server from Wi-Fi access point in Moscow subway.

N-map-like scanner comes to the server in a ~half an hour and checks TCP port availability. I say n-map as various packet fingerprints matched with this tool perfectly.

If the port is available then another scanner comes. It tries to establish a Socks5 connection to Telegram cloud without any Socks5 login and password and exchanges some data with Telegram servers.

If all those active checks are okay, then the IP address becomes blacklisted.

Once again, it’s unclear to me if alike proxy-hunting is legal within current blocking framework, but some people have opinion that it is a crime.

That story have attracted some non-zero media attention from local and international media. But the most interesting part of the story was left unnoticed.

42 of 48

The only reason for me to replicate that study in Moscow subway was the ability to manipulate the blocklist and add IP addresses of my servers to it.

It allowed me to conduct an experiment that I consider extremely important. We were able to measure the timeline of the blocklist rollout using diverse set of RIPE Atlas probes.

Of course, I would like to repeat the experiment with the spooky scan announced by Roya Ensafi on the first day of the congress to cover more networks :)

So, the statistics we got was very interesting. It allowed us to verify the tip that we got in May. The tip was saying that there are actually two tiers of ISPs with regards to timing of blocklist execution.

And that some ISPs start blocking IP addresses before oficial digitally signed XML blocklist is actually delivered via SOAP API.

Once again, it’s unclear to me if presence of those two tiers is legal within current blocking framework, but some people, who are quite educated in legal stuff, they have an opinion that it is a crime.

43 of 48

Secret blocklist private to some ISPs?

> 4. IP unreachable via some ISPs

Some other blacklists exist… regional?…

…at least List of Extremist Materials

Block-race is still observed

My personal opinion is that it’s not necessary a crime. There are other sources of “blacklisted” endpoints besides official Roskomnadzor XML blacklist.

For example, there is still the Federal List of Extremist Materials and some ISPs were using that in addition to well-formed XML blacklist.

Also, Digital Security Lab Ukraine reported that Crimea has both regional blacklist and federal blacklist. But I know nothing about stuff happening in that region. Go and ask Xenia who may be in this room if you want to know more.

Also, there were private tips coming from Russian Caucasus about some regional blocklists there, but we were unable to verify those tips.

The thing we know with high level of confidence is that some regional “Governmental Internet Blackout” happened in Ingushetia this autumn during protests.

Another thing we know is that ISPs race to block the IP address before official blocklist publication is also still observed.

All in all, Russia is Federation, so, maybe it is actually legal to do stuff that way.

44 of 48

And then things got worse

45 of 48

Roskomnadzor: The Phantom Menace

RKN deploys “anti-threat” equipment

That also acts as filter

RKN directly controls IP routing & DNS

Registry of “good” Internet Exchanges

There was a law introduced to the Parliament two weeks ago.

The law says that Roskomnadzor will give some ISPs anti-threat equipment for free and that the chosen ISPs must actually deploy it. They can’t refuse that new-year gift.

And that anti-threat equipment will be acting as filter as well, as soon as ISPs deploying the anti-threat equipment don’t have to buy their own filters anymore.

And, there is a note that is scary to me. The note says that Roskomnadzor will be able to control Routing and DNS directly, as well as the filter.

It’s scary twice. First, we’ve already seen the level of technical competence and lack of safety checks and safeguard nets. I mean all the aforementioned mistakes.

Second, it’s huge centralisation of controlling power that can be abused without any transparency. I assume, that the luxury of signed XML blacklist will be gone.

Another option is that the blacklist will just become stale as most of “funny” filtering will happen within anti-threat equipment.

That law also introduces Registry of “legal” Internet Exchanges and list of cables crossing the border. To make absolutely sure that Russia is well-connected.

46 of 48

so it goes

47 of 48

Thanks to!

Philipp Kulin, ValdikSS, Mikhael Klimarev,�Dmitry Nazarov,�Alex Rudenko,�Dmitry Belyavskiy,�Wartan Hachaturow,�Dmitry Moskin,�Dmitry Morozovsky,

Simone Basso, Maria Xynou, Moritz Bartl,�zapret-info, SPb CTF, Roskomsvoboda, Digital Resistance Measurement Squadron, “the one who is to blame”, “Revisor” fans,�NAG, RIPE Atlas, …

48 of 48

Thanks RKN & Durov for fun!

Questions?

Leonid Evdokimov, 2018, CC-BY 4.0

usher2.club

darkk.net.ru/35c3