Skip to content

GitLab

Trac
Trac

GitLab is used only for code review, issue tracking and project management. Canonical locations for source code are still https://gitweb.torproject.org/ https://git.torproject.org/ and git-rw.torproject.org.

Last edited by Alexander Færøy
Page history

Kazakhstan

PageOutline

= 2016–2017 (#20348) = #a20348

Updated 2017-06-15.

== Summary == #a20348-summary

Since 2016, we have been studying various kinds of censorship happening in Kazakhstan. The investigation has been taking place in ticket #20348. This is a summary of what is known so far.

  • #a20348-graphs
  • Since then, user numbers have remained more or less smooth.
  • #a20348-httpwww.change.org, tumblr.com, bash.im, and youporn.com.
    • #a20348-http-fp-source
    • After late December 2016, requests for blocked websites began simply timing out, without a distinctive signature.
  • #a20348-obfs4
    • The obfs4 blocks are unusual—they don't simply drop the SYN or SYN/ACK packet, which is the obvious way and what happens in China. Instead, the firewall allows a TCP connection to be established, and only after a few packets have been transmitted does it start dropping packets.
    • The unusual style of blocking, combined with the fact that blocking occasionally failed, initially made us suspect that the detection of obfs4 was perhaps based on a timing signature, rather than blacklisting of known bridges. However it does not appear to be quite that sophisticated, as at least some secret obfs4 bridges are not blocked.
  • #a20348-meek

[=#a20348-graphs]

userstats-relay-country-kz-2016-01-01-2017-06-14-off.png link

userstats-bridge-country-kz-2016-01-01-2017-06-14.png link

userstats-bridge-combined-kz-2016-01-01-2017-06-14.png link

== Code and data == #a20348-data

Some code and measurements, primarily consisting of hourly obfs4 connection attempts from a VPN in Kazakhstan: https://www.bamsoftware.com/proxy-probe/kz-data/.

OONI reports containing various types of measurements are available from https://explorer.ooni.torproject.org/country/KZ and https://measurements.ooni.torproject.org/api/v1/files?probe_cc=KZ\\ https://measurements.ooni.torproject.org/api/v1/files?probe_cc=KZ&page=1\\ etc.

== HTTP blocking and fingerprint == #a20348-http

Between November and late December 2016, HTTP sites were blocked with a distinctive injected response that redirected to !http://92.63.88.128/?NTDzLZcomment:159:ticket:20348

HTTP/1.1 302 Found
Content-Length: 210
Location: http://92.63.88.128/?NTDzLZ
Content-Type: text/html; charset=UTF-8

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Found</TITLE></HEAD><BODY>
<H1>302 Found</H1>
The document has moved
<A HREF="http://92.63.88.128/?NTDzLZ">here</A>
</BODY></HTML>

(Notice that the Content-Lengthcomment:202:ticket:20348210 but it should be 224.) Following the redirect leads to another server that, in turn, serves a different kind of redirect (meta-refresh and JavaScript) to the impossible URL !http://90.263.11.193comment:149:ticket:20348

HTTP/1.1 200 OK
Server: nginx
Date: Fri, 16 Dec 2016 17:01:22 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Expires: Thu, 21 Jul 1977 07:30:00 GMT
Last-Modified: Fri, 16 Dec 2016 17:01:22 GMT
Cache-Control: max-age=0
Pragma: no-cache
Set-Cookie: cfb9f=%7B%22streams%22%3A%5B1481907682%5D%2C%22campaigns%22%3A%7B%221%22%3A1481907682%7D%2C%22time%22%3A1481907682%7D; expires=Mon, 16-Jan-2017 17:01:22 GMT; Max-Age=2678400; path=/

<html>
            <head>
                <meta http-equiv="REFRESH" content="1; URL='http://90.263.11.193'">
                <script type="text/javascript">window.location = "http://90.263.11.193";</script>
            </head>
            </html>

comment:149:ticket:20348http://mwtv.lv/comment:148:ticket:20348

That the responses are injected is detectable from inconsistencies in the IP TTL and TCP options. Notice, in this [youporn.com.pcap:ticket:20348|packet capture of requesting http://youporn.com](None/youporn.com.pcap:ticket:20348|packet capture of requesting http://youporn.com), that the TTL of the SYN/ACK is 50 while the TTL of the HTTP response is 58. Notice also that there are TCP options in the SYN/ACK ([mss 1304,sackOK,TS val 845116384 ecr 17593903,nop,wscale 7]) but none in the HTTP response; the TS option, at least, should oblige the server to include timestamps in all its subsequent segments.

<pre style="font-size:80%">
<span style="background:cornsilk">10:40:31.768987 IP (tos 0x0, ttl 64, id 8730, offset 0, flags [DF], proto TCP (6), length 60)
    10.11.0.150.52824 &gt; 31.192.120.44.http: Flags [S], cksum 0x1df2 (correct), seq 2069320757, win 29200, <strong>options [mss 1460,sackOK,TS val 17593903 ecr 0,nop,wscale 7]</strong>, length 0</span>
<span style="background:lavender">10:40:32.162036 IP (tos 0x20, <strong>ttl 50</strong>, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    31.192.120.44.http &gt; 10.11.0.150.52824: Flags [S.], cksum 0x4cf4 (correct), seq 3620557931, ack 2069320758, win 28960, options [mss 1304,sackOK,TS val 845116384 ecr 17593903,nop,wscale 7], length 0</span>
<span style="background:cornsilk">10:40:32.162067 IP (tos 0x0, ttl 64, id 8731, offset 0, flags [DF], proto TCP (6), length 52)
    10.11.0.150.52824 &gt; 31.192.120.44.http: Flags [.], cksum 0xeafc (correct), ack 1, win 229, options [nop,nop,TS val 17594002 ecr 845116384], length 0
10:40:32.162223 IP (tos 0x0, ttl 64, id 8732, offset 0, flags [DF], proto TCP (6), length 161)
    10.11.0.150.52824 &gt; 31.192.120.44.http: Flags [P.], cksum 0x9075 (correct), seq 1:110, ack 1, win 229, options [nop,nop,TS val 17594002 ecr 845116384], length 109: HTTP, length: 109
	GET / HTTP/1.1
	User-Agent: Wget/1.16 (linux-gnu)
	Accept: */*
	Host: youporn.com
	Connection: Keep-Alive
	</span>
<span style="background:lavender">10:40:32.457302 IP (tos 0x20, <strong>ttl 58</strong>, id 0, offset 0, flags [DF], proto TCP (6), length 386)
    31.192.120.44.http &gt; 10.11.0.150.52824: Flags [FP.], cksum 0x55d6 (correct), seq 1:347, ack 110, win 229, length 346: HTTP, length: 346
	HTTP/1.1 302 Found
	Content-Length: 210
	Location: http://92.63.88.128/?NTDzLZ
	Content-Type: text/html; charset=UTF-8
	
	&lt;HTML&gt;&lt;HEAD&gt;&lt;meta http-equiv="content-type" content="text/html;charset=utf-8"&gt;
	&lt;TITLE&gt;302 Found&lt;/TITLE&gt;&lt;/HEAD&gt;&lt;BODY&gt;
	&lt;H1&gt;302 Found&lt;/H1&gt;
	The document has moved
	&lt;A HREF="http://92.63.88.128/?NTDzLZ"&gt;here&lt;/A&gt;
	&lt;/BODY&gt;&lt;/HTML&gt;
	</span>
<span style="background:cornsilk">10:40:32.493859 IP (tos 0x0, ttl 64, id 8733, offset 0, flags [DF], proto TCP (6), length 52)
    10.11.0.150.52824 &gt; 31.192.120.44.http: Flags [.], cksum 0xe8d9 (correct), ack 348, win 237, options [nop,nop,TS val 17594085 ecr 845116384], length 0
10:40:34.829753 IP (tos 0x0, ttl 64, id 8734, offset 0, flags [DF], proto TCP (6), length 52)
    10.11.0.150.52824 &gt; 31.192.120.44.http: Flags [R.], cksum 0xe68e (correct), seq 110, ack 348, win 237, options [nop,nop,TS val 17594668 ecr 845116384], length 0</span>
</pre>

comment:173:ticket:20348

echo -n $'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n' | nc government.kz 80
echo -n $'GET / HTTP/1.0\r\nHost: bash.im\r\n\r\n' | nc government.kz 80

The firewall is stateful in that you wouldn't get an injection if you sent a naked payload without establishing a TCP connection first. I.e., in scapy, this didn't get an injection:

sr(IP(dst="government.kz")/TCP(flags="PA", seq=123456, ack=1000)/"GET / HTTP/1.0\r\nHost: bash.im\r\n\r\n")

but this did:

r = sr(IP(dst="government.kz")/TCP(flags="S", seq=1000))[0][0][1]
sr(IP(dst="government.kz")/TCP(flags="PA", seq=123456, ack=r.seq+1)/"GET / HTTP/1.0\r\nHost: bash.im\r\n\r\n")

=== Partial list of blocked sites === #a20348-blocked-sites

Using OONI data, requests for the following 50 sites were found to be blocked with the ?NTDzLZ#a20348-http

From other sources we also know of blocks of:

=== Finding the source of the HTTP fingerprints === #a20348-http-fp-source

The injected HTML, with its The document has moved\n<A HREF="...">here</A>comment:203:ticket:20348Project Sonarcomment:161:ticket:20348

We found one of the matches still live under 2090000.ru, a Russian ISP, and tested it. We found responses being injected for two purposes: for customer payment enforcement and for censorship. Requesting an ordinary domain would redirect to http://0.2090000.ru, a page that says (in Russian): The balance of your personal account is negative. To continue working on the Internet, please top up the balance. Requesting a site that is blocked in Russia, on the other hand, serves a iframe from http://zapret.2090000.ru/ ("zapret" = запрет = "prohibition, interdiction, ban"), which says: We apologize, but access to the requested resource is limited. Check the availability of the domain name and / or index of the site page, the network address in the Unified Register can be found at http://blocklist.rkn.gov.ru/comment:173:ticket:20348

Example of accessing an ordinary site:

echo -n $'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n' | nc 37.192.17.117 80
HTTP/1.1 302 Found
Content-Length: 202
Location: http://0.2090000.ru
Content-Type: text/html; charset=UTF-8

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Found</TITLE></HEAD><BODY>
<H1>302 Found</H1>
The document has moved
<A HREF="http://0.2090000.ru">here</A>
</BODY></HTML>

Example of accessing a site blocked in Russia (http://ej.ru):

echo -n $'GET / HTTP/1.0\r\nHost: ej.ru\r\n\r\n' | nc 37.192.17.117 80
HTTP/1.1 200 OK
Connection: close
Content-Type: text/html; charset=iso-8859-1

<HTML>
<HEAD><TITLE>Access Denied</TITLE></HEAD>
<BODY>
<div align="left">
<iframe src="http://zapret.2090000.ru" width=100%" height="1250" frameborder="0"> </iframe>
<p><p>
</div>
</BODY>
</HTML>

Our conclusion is: The similarity of the injections and the structure of the HTML suggests that the same or similar hardware is used in Kazakhstan and in this Russian ISP (and also probably the other cases we found, e.g. telcom.co.id, afrihost.com).

comment:169:ticket:20348

== HTTPS blocking == #a20348-https

HTTPS blocking appears to be blocked by SNI. A TCP connection succeeds but the connection stalls after the TLS handshake. Here, the first request is blocked but the second one (without SNI) succeeds:

wget https://www.tumblr.com/
wget https://87.248.116.13/ --header 'Host: www.tumblr.com' --no-check-certificate

#a20348-http

echo -n $'GET / HTTP/1.0\r\nHost: gohost.kz\r\n\r\n' | openssl s_client -ign_eof -connect gohost.kz:443 -servername example.com
echo -n $'GET / HTTP/1.0\r\nHost: gohost.kz\r\n\r\n' | openssl s_client -ign_eof -connect gohost.kz:443 -servername www.tumblr.com

== obfs4 blocking == #a20348-obfs4

#a20348-graphs

comment:195:ticket:20348

Here is a summary of early observations (not all necessarily correct) that took place in ticket #20348 before we began systematic measurements. This isn't meant to be a summary of the actual state of affairs; rather a summary of the scattered discussion and the false leads we followed.

comment:193:ticket:20348

ticket:20348:kz-data-20170605.png

The next table shows the average maximum bootstrap percentage reached during the times when both the US and KZ sites were taking measurements (i.e., excluding the gaps mentioned above). What we see from this is that most measured bridges are not in fact blocked. The only bridges where the KZ rate is lower than the US rate are ndnop3, ndnop5, GreenBelt, Lisbeth, and NX01. Bridges that were added more recently, or were never used, are not more blocked in KZ.

<table class="wiki">
<tr><th>bridge</th><th>US average bootstrap %</th><th>KZ average bootstrap %</th></tr>
<tr><td>ndnop3</td><td align=right style="background: #EBCAC9">63.33%</td><td align=right style="background: #DD918A">20.68%</td></tr>
<tr><td>ndnop5</td><td align=right style="background: #ECCDCC">65.75%</td><td align=right style="background: #DD9189">20.53%</td></tr>
<tr><td>GreenBelt:80</td><td align=right style="background: #EAC7C6">61.21%</td><td align=right style="background: #D9837A">10.21%</td></tr>
<tr><td>GreenBelt:443</td><td align=right style="background: #EAC7C6">61.21%</td><td align=right style="background: #DD918A">21.21%</td></tr>
<tr><td>GreenBelt:5881</td><td align=right style="background: #EAC7C6">61.24%</td><td align=right style="background: #DD918A">20.86%</td></tr>
<tr><td>Lisbeth</td><td align=right style="background: #F7FAFE">99.32%</td><td align=right style="background: #E4AEAA">42.84%</td></tr>
<tr><td>NX01:443</td><td align=right style="background: #F7FAFE">99.58%</td><td align=right style="background: #E4AFAB">43.50%</td></tr>
<tr><td>cymrubridge30:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFD">98.93%</td></tr>
<tr><td>cymrubridge31:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFD">98.94%</td></tr>
<tr><td>cymrubridge32:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFE">99.48%</td></tr>
<tr><td>cymrubridge33:80</td><td align=right style="background: #F6F9FD">98.38%</td><td align=right style="background: #F7FAFD">98.97%</td></tr>
<tr><td>frosty:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFE">99.04%</td></tr>
<tr><td>frosty:40035</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F6F9FD">98.42%</td></tr>
<tr><td>camille:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFE">99.26%</td></tr>
<tr><td>camille:42779</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7F9FD">98.68%</td></tr>
<tr><td>dragon:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFE">99.26%</td></tr>
<tr><td>dragon:38224</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7F9FD">98.68%</td></tr>
<tr><td>dimple:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFD">98.97%</td></tr>
<tr><td>dimple:44769</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FBFE">99.63%</td></tr>
<tr><td>unused-cypherpunkskludge</td><td align=right style="background: #F7FAFE">99.61%</td><td align=right style="background: #F6F8FC">97.70%</td></tr>
<tr><td>unused-iat0</td><td align=right style="background: #F7FAFE">99.62%</td><td align=right style="background: #F6F5F8">95.60%</td></tr>
<tr><td>unused-iat1</td><td align=right style="background: #F7FAFE">99.61%</td><td align=right style="background: #F6F9FD">98.31%</td></tr>
<tr><td>unused-iat2</td><td align=right style="background: #E6B4B1">47.37%</td><td align=right style="background: #E6B5B2">47.92%</td></tr>
</table>

== meek blocking == #a20348-meek

/doc/meekTLS fingerprint and SNI, similarly to what has been documented in the past for Cyberoam and FortiGuard/doc/meek#Howtochangethefrontdomain

== Identifying hardware vendors == #a20348-vendors

Initially we suspected that firewall hardware was supplied by Sophos Cyberoam (this accounts for the many references to Cyberoam in the comments at #20348). The only reason for this suspicion was that Cyberoam was one of the firewalls that had been reported to block obfs4 connections (the other was iBoss):

Later, we found signs that a vendor may instead be Allot Communications. Observations supporting this are:

  1. #a20348-httpforum postcomment:175:ticket:20348
  2. Allot specifically advertises support (in their "DART"comment:184:ticket:20348allot-804579667973963776.jpg:ticket:20348|screenshot, more discussion). Allot's firewall hardware also does URL blocking (archive). https://www.allot.com/products/platforms/supported-protocols/#1460974307058-a61550f0-8196 ([https://archive.is/AuA8b archive])

    ==== June 13, 2016 ==== In Allot’s latest DART Protocol Pack, we refined our signature for the Tor obfs4 safe transport, to assure accruate identification of this kind of traffic on your network:

    • Tor Obfs4 ==== April 4th, 2016 ==== In Allot’s latest DART Protocol Pack we added signatures that identify these TOR transport protocols that use the Onion Router network:
    • TOR ScrambleSuit (pluggable proxy transport protocol)
    • TOR Obfs4 (TCP obfuscation layer) ==== April 27th, 2015 ==== This week we focused on updating and refining existing DART signatures for these popular VPN and encryption protocols:
    • TOR (default mode, 3 available bridge modes, CDN meek)
    • Psiphon ==== January 26th, 2015 ==== Allot’s latest DART Protocol Pack helps you identify traffic from users of the Psiphon circumvention system, which has becoming a popular way to bypass content-filtering systems in order to access sites that have been blocked due to geographical or regulatory restrictions.
    • Psiphon Proxy Server
    • Psiphon CDN (Meek mode)
  3. Customs import applications on the site of the National Center of Accreditation dated 2014-11-07 have "АО 'Казахтелеком'" (JSC Kazakhtelecom) importing equipment from "Allot Communications LTD" in Israel, 7 SG-Tera 14 devices and 1 SG-Sigma E6 device.

More information about Allot can be found in comment:184:ticket:20348 and its following comments.

2015

In March 2015 blocking is observable again, the stock tor browser bundle cannot connect.

dcf: Do you know when in March 2015 it started happening? It doesn't seem to have affected the user graph yet: https://metrics.torproject.org/userstats-relay-country.png?start=2014-10-01&events=off&end=2015-05-31&country=kz

In April 2015 blocking is observable again, some user reporting about problems while another still using it. User graph confirms report. https://metrics.torproject.org/userstats-relay-country.png?start=2015-03-01&end=2015-04-21&country=kz&events=on

Graze on, graze on, submissive nation! \ You will not wake to honor's call.\ Why offer herds their liberation?\ For them are shears or slaughter-stall,\ Their heritage each generation\ The yoke with jingles, and the gall.

= 2012–2013 (#6140) = #Kazakhstan6140

First witnessed

Tor blocking started between February and March 2012 and is mentioned in a blog post. Another blog post was published two weeks afterwards.

Last witnessed

Sometimes around the summer of 2013 the blocking has been disabled (and an old tor bundle started to work that didn't work before).

Types of Tor censorship

  • Deep packet inspection: #6140
  • Fingerprint: The TLS client cipher-list in the ClientHello record, parts of the Tor TLS server hello record, and probably more fingerprints in other nearby TLS records.

Types of non-Tor censorship

Ways to bypass censorship

Type of firewall

  • Unknown.

Reproducing the blocking