BlueCoat's Presence in Syria Finally Uncovered
As the reader probably knows, there has been a release of 54GB of BlueCoat logfiles a few weeks ago, finally leading to several mainstream medias mentioning the case, including BBC, the Washington Post and the Bureau of Investigative Journalism. However, I had feedback concerning this log release, notably about the fact that they had been redacted, i.e. that Telecomix removed a part of the information inside them.
However, I had feedback concerning this log release, notably about the fact that they had been redacted, i.e. that Telecomix removed a part of the information inside them. I am not entering into the debate on whether this was a "good" or a "bad" idea, so let us keep this purely factual : the original log files contain the IP addresses of the (innocent) Syrian Internet subscribers who were visiting Internet websites while being watched by the BlueCoat devices. These IP addresses were all replaced by the fake IP address "0.0.0.0" before the release.
This notably allowed BlueCoat to firstly deny the presence of their devices in Syria, by stating that "0.0.0.0" were obviously not Syrian IP addresses and/or that Telecomix could have "invented" these log files. Given that the US State Departement now seems to be looking into this case, this possibility of denying the facts was reported to me as a particularly annoying thing by some people who are taking care of the case. The issue was notably mentioned by Jacob Appelbaum at the "Power of Adhocracy" conference held in Sweden. EDIT : The Wall Street Journal just published a complete article, notably stating that BlueCoat does acknowledge the presence of its devices in Syria. It seems however that the acknowledgment underestimates the number of BlueCoat devices being present in Syria, as this present article may point out.
This article aims at clearing up things for those who do not have all the technical skills, including journalists, as it makes strictly no doubt, on our side, that (many) BlueCoat devices are present in Syria. I will try to refrain from using too many technical words, though all the proofs come from technical elements.
Firstly, one has to know that Telecomix has been aware of BlueCoat's devices presence for a long time. The information - without any evidence - was brought to us even before we found the devices ourselves. Telecomix's particular focus on Syria began around the beginning of July, and one of our first action was to "poke" lots of IP addresses that are publicly registered as belonging to Syria.
I will not technically detail what "poking" refers to. The reader should however know that there is a well-known software called "nmap" that can "poke" distant systems and guess what is the physical device and the operating system running on it by recognizing its network packet signatures.
This tool has many more features, and it is basically thanks to it and some hard work that we found both the BlueCoat devices and the server on which the log files were hosted.
Let us now have some explanations about the technical elements that we have.
Log Files Evidence
This is the most visible part of the "iceberg" of technical elements that we collected, as it is the thing that triggered the journalistic shitstorm that has been happening for a few days. I encourage the reader to have a look to one of the repository where the logs are stored, here, for instance. This repository is an exact copy of what was found on the Syrian server that used to host log files. Technical insight : this server was a FTP server on the IP address 220.127.116.11 (this is a Syrian IP), accessible without any authentication. It has been taken down since then.
Before diving into a log file, let us have a look at another file in the repository really easy to understand. It is automatically produced by the BlueCoat device that has automatically uploaded a log file to the repository as an indicator that the upload was successful. Its name is main_upload_result, and one can find one such file in each of the 15 "SG-XX" subdirectories on the repository. This firstly means that there were apparently 15 different BlueCoat devices uploading there log files on the repository. Let us now look at the file's content, for instance in the "SG-43" subdirectory :
********************START OF TEST FILE******************** NOTE: This is a verification file sent to test access log uploading. Please check the file for correctness and the event log for errors. For security purposes, please delete this file after perusal. ProxySG Appliance Date: 2011-07-17 ProxySG Appliance Time: 10:19:12 UTC ProxySG Appliance Name: SG-43 ProxySG Appliance IP: 18.104.22.168 ProxySG Appliance Type: Blue Coat SG9000 Series Sent to FTP server using the following configuration: Host: 22.214.171.124 Port: 21 Path: SG-43 User: administrator Password: ***** This file should contain approx. 800 bytes ********************END OF TEST FILE********************
Important points are highlighted. One can see that a good network administrator should have deleted the file because it obviously gives important information, notably the machine's model name (Blue Coat ProxySG Appliance SG-9000) and its IP address. The conclusion of all this is that fifteen BlueCoat ProxySG appliances SG-9000 uploaded log files to the FTP repository and that this repository has a Syrian IP address as well as the BlueCoat SG-9000 devices.
Now, let us look at what is inside some log file. These are text files compressed using the "gzip" format. Let us look at the very first lines of a log file, for instance "SG-42/SG_main__420802210000.log.gz". Here they are :
#Software: SGOS 126.96.36.199 #Version: 1.0 #Start-Date: 2011-08-02 11:15:58 #Date: 2011-08-02 11:15:58 #Fields: date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virus-id
The first highlight shows that the appliance that uploaded the log file is using the operating system " SGOS", version 188.8.131.52. Any Google search with this info will directly lead you to a lot of BlueCoat pages. The second highlighted line is here to tell an administrator how he should interpret the rest of the log file by naming the various information he is going to find further in the file. What about comparing this piece of data to a forum post on an official BlueCoat website?
Now, let us continue with the rest of that same log file. There is exactly one line for each request that the device intercepted. A "request" is in general the connection by a user to some Web resource (page, image, etc.). Look, for instance, at line number 33. Here is what it looks like (I split it in several lines for readability):
2011-08-02 11:16:19 386 0.0.0.0 - - - OBSERVED "unavailable" - 200 TCP_NC_MISS GET text/html http search.handycafe.com 80 /start ?sy - "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; AskTbPTV2/184.108.40.20690; handyCafeCln/3.3.21)" 220.127.116.11 5176 420 -
Let us only focus on highlighted points. The two first keywords describe how the BlueCoat proxy "reacted" to the request emitted by the user. These reactions are defined by an administrator thanks to filtering rules which are out of scope here. Then we have the host that was requested by the user (search.handycafe.com) as well as the protocol (http), the path (/start) and an additional string appended to the URL (?sy). In other words, there was a request emitted towards http://search.handycafe.com/start?sy (Notice the "?sy" giving a hint to link it with Syria; click on the link and you will see that the search is in Arabic). Next we have what is called the User Agent string, which identifies the browser used by the user. Here, "MSIE 7.0" indicates that Internet Explorer is used.We have finally, once again, the BlueCoat appliance IP address which monitored this request. An experienced user would see immediately that this request most probably comes from an Internet café computer, as "Handy Café" is a software dedicated to managing Internet cafés networks (which is strongly suspected to have spying abilities, by the way). Anyway, this shows that the logs uploaded by the BlueCoat SG-9000 appliances monitored millions of Web requests emitted by Internet subscribers. Of course, the fact that the source IP address is "0.0.0.0" gives a way to deny that the filtered user is actually inside Syria. But would a BlueCoat proxy using a Syrian IP address and intercepting millions of requests each day filter any other people than the ones in Syria? Also, would the URL http://search.handycafe.com/start?sy be requested by something else than an Internet café using HandyCafé and located inside Syria?
Anyway, let us make things clearer by examining one line coming from an unredacted log file. Its name is "SG_main__420726000000.log.gz" and is not publicly available (not to my knowledge, at least). The question on whether it should become public or not is not to be discussed here, but anyone can raise the point on the Telecomix IRC. So, here is the line:
2011-07-25 00:00:00 113 18.104.22.168 - - - OBSERVED "unavailable" http://ads.handycafe.com/adv.php?l=sy&rndID=34664 200 TCP_NC_MISS GET text/html;%20charset=utf-8 http adserving.cpxinteractive.com 80 /st ?ad_type=iframe&ad_size=160x600§ion=306277 - "Mozilla/4.0 (Windows NT 6.1); Gecko/20100101 Firefox/4.0.1; 10; Windows NT 6.1; SV1; handyCafeCln/3.3.21)" 22.214.171.124 1025 458 -
I've highlighted only one piece of information: the IPaddress from which came the HTTP request. All the 65535 IP addresses starting with "31.9" belong to the Syrian addressing space and are in practice assigned to DSL subscribers, either individuals or Internet cafés. Once again, this request most probably comes from an Internet café. I deliberately chose it for the example, as it hardly points to any particular individual and does not contain sensitive data in the URL. So this time, it clearly shows that, in a log file produced by a BlueCoat SG-9000 device, there is an undeniable trace of monitoring of Syrian Internet subscribers' Web activity.
But those log files are not the only correlating elements that we have. Let us examine additional ones.
Poking some Syrian Devices
As explained at the beginning, a tool named "nmap" can be used to poke or, more precisely scan IP addresses. Its principle is roughly to emit various requests towards the IP address and analyse the data received when the distant host replies. Out of this, nmap is sometimes able to give very valuable info on the nature of the device that is using the IP address.
On some Syrian IP addresses, which are not the same as the SG-9000 appliances mentioned above, nmap gave pretty clear results.
Let us begin by the IP address 126.96.36.199 , once again belonging to the Syrian space and more precisely to the Syrian Computer Society, that Bashar Al-Assad used to be the president. The result produced by nmap is available here. Let us highlight a few points:
- The output clearly shows that it recognized "Blue Coat proxy server" from some replies from the device;
- On a port it recognized the reply as coming from a service issued by a "BlueCoat SG-400 http proxy", whereas on another one it recognized it as issued by "Blue Coat SG-210 http proxy config";
- The "Service Info" line tells us that the operating system is "SGOS", just like the "SGOS" we observed in the SG-9000 log files.
It seems quite clear that we reach at least one BlueCoat device when we connect to this IP address. It is also indeed possible that we actually reach a network router that is in charge to dispatch requests to several BlueCoat devices, thus explaining why nmap hesitates between two distinct models. Anyway, this looks like another example of a BlueCoat device present in Syria.
There is another, even clearer, example, of another BlueCoat device detected by nmap. The reader is encouraged to get a look at the output given for the scan of the IP 188.8.131.52, also belonging the Syrian addressing space. The output has the same format as previously and the brand's presence is even clearer.
A much easier way to see it, for a non-experienced user, may be to visit the following website: https://184.108.40.206:8082. Your browser should display an SSL security warning, from which you should be able to see the details of the certificate that identifies this host, looking like this:
According to the information that I gathered, this kind of certificate is generated during the first bootup of the device. So, yet another - easy to see for anybody - element showing the presence of the brand in Syria.
As a bonus, I suggest to the reader to have a look at this online traffic monitoring system which clearly mentions a few BlueCoat devices inside the Syrian network.
BlueCoat Devices Give Their Serial Numbers
Detecting the devices actually does not necessarily require to do anything excepted "encouraging" the Syrians to visit your Web server, and then monitoring what their requests look like. More precisely, unless configured not to, a BlueCoat proxy that relays a user's request sends some additional data. As most Syrian users' Web requests are relayed by BlueCoat proxies (in order to monitor and possibly filter them), any administrator of a website that is visited by Syrians can easily catch this additional data.
This additional data (roughly) consists in telling the visited website that the initial request that was sent by the user actually passed through a BlueCoat proxy before reaching the website. The good thing is that this data includes a string that uniquely identifies the BlueCoat device. No two BlueCoat devices can have the same identification string. Let me be more precise: catching such strings can probably allow to trace back to what entity BlueCoat initially sold the corresponding devices.
Here is a quick summary and technical insight on how several of these identification strings were caught. Some monitoring was done on the Web server at http://telecomix.ceops.eu, and it was visited several times from Syria. For instance, a Syrian computer emitted a request towards this server, and the result of the monitoring - using a tool called "tcpdump" designed to monitor network streams - is the following:
17:40:11.358106 IP 220.127.116.11.60240 > 18.104.22.168.80: P 0:988(988) ack 1 win 1032 <nop,nop,timestamp 1854459201 1793220100> E.......,...R..0X....P.PIipM.6#!....>...... n..Aj.^.GET /guidelines_EN-AR.odt HTTP/1.1 Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */* Referer: http://www.google.com/custom?client=pub-3794288947762788&forid=1&channel=4183257091&ie=UTF-8&oe=UTF-8&safe=active&cof=GALT%3A%23008000%3BGL%3A1%3BDIV%3A%23336699%3BVLC%3A663399%3BAH%3Acenter%3BBGC%3AFFFFFF%3BLBGC%3A336699%3BALC%3A0000FF%3BLC%3A0000FF%3BT%3A000000%3BGFNT%3A0000FF%3BGIMP%3A0000FF%3BFORID%3A1&hl=ar&q=%D8%AA%D9%8A%D9%84%D9%8A%D9%83%D9%88%D9%85%D9%8A%D9%83%D8%B3 Accept-Language: ar-SY User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0) Accept-Encoding: gzip, deflate Host: telecomix.ceops.eu X-Forwarded-For: 31.9.XX.XX Cache-Control: max-stale=0 Connection: Keep-Alive X-BlueCoat-Via: AACDEBEFA2CBA614
The two first highlighted strings are of course IP addresses. 22.214.171.124 corresponds to one of the fifteen SG-9000 that were found in the log files, whereas the other IP (126.96.36.199) is simply associated to the host "telecomix.ceops.eu". The request shows that the user asked for the URL http://telecomix.ceops.eu/guidelines_EN-AR.odt. His browser prefers the Syrian Arabic language (as shown by "ar-SY"). The two last fields indicate respectively the IP address of the Syrian user who emitted the request (the last two digits were stripped off) and of course, a line that is peculiar to BlueCoat devices and that contains this famous identification string : AACDEBEFA2CBA614.
Using this exact same passive monitoring method, here are the identification strings that were gently handed to us by Syrian BlueCoat devices, as well as the (supposed) model, the associated IP who transmitted the request to us and the Syrian Internet Service Provider that is supposed to be managing the device:
ID: AACDEBEFA2CBA614, IP: 188.8.131.52, ISP: Tarassul, Model: ProxySG SG-9000 ID: 72CE221CEAE15612, IP: 184.108.40.206, ISP: Tarassul, Model: ProxySG SG-9000 ID: 2C044BEC00210EB6, IP: 220.127.116.11, ISP: Tarassul, Model: ProxySG SG-9000 ID: E4007B080BF520E6, IP: 18.104.22.168, ISP: Syrian Computer Society, Model: SG-400 ID: 6FA167DDB2C1D144, IP: 22.214.171.124 and 126.96.36.199, ISP: Syriatel Mobile Telecom, Model: ? ID: C96E97569D835384, IP: 188.8.131.52, ISP: RUNNET, Model: SG-400 or SG-810 ID: A9771E930F3ACE0C, IP: 184.108.40.206, ISP: Syriatel Mobile Telecom, Model: ?
One can find raw scan results here and a previous article that tried to describe the place of the BlueCoat devices within the Syrian Telecommunications Establishment network using the same kind of experiments.
So, here we have seven unique BlueCoat devices identification strings that were all emitted by Syrian IP addresses while filtering a request from within Syria, a quite accurate idea of the models that are used as well as four different Internet Service Provider names that use such devices.
BlueCoat Could Have Known
The BlueCoat company, apart from selling filtering devices, maintains a website named cwfservice.net. A cryptic string can be seen if one visits http://sp.cwfservice.net. The website is in fact an access to a database which can be remotely used by BlueCoat ProxySG devices to determine whether some website should be blocked or not. Collin David Anderson notably suggested that Syrian BlueCoat devices were effectively retrieving data from this database regularly, according to the redacted log files. He concludes that BlueCoat were undoubtedly aware of the BlueCoat devices presence in Syria. The lacking piece of data implicated by the source IP being "0.0.0.0" can of course make this suggestion believable, because the redacted IP address could replace a BlueCoat device IP that performed the request (although I am not sure that a BlueCoat device would log its own requests).
However, having a look to a piece of unredacted log shows that these kind of requests more probably from "classical" Syrian Internet subscribers' IP. The explanation is simple: there exists anti-virus software that uses this same database. To say it briefly, some Syrian people use anti-virus software that silently access the online BlueCoat database. As the access is made through a webpage, these requests appear in the BlueCoat log files.
We saw in the previous section that anyone monitoring the HTTP traffic incoming from Syria would quickly and easily notice some BlueCoat identification strings as a result of the Syrian monitoring. As a conclusion, even though it is not sure that it is the BlueCoat devices that are "calling home", the fact that users' requests to cwfservice.net pass through these devices implicates that BlueCoat could have seen some identification strings coming from Syria , in the case they have monitored - even for short periods - incoming HTTP traffic.
Article from the Wall Street Journal states that BlueCoat acknowledged that contact was made from their devices located in Syria towards the company's servers, but no deeper details are given, thus allowing both assumption to be potentially true.
Telecomix is, obviously, in contact with Syrian people, who are more or less aware of how the monitoring and censorship system works in the country. While these people's identities will not be revealed, several independant Syrian sources came to confirm firmly the brand's presence in the country. As a "funny" bonus, well-aware people provided us with the exact geographical location of the ProxySG SG-9000 devices. They are apparently located inside the building at GPS coordinates 33.519593,36.306148.
BlueCoat is not alone
BlueCoat devices were relatively easy to spot. Although we lack firm evidence, there are high suspiscions that other brands' devices participate in monitoring the whole country. The following brands may be concerned to various extent:
- Fortinet, a German brand that manufactures DPI-capable devices named FortiGate
- Nokia-Siemens and/or Trovicor, who may be monitoring cellphone communications
- The Chinese brands ZTE and Huawei.
I - personally - encourage people who would like to point out the role of other major brands in attacks against freedom of speech to try to find evidence of the role of such brands in monitoring and blocking connections in Syria.
We have shown that:
- Fifteen ProxySG 9000 are used and controlled by Tarassul ISP to monitor and filter a major part of Syrian Internet subscribers
- A bunch of other BlueCoat devices (probably SG-400 or SG-810) are used by other Syrian ISPs such as the Syrian Computer Society and Syriatel Mobile Telecom (these would thus be dedicated to censoring 3G connections)
- Several identificating strings from these BlueCoat devices were obtained, coming from various Syrian ISPs
- It looks like the BlueCoat's acknowledgment reported by the Wall Street Journal does not cover the complete set of BlueCoat devices used to monitor and censor Syrian Internet.
It is unclear how the devices were shipped into Syria. They undoubtedly passed through third-party resellers, but it is indeed possible that they did not take all the same path towards Syria.
All the people who have grabbed log files, participated to the debate on whether they should released or not, redacted or not, etc. Thanks to everyone. All the people on Telecomix IRC, #opsyria, and in particular Punkbob, okhin, TheDoctor, n0pants and lejonet. As well as those who worked hard and silently.
Thanks to the ones we do not see on the IRC channel but who indeed gave time and energy in this. I'm thinking of Jacob Appelbaum, Collin David Anderson and all the people they may have worked with. Apologies for the potential mistakes, though all of us were trying to do the good thing.
Many thanks to Telecomix IRC staff, notably to chrisk and jaywalk, who underwent criticism for actions they did not do.
Greetings to the people of Syria.