Playing with SPDY Wireshark and Nginx

Introduction

Currently I am playing and developing around SPDY: a protocol developed by Google with the aim of reducing web page load time.  And for this task I need tools to help me understanding how the procotol works.
I will show you how I set up a SPDY web server, and how I use Wireshark to analyze the exchanges between my browser and the server.

I had to install my own web server because, for decoding TLS traffic, Wireshark needs the private key of the server. And of course this private key is always kept secret, so none of the web server on the internet provide it, they only provide there public key. (See PKI for more details)

I have chosen Nginx which seems to be perfect for this job, but we have to tweak the spdy support.

Nginx

Download compile and install nginx:

$ wget http://nginx.org/download/nginx-1.7.0.tar.gz
$ tar xvf wireshark-1.7.0.tar.gz
$ ./configure --with-http_ssl_module --with-http_spdy_module --prefix=~/local/nginx-1.7.0
$ make && make install 

From here, we edit the conf/nginx.conf file to tell nginx to start the web server with https and spdy support on port 443. We also force the SSL cipher to AES128-SHA, because Wireshark can only decrypt TLS traffic that only use a subset of the ciphers suite. It does not support DHE and Ephemeral RSA (see here).

$ cat conf/nginx.conf
[...]
server {
  listen      443 ssl spdy;
  server_name localhost;

  ssl_certificate     cert.pem;
  ssl_certificate_key server.key;

  ssl_ciphers               AES128-SHA;
  ssl_prefer_server_ciphers on;

  location /home/jpichon/www {
    index index.html index.htm;
  }
} 

Generate a self signed certificate and a private key in one command:

$ openssl req -x509 -newkey rsa:2048 -keyout ~/local/nginx-1.7.0/conf/server.key -out ~/local/nginx-1.7.0/conf/cert.pem -days 365 

Convert the private key in a format that Wireshark understand:

$ openssl pkcs8 -topk8 -in ~/local/nginx-1.7.0/conf/server.key -out temp.pem$ openssl rsa -in temp.pem -out ~/local/nginx-1.7.0/conf/rsa.pem 

Start the server:

$ cd ~/local/nginx-1.7.0
$ ./sbin/nginx 

 Wireshark and spdyshark plugin

Download and extract wireshark sources:

$ wget http://www.wireshark.org/download/src/wireshark-1.7.1.tar.bz2
$ tar jxvf wireshark-1.7.1.tar.bz2

Download spdyshark, the spdy dissector:

$ git clone https://code.google.com/p/spdyshark/
$ cp -r spdyshark/spdyshark wireshark-1.7.1/plugins/

Patch wireshark

$ cd wireshark-1.7.1
$ patch -p1 < ../spdyshark/spdyshark_build.patch
$ ./autogen.sh

Configure and build it:

$ ./configure --with-ssl && make

SPDY traffic Analysis

Start a capture of the traffic:

$ sudo tcpdump -i any -s 0 -w ~/spdy.pcap

Open the following url in a browser that support SPDY protocol: https://127.0.0.1. Close the page and go back to your terminal to stop the tcpdump capture.

Then you can use wireshark to analyze the spdy traffic between your browser and your SPDY server. To do so, start your freshly compiled wireshark:

$ cd wireshark-1.7.1
$ ./wireshark ~/spdy.pcap

Head to Edit > Preferences > Protocols > SSL. Click on the Edit button of the RSA keys list field, and add two entries in the array:
* ip: 127.0.0.1 port: 443 protocol: http key file: ~/local/nginx-1.7.0/conf/rsa.pem
* ip: 127.0.0.1 port: 443 protocol: spdy key file: ~/local/nginx-1.7.0/conf/rsa.pem

Like this:

Finally you should see the result: the TLS application data decrypted, and the SPDY (which is the decrypted application data) decoded:

Can't trace userspace processes with SystemTap

What is System Tap ?

It is an infrastructuer coming with a scripting language that allows collecting information on a running linux kernel. It can be used by System Administrators to see for example which process are hitting hardly the disks, or by Kernel/UserSpace developers to instrument their code and gather information about disk access,  network usage... A great tool to fine tune and optimize application.
It is similar to DTrace available for Solaris and BDS Oses.

A good documentation exists for System Tap: http://sourceware.org/systemtap/tutorial/

User Space debugging

For Debian Squeeze System Tap works out of the box for probing kernel code. But when I wanted to trace userspace code I was facing the following error:

# cat test.stp

probe process("./test").function("main").return
{
    printf ("Exit from the main\n");
}

#stap -v test.stp -c ./test

Pass 1: parsed user script and 68 library script(s) using 49964virt/20928res/1768shr kb, in  10usr/20sys/424real ms.
semantic error: process probes not available without kernel CONFIG_UTRACE while resolving probe point process("./test").function("main").return

So it seems like my Debian kernel is lacking of utrace supports. Let's recompile it :). First of all we have to get a vanilla kernel (because with the debian I can't apply utrace patches properly):

#wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.tar.bz2

Then the utrace patches:

#wget http://web.elastic.org/~fche/frob-utrace/2.6.32/tracehook.patch

#wget http://web.elastic.org/~fche/frob-utrace/2.6.32/utrace.patch

#wget http://web.elastic.org/~fche/frob-utrace/2.6.32/utrace-ptrace.patch

The trick here is that tracehook.patch is a gzip archive and not a plain text file:

#mv tracehook.patch tracehook.gz

#gunzip tracehook.gz

#mv tracehook tracehook.patch

Untar linux sources and apply the patches:

#tar -xvf linux-2.6.32.tar.bz2

#cd linux-2.6.32

#patch -p1 < ../tracehook.patch

#patch -p1 < ../utrace.patch

#patch -p1 < ../utrace-ptrace.patch

Compile the kernel:

#cp /boot/config-`uname -r` .config

#make oldconfig

Enable the option CONFIG_UTRACE

#make menuconfig (->General Setup -> Infrastructure for debugging and tracing user processes ) 


#sudo make

#sudo make modules_install install

#mkinitramfs -o /boot/initrd-img.2.6.32 2.6.32

Also modify grub to add an entry pointing to your new kernel and initrd image.

Reboot and that's it !