Wednesday, January 18, 2012

Displaying Gluster traffic in Wireshark

As part of my job, I am doing some tests with the Red Hat Storage Software Appliance. The current version of RHSSA is based on Gluster 3.2.5. After some experiments, it seem that Gluster is pretty cool and surprisingly easy to setup.

In order to see what is going happening on the network, I captured some tcpdumps. Reading the files in Wireshark, does not show any Gluster specifics. It seems that Wireshark does not know how to decode (or rather dissect) the Gluster traffic. Very unfortunate, as quite some future troubleshooting and performance analysis may require investigating the network packets.

Luckily the Wireshark Developer's guide contains a chapter on Adding a basic dissector. After writing some code and tests, I now have some Wireshark packages that recognize some Gluster communication. The RPMs are available for testing, feedback over email is appreciated.

With the updated packages, the output of tshark (the terminal version of Wireshark) identifies some Gluster packets:
$ tshark -r gluster-communication.cap 'tcp.len > 0' | head
  7   0.002572 -> Gluster Dump V1 DUMP Call
  8   0.002633 -> Gluster Dump V1 DUMP Call
 11   0.002909 -> Gluster Dump V1 DUMP Reply (Call In 7)
 12   0.002918 -> Gluster Dump V1 DUMP Reply (Call In 8)
 15   0.003104 -> Gluster Portmap V1 PORTBYBRICK Call
 16   0.003158 -> Gluster Portmap V1 PORTBYBRICK Call
 17   0.003298 -> Gluster Portmap V1 PORTBYBRICK Reply (Call In 15)
 18   0.003310 -> Gluster Portmap V1 PORTBYBRICK Reply (Call In 16)
 31   3.013909 -> Gluster Dump V1 DUMP Call
 32   3.013965 -> PCEP Unknown Message (0). 

As with several other protocols, Wireshark detects some packets as non-gluster ones. In this tcpdump, there surely is no PCEP traffic (last line in the above output). Each dissector for a protocol should do some sanity checks to see if a packet belongs to its protocol. These checks are not easy to do, and hence quite some protocols detect packets from Gluster as their communication stream.

Luckily it is possible to disable a protocol in the ~/.wireshark/disabled_protos file. Finding the correct names of a protocol isn't always straight forward. Use Wireshark to graphically create the file is the easiest, it also takes care of disabling the protocols that are possibly encapsulated. In Wireshark
  1. go to Analyze in the menu
  2. click "Enabled Protools"
  3. uncheck PCEP (and while you are at it, also uncheck SSL as it gives the same issues)

After these steps, tshark should recognize all traffic to and from port 24007 as belonging to one of the Gluster protocols. I have only tested the Wireshark dissectors on Gluster 3.2.5, later releases use some newer versions of some protocols and these may not be detected yet.
Post a Comment