Wednesday, January 18, 2012

Displaying Gluster traffic in Wireshark

As part of my job, I am doing some tests with the Red Hat Storage Software Appliance. The current version of RHSSA is based on Gluster 3.2.5. After some experiments, it seem that Gluster is pretty cool and surprisingly easy to setup.

In order to see what is going happening on the network, I captured some tcpdumps. Reading the files in Wireshark, does not show any Gluster specifics. It seems that Wireshark does not know how to decode (or rather dissect) the Gluster traffic. Very unfortunate, as quite some future troubleshooting and performance analysis may require investigating the network packets.

Luckily the Wireshark Developer's guide contains a chapter on Adding a basic dissector. After writing some code and tests, I now have some Wireshark packages that recognize some Gluster communication. The RPMs are available for testing, feedback over email is appreciated.

With the updated packages, the output of tshark (the terminal version of Wireshark) identifies some Gluster packets:
$ tshark -r gluster-communication.cap 'tcp.len > 0' | head
  7   0.002572 192.168.122.184 -> 192.168.122.137 Gluster Dump V1 DUMP Call
  8   0.002633 192.168.122.184 -> 192.168.122.137 Gluster Dump V1 DUMP Call
 11   0.002909 192.168.122.137 -> 192.168.122.184 Gluster Dump V1 DUMP Reply (Call In 7)
 12   0.002918 192.168.122.137 -> 192.168.122.184 Gluster Dump V1 DUMP Reply (Call In 8)
 15   0.003104 192.168.122.184 -> 192.168.122.137 Gluster Portmap V1 PORTBYBRICK Call
 16   0.003158 192.168.122.184 -> 192.168.122.137 Gluster Portmap V1 PORTBYBRICK Call
 17   0.003298 192.168.122.137 -> 192.168.122.184 Gluster Portmap V1 PORTBYBRICK Reply (Call In 15)
 18   0.003310 192.168.122.137 -> 192.168.122.184 Gluster Portmap V1 PORTBYBRICK Reply (Call In 16)
 31   3.013909 192.168.122.184 -> 192.168.122.137 Gluster Dump V1 DUMP Call
 32   3.013965 192.168.122.184 -> 192.168.122.137 PCEP Unknown Message (0). 

As with several other protocols, Wireshark detects some packets as non-gluster ones. In this tcpdump, there surely is no PCEP traffic (last line in the above output). Each dissector for a protocol should do some sanity checks to see if a packet belongs to its protocol. These checks are not easy to do, and hence quite some protocols detect packets from Gluster as their communication stream.

Luckily it is possible to disable a protocol in the ~/.wireshark/disabled_protos file. Finding the correct names of a protocol isn't always straight forward. Use Wireshark to graphically create the file is the easiest, it also takes care of disabling the protocols that are possibly encapsulated. In Wireshark
  1. go to Analyze in the menu
  2. click "Enabled Protools"
  3. uncheck PCEP (and while you are at it, also uncheck SSL as it gives the same issues)

After these steps, tshark should recognize all traffic to and from port 24007 as belonging to one of the Gluster protocols. I have only tested the Wireshark dissectors on Gluster 3.2.5, later releases use some newer versions of some protocols and these may not be detected yet.

Sunday, January 8, 2012

Unexpected requirements for creating a video DVD with Brasero

With the current standard of mobiles and photo cameras that can be used to film things, it would be fun to create a little video. It seems that Brasero is a standard tool for the GNOME desktop, which should work without issues in my preferred XFCE as well. Brasero (and seemingly all of its dependencies) is even included in the Red Hat Enterprise Linux 6.2 Workstation repository/channel.

Unfortunately, after creating a video project in Brasero, a little warning gets displayed:
It is not possible to write with the current set of plugins.

There is the need for a more detailed look into how Brasero does the detection/loading of plugins. In the search for more verbosity from the command-line, I found that the command
brasero --help shows that there is a --help-brasero-burn option. Giving that a try, dumps an immense list of things on the console. This includes the following:

  • ffmpeg_mpeg2video is missing
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:452: loading /usr/lib64/brasero/plugins/libbrasero-vob.so
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin.c:1041: Module encountered an error while registering its capabilities:
    "ffenc_mpeg2video" element could not be created
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:481: Load failure, no GType was returned "ffenc_mpeg2video" element could not be created
  • libdvdcss is missing
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:452: loading /usr/lib64/brasero/plugins/libbrasero-dvdcss.so
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin.c:1041: Module encountered an error while registering its capabilities:
    Encrypted DVD: please install libdvdcss version 1.2.x
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:481: Load failure, no GType was returned Encrypted DVD: please install libdvdcss version 1.2.x
    
  • dvdauthor is missing
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:452: loading /usr/lib64/brasero/plugins/libbrasero-dvdauthor.so
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin.c:1041: Module encountered an error while registering its capabilities:
    "dvdauthor" could not be found in the path
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:481: Load failure, no GType was returned "dvdauthor" could not be found in the path
  • vcdimager is missing
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:452: loading /usr/lib64/brasero/plugins/libbrasero-vcdimager.so
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin.c:1041: Module encountered an error while registering its capabilities:
    "vcdimager" could not be found in the path
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:481: Load failure, no GType was returned "vcdimager" could not be found in the path

After some googling, it seems that these packages are made available for RHEL6 in Repoforge (previously RPMForge). The additional packages that need to get installed (first pass):
  • gstreamer-ffmpeg
  • libdvdcss
  • dvdauthor
  • vcdimager

Getting closer, but not completely there yet, as brasero --brasero-burn-debug still misses something:
(brasero:19508): BraseroBurn-DEBUG: At burn-plugin-manager.c:452: loading /usr/lib64/brasero/plugins/libbrasero-vob.so
(brasero:19508): BraseroBurn-DEBUG: At burn-plugin.c:1041: Module encountered an error while registering its capabilities:
"mplex" element could not be created
(brasero:19508): BraseroBurn-DEBUG: At burn-plugin-manager.c:481: Load failure, no GType was returned "mplex" element could not be created

Knowing that Brasero uses the GStreamer framework, it seems logical to try to install a GStream plugin called mplex. All GStreamer plugins seem to be located in /usr/lib64/gstreamer-0.10, and installing the mplex plugin with yum seems to work:
# yum install /usr/lib64/gstreamer-0.10/libgstmplex.so
Loaded plugins: refresh-packagekit, rhnplugin
Setting up Install Process
...
Resolving Dependencies
--> Running transaction check
---> Package gstreamer-plugins-bad.x86_64 0:0.10.19-3.el6.rf will be installed
--> Processing Dependency: libxvidcore.so.4()(64bit) for package: gstreamer-plugins-bad-0.10.19-3.el6.rf.x86_64
--> Processing Dependency: libmms.so.0()(64bit) for package: gstreamer-plugins-bad-0.10.19-3.el6.rf.x86_64
--> Processing Dependency: libmusicbrainz.so.4()(64bit) for package: gstreamer-plugins-bad-0.10.19-3.el6.rf.x86_64
--> Processing Dependency: libamrwb.so.3()(64bit) for package: gstreamer-plugins-bad-0.10.19-3.el6.rf.x86_64
--> Processing Dependency: libcdaudio.so.1()(64bit) for package: gstreamer-plugins-bad-0.10.19-3.el6.rf.x86_64
--> Running transaction check
---> Package amrwb.x86_64 0:7.0.0.3-1.el6.rf will be installed
---> Package libcdaudio.x86_64 0:0.99.12p2-13.el6 will be installed
---> Package libmms.x86_64 0:0.5-1.el6.rf will be installed
---> Package libmusicbrainz.x86_64 0:2.1.5-11.1.el6 will be installed
---> Package xvidcore.x86_64 0:1.2.2-1.el6.rf will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================================
 Package                                     Arch                         Version                                 Repository                                       Size
========================================================================================================================================================================
Installing:
 gstreamer-plugins-bad                       x86_64                       0.10.19-3.el6.rf                        rpmforge                                        1.2 M
Installing for dependencies:
 amrwb                                       x86_64                       7.0.0.3-1.el6.rf                        rpmforge                                        180 k
 libcdaudio                                  x86_64                       0.99.12p2-13.el6                        epel                                             39 k
 libmms                                      x86_64                       0.5-1.el6.rf                            rpmforge                                         60 k
 libmusicbrainz                              x86_64                       2.1.5-11.1.el6                          rhel-x86_64-workstation-6                        88 k
 xvidcore                                    x86_64                       1.2.2-1.el6.rf                          rpmforge                                        555 k

Transaction Summary
========================================================================================================================================================================
Install       6 Package(s)

Total download size: 2.1 M
Installed size: 7.4 M
Is this ok [y/N]: y
Downloading Packages:
...
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : libcdaudio-0.99.12p2-13.el6.x86_64                                                                                                                   1/6 
  Installing : libmms-0.5-1.el6.rf.x86_64                                                                                                                           2/6 
  Installing : amrwb-7.0.0.3-1.el6.rf.x86_64                                                                                                                        3/6 
  Installing : libmusicbrainz-2.1.5-11.1.el6.x86_64                                                                                                                 4/6 
  Installing : xvidcore-1.2.2-1.el6.rf.x86_64                                                                                                                       5/6 
  Installing : gstreamer-plugins-bad-0.10.19-3.el6.rf.x86_64                                                                                                        6/6 

Installed:
  gstreamer-plugins-bad.x86_64 0:0.10.19-3.el6.rf                                                                                                                       

Dependency Installed:
  amrwb.x86_64 0:7.0.0.3-1.el6.rf         libcdaudio.x86_64 0:0.99.12p2-13.el6        libmms.x86_64 0:0.5-1.el6.rf        libmusicbrainz.x86_64 0:2.1.5-11.1.el6       
  xvidcore.x86_64 0:1.2.2-1.el6.rf       

Complete!

This whole troubleshooting took quite a while, and I do not know if all the new packages are really needed, or if a subset is sufficient. Glad it's over, working and documented for a next time too (and possibly others).