Thursday, July 26, 2012

Wireshark 1.8.x included in the upcoming Fedora 18

It seems that the maintainer of the wireshark package in Fedora has updated to version 1.8.1 in the current Fedora Rawhide, which will become Fedora 18. The schedule tells us that Fedora 18 is planned to be released on 2012-11-06 (the latest schedule may have an updated date).

This is good news for the Gluster Community (and support engineers like me), as Wireshark 1.8 contains support for most of the Gluster protocols:
Comments on the support in Wireshark are much appreciated, send them to gluster-devel@nongnu.org, file an issue in my development repository or in the Wireshark Bugzilla.

Sunday, May 27, 2012

Gluster support for Wireshark is maturing!

A lot of changes were committed recently to the gluster-wireshark repository. A lot of effort was put into the details (click on the image to enlarge):

  • UUIDs and GFIDs are now displayed as 4-2-2-2-6 bytes
  • flags for OPEN, CREATE etc are now shown in detail
  • mode/umask permissions are now shown in detail
  • dictionaries are displayed more user friendly
Most of the work was done so that the dissector files get in shape for (requesting) inclusion in upstream.

The full log is available, and so are updated RPMs for Fedora-16, Fedora-17 and EPEL-6. If installing a patched Wireshark isn't an option, you can build a wireshark-plugin-gluster easily with the steps in the provided README. On the project wiki, there are some pre-captured tcpdumps for consumption. Only a hand full of minor issues are know at this time, more reviewing and reporting is definitely welcome!

If you notice that some packets/frames are not displayed as Gluster, and you think they should, check gluster-wireshark wiki where is explained how to prevent PCEP and other protocols from claiming packers/frames.

Thursday, May 17, 2012

Updated Wireshark packages for RHEL-6 and Fedora-17 available for testing

[From an email to the gluster-devel mailinglist] 
 
today I have merged support for GlusterFS 3.2 and 3.3 into one Wireshark 
'dissector'. The packages with date 20120516 in the version support both 
the current stable 3.2.x version, and the latest 3.3.0qa41. Older 3.3.0 
versions will likely have issues due to some changes in the RPC-AUTH 
protocol used. Updating to the latest qa41 release (or newer) is 
recommended anyway. I do not expect that we'll add support for earlier 
3.3.0 releases.

My repository with packages for RHEL-6 and Fedora-17 contains a .repo 
file for yum (save it in /etc/yum.repos.d):
- http://repos.fedorapeople.org/repos/devos/wireshark-gluster/

RPMs for other Fedora or RHEL versions can be provided on request. Let 
me know if you need an other version (or architecture).

Single patches for some different Wireshark versions are available from 
https://github.com/nixpanic/gluster-wireshark.

A full history of commits can be found here:
- https://github.com/nixpanic/gluster-wireshark-1.4/commits/master/
   (Support for GlusterFS 3.3 was added by Akhila and Shree, thanks!)

Please test and report success and problems, file a issues on github: 
https://github.com/nixpanic/gluster-wireshark-1.4/issues
Some functionality is still missing, but with the current status, it 
should be good for most analysing already. With more issues filed, it 
makes it easier to track what items are important.

Of course, you can also respond to this email and give feedback :-)

After some more cleanup of the code, this dissector will be passed on 
for review and inclusion in the upstream Wireshark project. Some more 
testing results is therefore much appreciated.

Monday, March 5, 2012

Correcting broken startup of rsyslogd in a systemd unit file

My Fedora 17 Beefy Miracle alpha1 ARM system does not any contents in /var/log/messages. This is very impractical for troubleshooting. The command systemd-journalctl --no-tail shows that rsyslog.service fails to start correctly. Bummer!

Starting the daemon by hand, does not give any issues, so at least rsyslogd does not have an issue itself.

Checking the configuration, there are two unit files that may be used to start rsyslogd:
  1. /etc/systemd/system/syslog.service
  2. /etc/systemd/system/multi-user.target.wants/rsyslog.service
Both of these are symlinks to the unit file (/usr/lib/systemd/system/rsyslog.service) that comes with the rsyslogd package:
[Unit]
Description=System Logging Service

[Service]
EnvironmentFile=-/etc/sysconfig/rsyslog
ExecStartPre=/bin/systemctl stop systemd-kmsg-syslogd.service
ExecStart=/sbin/rsyslogd -n $SYSLOGD_OPTIONS
Sockets=syslog.socket
StandardOutput=null

[Install]
WantedBy=multi-user.target
All looks pretty neat, there is no very obvious issue here. However, on checking what this systemd-kmsg-syslogd.service is and does, I seem to be unable to find the unit file that defines this service. Just guessing, but if the ExecStartPre fails, it may not even try to start the rsyslog-daemon. Just commenting out the ExecStartPre is Not Done with the systemd configuration files. The advised way to change unit files which live under /usr/lib/systemd should be copied to the /etc/systemd/* directories instead:
# cd /etc/systemd/system
# rm syslog.service
# ln -s /usr/lib/systemd/system/rsyslog.service syslog.service
# cd multi-user.target.wants
# rm rsyslog.service
# ln -s ../syslog.service rsyslog.service
Now is is possible to change the unit file, without changing a file that may be replaced by a update of the rsyslog package. Commenting out the ExecStartPre in the new copy of syslog.service and rebooting, makes syslog work for me and /var/log/messages contains all the logs as expected.

More brokenness in my systemd configuration

One more annoying thing is that systemctl is not very usable:
# systemctl status rsyslog.service
Failed to get D-Bus connection: No connection to service manager.
Although D-Bus is running:
# ps v -C dbus-daemon
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
 1474 ?        Ss     0:00      6   265  2770  1436  0.3 /bin/dbus-daemon --system --address=systemd: --nofork --systemd-activation

I'm not sure what causes that (SElinux is disabled), but it is currently no big issue for me.

Saturday, March 3, 2012

WLAN configuration of NetworkManager from the console

It seems that configuring a WLAN connection in NetworkManager on Fedora (tested on F17 alpha on ARM) is pretty straight forward. Unfortunately I was not able to find documentation about the steps, so I just note them down here:

I must confess, that I cheated a little, as the configuration was initially created with nm-connection-editor by selecting the 'Available to all users' option. Copying the configuration files to the Fedora 17 rootfs image made my wireless work immediately.

The configuration consists out of two files:

  1. /etc/sysconfig/network-scripts/ifcfg-MyWLAN where MyWLAN is just a name for your connection
    ESSID="MyESSID"
    MODE=Managed
    KEY_MGMT=WPA-PSK
    TYPE=Wireless
    BOOTPROTO=dhcp
    DEFROUTE=yes
    IPV4_FAILURE_FATAL=yes
    IPV6INIT=no
    NAME=MyWLAN
    UUID=57fc7596-af3e-48af-8d90-1a06783083d7
    ONBOOT=yes
    PEERDNS=yes
    PEERROUTES=yes
    
    Of course it is needed to change at least the NAME and the ESSID. KEY_MGMT can probably have other values, but there is no need for me to research that atm.
  2. /etc/sysconfig/network-scripts/keys-MyWLAN
    WPA_PSK="My Top Secret Password"
    
    This password will likely not work anywhere, just change it to whatever is needed.

After putting these two files in place (copied from an other device after booting) made NetworkManager connect to the WLAN immediately.

Some tools for checking the connection are:
  • nm-tool which displays the detected networks and some more details.
  • nmcli con up MyWLAN in case you want to force connecting to the WLAN configured under /etc/sysconfig/network-scripts/ifcfg-MyWLAN.

Saturday, February 4, 2012

Improvements on displaying Gluster traffic with Wireshark

Slow but steadily I am improving the packet dissector for Gluster. Once it is in a more complete state, it'll be submitted for inclusion in Wireshark. Until that time (which will likely take some more months), updated versions of Wireshark for RHEL6 and F16 will occur on the Fedora People Repository at http://repos.fedorapeople.org/repos/devos/wireshark-gluster/.

The current packages can not only detect most of the Gluster traffic as explained in a previous post, but also display some of the more complex XDR structures used.

A screen shot of one example with Wireshark on Fedora 16:

Some of the captions in the display contain FIXME or similar notes. These are mainly reminders for myself, but also should be helpful for people trying the packages and suggesting improvements. Just by reading the Gluster sources, it is not always clear what the specific fields in the communication should be called in a user friendly way.

There is an open invite to improve the decoding done by the dissector. The code is (hopefully) easy to understand and should provide a low entry level. There are quite some notes in the code where it needs improvement (search for FIXME in the main .c file).

Coding is one thing, documenting the protocol is something that is closely related to the Wireshark dissector. How would reading traffic help, when you can not check if what you see is what should be happening? A start with documenting this can be found in my github repository which contains the latest sources.

Wednesday, January 18, 2012

Displaying Gluster traffic in Wireshark

As part of my job, I am doing some tests with the Red Hat Storage Software Appliance. The current version of RHSSA is based on Gluster 3.2.5. After some experiments, it seem that Gluster is pretty cool and surprisingly easy to setup.

In order to see what is going happening on the network, I captured some tcpdumps. Reading the files in Wireshark, does not show any Gluster specifics. It seems that Wireshark does not know how to decode (or rather dissect) the Gluster traffic. Very unfortunate, as quite some future troubleshooting and performance analysis may require investigating the network packets.

Luckily the Wireshark Developer's guide contains a chapter on Adding a basic dissector. After writing some code and tests, I now have some Wireshark packages that recognize some Gluster communication. The RPMs are available for testing, feedback over email is appreciated.

With the updated packages, the output of tshark (the terminal version of Wireshark) identifies some Gluster packets:
$ tshark -r gluster-communication.cap 'tcp.len > 0' | head
  7   0.002572 192.168.122.184 -> 192.168.122.137 Gluster Dump V1 DUMP Call
  8   0.002633 192.168.122.184 -> 192.168.122.137 Gluster Dump V1 DUMP Call
 11   0.002909 192.168.122.137 -> 192.168.122.184 Gluster Dump V1 DUMP Reply (Call In 7)
 12   0.002918 192.168.122.137 -> 192.168.122.184 Gluster Dump V1 DUMP Reply (Call In 8)
 15   0.003104 192.168.122.184 -> 192.168.122.137 Gluster Portmap V1 PORTBYBRICK Call
 16   0.003158 192.168.122.184 -> 192.168.122.137 Gluster Portmap V1 PORTBYBRICK Call
 17   0.003298 192.168.122.137 -> 192.168.122.184 Gluster Portmap V1 PORTBYBRICK Reply (Call In 15)
 18   0.003310 192.168.122.137 -> 192.168.122.184 Gluster Portmap V1 PORTBYBRICK Reply (Call In 16)
 31   3.013909 192.168.122.184 -> 192.168.122.137 Gluster Dump V1 DUMP Call
 32   3.013965 192.168.122.184 -> 192.168.122.137 PCEP Unknown Message (0). 

As with several other protocols, Wireshark detects some packets as non-gluster ones. In this tcpdump, there surely is no PCEP traffic (last line in the above output). Each dissector for a protocol should do some sanity checks to see if a packet belongs to its protocol. These checks are not easy to do, and hence quite some protocols detect packets from Gluster as their communication stream.

Luckily it is possible to disable a protocol in the ~/.wireshark/disabled_protos file. Finding the correct names of a protocol isn't always straight forward. Use Wireshark to graphically create the file is the easiest, it also takes care of disabling the protocols that are possibly encapsulated. In Wireshark
  1. go to Analyze in the menu
  2. click "Enabled Protools"
  3. uncheck PCEP (and while you are at it, also uncheck SSL as it gives the same issues)

After these steps, tshark should recognize all traffic to and from port 24007 as belonging to one of the Gluster protocols. I have only tested the Wireshark dissectors on Gluster 3.2.5, later releases use some newer versions of some protocols and these may not be detected yet.

Sunday, January 8, 2012

Unexpected requirements for creating a video DVD with Brasero

With the current standard of mobiles and photo cameras that can be used to film things, it would be fun to create a little video. It seems that Brasero is a standard tool for the GNOME desktop, which should work without issues in my preferred XFCE as well. Brasero (and seemingly all of its dependencies) is even included in the Red Hat Enterprise Linux 6.2 Workstation repository/channel.

Unfortunately, after creating a video project in Brasero, a little warning gets displayed:
It is not possible to write with the current set of plugins.

There is the need for a more detailed look into how Brasero does the detection/loading of plugins. In the search for more verbosity from the command-line, I found that the command
brasero --help shows that there is a --help-brasero-burn option. Giving that a try, dumps an immense list of things on the console. This includes the following:

  • ffmpeg_mpeg2video is missing
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:452: loading /usr/lib64/brasero/plugins/libbrasero-vob.so
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin.c:1041: Module encountered an error while registering its capabilities:
    "ffenc_mpeg2video" element could not be created
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:481: Load failure, no GType was returned "ffenc_mpeg2video" element could not be created
  • libdvdcss is missing
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:452: loading /usr/lib64/brasero/plugins/libbrasero-dvdcss.so
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin.c:1041: Module encountered an error while registering its capabilities:
    Encrypted DVD: please install libdvdcss version 1.2.x
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:481: Load failure, no GType was returned Encrypted DVD: please install libdvdcss version 1.2.x
    
  • dvdauthor is missing
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:452: loading /usr/lib64/brasero/plugins/libbrasero-dvdauthor.so
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin.c:1041: Module encountered an error while registering its capabilities:
    "dvdauthor" could not be found in the path
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:481: Load failure, no GType was returned "dvdauthor" could not be found in the path
  • vcdimager is missing
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:452: loading /usr/lib64/brasero/plugins/libbrasero-vcdimager.so
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin.c:1041: Module encountered an error while registering its capabilities:
    "vcdimager" could not be found in the path
    (brasero:19298): BraseroBurn-DEBUG: At burn-plugin-manager.c:481: Load failure, no GType was returned "vcdimager" could not be found in the path

After some googling, it seems that these packages are made available for RHEL6 in Repoforge (previously RPMForge). The additional packages that need to get installed (first pass):
  • gstreamer-ffmpeg
  • libdvdcss
  • dvdauthor
  • vcdimager

Getting closer, but not completely there yet, as brasero --brasero-burn-debug still misses something:
(brasero:19508): BraseroBurn-DEBUG: At burn-plugin-manager.c:452: loading /usr/lib64/brasero/plugins/libbrasero-vob.so
(brasero:19508): BraseroBurn-DEBUG: At burn-plugin.c:1041: Module encountered an error while registering its capabilities:
"mplex" element could not be created
(brasero:19508): BraseroBurn-DEBUG: At burn-plugin-manager.c:481: Load failure, no GType was returned "mplex" element could not be created

Knowing that Brasero uses the GStreamer framework, it seems logical to try to install a GStream plugin called mplex. All GStreamer plugins seem to be located in /usr/lib64/gstreamer-0.10, and installing the mplex plugin with yum seems to work:
# yum install /usr/lib64/gstreamer-0.10/libgstmplex.so
Loaded plugins: refresh-packagekit, rhnplugin
Setting up Install Process
...
Resolving Dependencies
--> Running transaction check
---> Package gstreamer-plugins-bad.x86_64 0:0.10.19-3.el6.rf will be installed
--> Processing Dependency: libxvidcore.so.4()(64bit) for package: gstreamer-plugins-bad-0.10.19-3.el6.rf.x86_64
--> Processing Dependency: libmms.so.0()(64bit) for package: gstreamer-plugins-bad-0.10.19-3.el6.rf.x86_64
--> Processing Dependency: libmusicbrainz.so.4()(64bit) for package: gstreamer-plugins-bad-0.10.19-3.el6.rf.x86_64
--> Processing Dependency: libamrwb.so.3()(64bit) for package: gstreamer-plugins-bad-0.10.19-3.el6.rf.x86_64
--> Processing Dependency: libcdaudio.so.1()(64bit) for package: gstreamer-plugins-bad-0.10.19-3.el6.rf.x86_64
--> Running transaction check
---> Package amrwb.x86_64 0:7.0.0.3-1.el6.rf will be installed
---> Package libcdaudio.x86_64 0:0.99.12p2-13.el6 will be installed
---> Package libmms.x86_64 0:0.5-1.el6.rf will be installed
---> Package libmusicbrainz.x86_64 0:2.1.5-11.1.el6 will be installed
---> Package xvidcore.x86_64 0:1.2.2-1.el6.rf will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================================
 Package                                     Arch                         Version                                 Repository                                       Size
========================================================================================================================================================================
Installing:
 gstreamer-plugins-bad                       x86_64                       0.10.19-3.el6.rf                        rpmforge                                        1.2 M
Installing for dependencies:
 amrwb                                       x86_64                       7.0.0.3-1.el6.rf                        rpmforge                                        180 k
 libcdaudio                                  x86_64                       0.99.12p2-13.el6                        epel                                             39 k
 libmms                                      x86_64                       0.5-1.el6.rf                            rpmforge                                         60 k
 libmusicbrainz                              x86_64                       2.1.5-11.1.el6                          rhel-x86_64-workstation-6                        88 k
 xvidcore                                    x86_64                       1.2.2-1.el6.rf                          rpmforge                                        555 k

Transaction Summary
========================================================================================================================================================================
Install       6 Package(s)

Total download size: 2.1 M
Installed size: 7.4 M
Is this ok [y/N]: y
Downloading Packages:
...
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : libcdaudio-0.99.12p2-13.el6.x86_64                                                                                                                   1/6 
  Installing : libmms-0.5-1.el6.rf.x86_64                                                                                                                           2/6 
  Installing : amrwb-7.0.0.3-1.el6.rf.x86_64                                                                                                                        3/6 
  Installing : libmusicbrainz-2.1.5-11.1.el6.x86_64                                                                                                                 4/6 
  Installing : xvidcore-1.2.2-1.el6.rf.x86_64                                                                                                                       5/6 
  Installing : gstreamer-plugins-bad-0.10.19-3.el6.rf.x86_64                                                                                                        6/6 

Installed:
  gstreamer-plugins-bad.x86_64 0:0.10.19-3.el6.rf                                                                                                                       

Dependency Installed:
  amrwb.x86_64 0:7.0.0.3-1.el6.rf         libcdaudio.x86_64 0:0.99.12p2-13.el6        libmms.x86_64 0:0.5-1.el6.rf        libmusicbrainz.x86_64 0:2.1.5-11.1.el6       
  xvidcore.x86_64 0:1.2.2-1.el6.rf       

Complete!

This whole troubleshooting took quite a while, and I do not know if all the new packages are really needed, or if a subset is sufficient. Glad it's over, working and documented for a next time too (and possibly others).