Sunday, December 7, 2014

Configured Zabbix to keep my server cool

Recently I got myself an APC NetShelterCX mini. It is a 12U rack, with integrated fans for cooling. At the moment it is populated with some ARM boards (not rack mounted), their PDUs, a switch and (for now) one 2U server.

Surprisingly, the fans of the NetShelter are louder than the server (the rest does not have fans at all, except for the switch). But, it is not an option to keep the fans turned off all the time. When the server is idle, its CPUs temperature stays somewhere between 40-50 Celsius. However, when starting several virtual machines for testing some Gluster changes, the temperature rises steadily. To prevent overheating, the fans of the cabinet need to be turned on.

Of course, turning on the fans manually is possible, but it requires me to plug in the power cable. This is not very convenient when the cabinet is normally closed to reduce the noise. With the PDUs and fence_netio from the fence-agents-netio package, the fans inside the cabinet can be controlled remotely. That was a great step already!

Well, things can be even better. I don't want to monitor the temperature of my server and then decide when to turn on the fans. After spending some time looking for and comparing different monitoring solutions, I settled to try Zabbix. Packages for Zabbix are available for Fedora and EPEL, which makes trying it out pretty simple.

In a couple of hours playing with the installation and configuration, I was able to monitor the basics of my server. With a little manual configuration, the Zabbix Agent on the server can send the temperatures of the CPUs. All I had to do was setup a UserParameter in /etc/zabbix_agentd.conf:

UserParameter=cpu.temp.0,sensors | sed -n -r '/Physical id 0/s/^.*:[[:space:]]+\+([[:digit:]]+\.[[:digit:]]+).*$/\1/p'
UserParameter=cpu.temp.1,sensors | sed -n -r '/Physical id 1/s/^.*:[[:space:]]+\+([[:digit:]]+\.[[:digit:]]+).*$/\1/p'

The above configuration snippet tells the Zabbix Agent on the server to execute sensors (from the lm_sensors package), and filter the output through a sed command. The result is captured and sent to the Zabbix Server.

With the new cpu.temp.0 and .1 keys, the Zabbix webui can use these temperature items to setup a trigger which then invokes an action when the temperature rises above 55 Celsius. When the trigger enters the PROBLEM state, the action calls fence_netio and turns on the port that has the cable for the fans connected. When the trigger returns back to normal (checked every 5 minutes, now moved to 10), the port is disabled again.

This is the first time that I actually have setup monitoring with some custom actions. It was quite fun, and I'm certainly happy with the result.

Monday, November 17, 2014

Testing GlusterFS with very fast disks on Fedora 20

In the past I used to test with RAM-disks, provided by /dev/ram*. Gluster uses extended attributes on the filesystem, that makes is not possible to use tmpfs. While thinking about improving some of the GlusterFS regression tests, I noticed that Fedora 20 (and possibly earlier versions too) does not provide the /dev/ram* devices anymore. I could not find the needed kernel module quickly, so I decided to look into the newer zram module.

Getting zram working seems to be pretty simple. By default one /dev/zram0 is made available after loading the module. But, if needed, the module offers a parameter num_devices to create more devices. After loading the module with modprobe zram, you can do the following to create your high-performance volatile storage:

# SIZE_2GB=$(expr 1024 \* 1024 \* 1024 \* 2)
# echo ${SIZE_2GB} > /sys/class/block/zram0/disksize
# mkfs -t xfs /dev/zram0
# mkdir /bricks/fast
# mount /dev/zram0 /bricks/fast

With this mountpoint it is now possible to create a Gluster volume:

# gluster volume create fast ${HOSTNAME}:/bricks/fast/data
# gluster volume start fast

Once done with testing, stop and delete the Gluster volume, and free the zram like this:

# umount /bricks/fast
# echo 1 > /sys/class/block/zram0/reset

Of course, unloading the module with rmmod zram would free the resources too.

It is getting more important for Gluster to be prepared for very fast disks. Hardware like Fusion-io Flash drives and in future Persistent Memory/NVM will get more available in storage clouds, and of course we would like to see Gluster staying part of that!

Wednesday, November 5, 2014

Installing GlusterFS 3.4.x, 3.5.x or 3.6.0 on RHEL or CentOS 6.6

With the release of RHEL-6.6 and CentOS-6.6, there are now glusterfs packages in the standard channels/repositories. Unfortunately, these are only the client-side packages (like glusterfs-fuse and glusterfs-api). Users that want to run a Gluster Server on a current RHEL or CentOS now have difficulties installing any of todays current version of the Gluster Community packages.

The most prominent issue is that the glusterfs package from RHEL has a version of 3.6.0.28, and that is higher than the last week released version of 3.6.0. RHEL is shipping a pre-release that was created while the Gluster Community was still developing 3.6. An unfortunate packaging decision added a .28 to the version, where most other pre-releases would fall-back to a (rpm-)version like 3.6.0-0.1.something.bla.el6. The difference might look minor, but the result is a major disruption in the much anticipated 3.6 community release.

For the immediate need to fix this in a most easy way for our community users, we have decided to release version 3.6.1 later this week (maybe on Thursday November 6). This version is higher than the version in RHEL/CentOS, and therefore yum will prefer the package from the community repository over the one available in RHEL/CentOS. This is also the main reason why there have been no 3.6.0 packages provided on the download server.

Installing an older stable release (like 3.4 or 3.5) on RHEL/CentOS 6.6 requires a different approach. At the moment we can offer two solutions that can be used. We are still working on making this easier, until that is finalized, some manual actions are required.

Lets assume you want to verify if todays announced glusterfs-3.5.3beta2 packages indeed fix that bug you reported. (These steps apply to the other versions as well, this just happens to be what I have been testing.)

Option A: use exclude in the yum repository files for RHEL/CentOS

  1. download the glusterfs-353beta2-epel.repo file and save it under /etc/yum.repos.d/

  2. edit /etc/yum.repos.d/redhat.repo or /etc/yum.repos.d/CentOS-Base.repo and under each repository that you find, add the following line

    exclude=glusterfs*

This prevents yum from installing the glusterfs* packages from the standard RHEL/CentOS repositories, but allows those packages from others. The Red Hat Customer Portal has an article about this configuration too.

Option B: install and configure yum-plugin-priorities

Using yum-plugin-priorities is probably a more stable solution. This does not require changes to the standard RHEL/CentOS repositories. However, an additional package needs to get installed.

  1. enable the optional repository when on RHEL, CentOS users can skip this step

    # subscription-manager repos --list | grep optional-rpms
        # subscription-manager repos --enable=*optional-rpms

  2. install the yum-plugin-priorities package:

    # yum install yum-plugin-priorities

  3. download the glusterfs-353beta2-epel.repo file and save it under /etc/yum.repos.d/

  4. edit the /etc/yum.repos.d/glusterfs-353beta2-epel.repo file and add the following option to each repository definition:

    priority=50

The default priority for repositories is 99. The repositories with the lowest number have the highest priority. As long as the RHEL/CentOS repositories do not have the priority option set, the packages from the glusterfs-353beta2-epel.repo will get preferred by yum.

When using the yum-plugin-priorities approach, we highly recommend that you check if all your repositories have a suitable (or missing) priority option. In case some repositories have the option set, but yum-plugin-priorities was not installed yet, the order of the repositories might have changed. Because of this, we do not want to force using yum-plugin-priorities on all the Gluster Community users that run on RHEL/CentOS.

In case users still have issues installing the Gluster Community packages on RHEL or CentOS, we recommend getting in touch with us on the Gluster Users mailinglist (archive) or in the #gluster IRC channel on Freenode.

GlusterFS 3.5.3beta2 is now available for testing

Even though GlusterFS 3.6.0 has been released last week, the 3.5 stable series continues to live on! The 2nd beta for GlusterFS 3.5.3 is now available for testing. Many bugs have been fixed since the 3.5.2 release, check the references below for details.

Bug reporters are encouraged to verify the fixes, and we invite others to test this beta to check if there are no regressions. Please see the announcement email for further information. When the release proves to be stable, a final 3.5.3 will be made available in a week or so.

Packages for different distributions can be found on the main download server.

Release Notes for GlusterFS 3.5.3

This is a bugfix release. The Release Notes for 3.5.0, 3.5.1 and 3.5.2 contain a listing of all the new features that were added and bugs fixed in the GlusterFS 3.5 stable release.

Bugs Fixed:

  • 1081016: glusterd needs xfsprogs and e2fsprogs packages
  • 1100204: brick failure detection does not work for ext4 filesystems
  • 1126801: glusterfs logrotate config file pollutes global config
  • 1129527: DHT :- data loss - file is missing on renaming same file from multiple client at same time
  • 1129541: [DHT:REBALANCE]: Rebalance failures are seen with error message " remote operation failed: File exists"
  • 1132391: NFS interoperability problem: stripe-xlator removes EOF at end of READDIR
  • 1133949: Minor typo in afr logging
  • 1136221: The memories are exhausted quickly when handle the message which has multi fragments in a single record
  • 1136835: crash on fsync
  • 1138922: DHT + rebalance : rebalance process crashed + data loss + few Directories are present on sub-volumes but not visible on mount point + lookup is not healing directories
  • 1139103: DHT + Snapshot :- If snapshot is taken when Directory is created only on hashed sub-vol; On restoring that snapshot Directory is not listed on mount point and lookup on parent is not healing
  • 1139170: DHT :- rm -rf is not removing stale link file and because of that unable to create file having same name as stale link file
  • 1139245: vdsm invoked oom-killer during rebalance and Killed process 4305, UID 0, (glusterfs nfs process)
  • 1140338: rebalance is not resulting in the hash layout changes being available to nfs client
  • 1140348: Renaming file while rebalance is in progress causes data loss
  • 1140549: DHT: Rebalance process crash after add-brick and `rebalance start' operation
  • 1140556: Core: client crash while doing rename operations on the mount
  • 1141558: AFR : "gluster volume heal <volume_name> info" prints some random characters
  • 1141733: data loss when rebalance + renames are in progress and bricks from replica pairs goes down and comes back
  • 1142052: Very high memory usage during rebalance
  • 1142614: files with open fd's getting into split-brain when bricks goes offline and comes back online
  • 1144315: core: all brick processes crash when quota is enabled
  • 1145000: Spec %post server does not wait for the old glusterd to exit
  • 1147156: AFR client segmentation fault in afr_priv_destroy
  • 1147243: nfs: volume set help says the rmtab file is in "/var/lib/glusterd/rmtab"
  • 1149857: Option transport.socket.bind-address ignored
  • 1153626: Sizeof bug for allocation of memory in afr_lookup
  • 1153629: AFR : excessive logging of "Non blocking entrylks failed" in glfsheal log file.
  • 1153900: Enabling Quota on existing data won't create pgfid xattrs
  • 1153904: self heal info logs are filled with messages reporting ENOENT while self-heal is going on
  • 1155073: Excessive logging in the self-heal daemon after a replace-brick
  • 1157661: GlusterFS allows insecure SSL modes

Known Issues:

  • The following configuration changes are necessary for 'qemu' and 'samba vfs plugin' integration with libgfapi to work seamlessly:
    1. gluster volume set <volname> server.allow-insecure on
    2. restarting the volume is necessary
       gluster volume stop <volname>
       gluster volume start <volname>
      
    3. Edit /etc/glusterfs/glusterd.vol to contain this line:
       option rpc-auth-allow-insecure on
      
    4. restarting glusterd is necessary
       service glusterd restart
      
      More details are also documented in the Gluster Wiki on the Libgfapi with qemu libvirt page.
  • For Block Device translator based volumes open-behind translator at the client side needs to be disabled.
    gluster volume set <volname> performance.open-behind disabled
    
  • libgfapi clients calling glfs_fini before a successfull glfs_init will cause the client to hang as reported here. The workaround is NOT to call glfs_fini for error cases encountered before a successfull glfs_init. This is being tracked in Bug 1134050 for glusterfs-3.5 and Bug 1093594 for mainline.
  • If the /var/run/gluster directory does not exist enabling quota will likely fail (Bug 1117888).

Sunday, October 5, 2014

GlusterFS 3.5.3beta1 has been released for testing

The first beta for GlusterFS 3.5.3 is now available for download.

Packages for different distributions will land on the download server over the next few days. When packages become available, the package maintainers will send a notification to the gluster-users mailinglist.

With this beta release, we make it possible for bug reporters and testers to check if issues have indeed been fixed. All community members are invited to test and/or comment on this release.

If a bug from the list below has not been sufficiently fixed, please open the bug report, leave a comment with details of the testing and change the status of the bug to ASSIGNED.

In case someone has successfully verified a fix for a bug, please change the status of the bug to VERIFIED.

The Release Notes for 3.5.0, 3.5.1 and 3.5.2 contain a listing of all the new features that were added and bugs fixed in the GlusterFS 3.5 stable release.

Bugs Fixed:

  • 1081016: glusterd needs xfsprogs and e2fsprogs packages
  • 1129527: DHT :- data loss - file is missing on renaming same file from multiple client at same time
  • 1129541: [DHT:REBALANCE]: Rebalance failures are seen with error message " remote operation failed: File exists"
  • 1132391: NFS interoperability problem: stripe-xlator removes EOF at end of READDIR
  • 1133949: Minor typo in afr logging
  • 1136221: The memories are exhausted quickly when handle the message which has multi fragments in a single record
  • 1136835: crash on fsync
  • 1138922: DHT + rebalance : rebalance process crashed + data loss + few Directories are present on sub-volumes but not visible on mount point + lookup is not healing directories
  • 1139103: DHT + Snapshot :- If snapshot is taken when Directory is created only on hashed sub-vol; On restoring that snapshot Directory is not listed on mount point and lookup on parent is not healing
  • 1139170: DHT :- rm -rf is not removing stale link file and because of that unable to create file having same name as stale link file
  • 1139245: vdsm invoked oom-killer during rebalance and Killed process 4305, UID 0, (glusterfs nfs process)
  • 1140338: rebalance is not resulting in the hash layout changes being available to nfs client
  • 1140348: Renaming file while rebalance is in progress causes data loss
  • 1140549: DHT: Rebalance process crash after add-brick and `rebalance start' operation
  • 1140556: Core: client crash while doing rename operations on the mount
  • 1141558: AFR : "gluster volume heal <volume_name> info" prints some random characters
  • 1141733: data loss when rebalance + renames are in progress and bricks from replica pairs goes down and comes back
  • 1142052: Very high memory usage during rebalance
  • 1142614: files with open fd's getting into split-brain when bricks goes offline and comes back online
  • 1144315: core: all brick processes crash when quota is enabled
  • 1145000: Spec %post server does not wait for the old glusterd to exit
  • 1147243: nfs: volume set help says the rmtab file is in "/var/lib/glusterd/rmtab"

Known Issues:

  • The following configuration changes are necessary for 'qemu' and 'samba vfs plugin' integration with libgfapi to work seamlessly:
    1. gluster volume set <volname> server.allow-insecure on
    2. restarting the volume is necessary
       gluster volume stop <volname>
       gluster volume start <volname>
      
    3. Edit /etc/glusterfs/glusterd.vol to contain this line:
       option rpc-auth-allow-insecure on
      
    4. restarting glusterd is necessary
       service glusterd restart
      
      More details are also documented in the Gluster Wiki on the Libgfapi with qemu libvirt page.
  • For Block Device translator based volumes open-behind translator at the client side needs to be disabled.
    gluster volume set <volname> performance.open-behind disabled
    
  • libgfapi clients calling glfs_fini before a successfull glfs_init will cause the client to hang as reported here. The workaround is NOT to call glfs_fini for error cases encountered before a successfull glfs_init. This is being tracked in Bug 1134050 for glusterfs-3.5 and Bug 1093594 for mainline.
  • If the /var/run/gluster directory does not exist enabling quota will likely fail (Bug 1117888).

Monday, September 22, 2014

Experimenting with Ceph support for NFS-Ganesha

NFS-Ganesha is a user-space NFS-server that is available in Fedora. It contains several plugins (FSAL, File System Abstraction Layer) for supporting different storage backends. Some of the more interesting are:

Setting up a basic NFS-Ganesha server

Exporting a mounted filesystem is pretty simple. Unfortunately this failed for me when running with the standard nfs-ganesha packages on a minimal Fedora 20 installation. The following changes were needed to make NFS-Ganesha work for a basic export:

  • install rpcbind and make the nfs-ganesha.service depend on it
  • copy /etc/dbus-1/system.d/org.ganesha.nfsd.conf from the sources
  • create a /etc/sysconfig/nfs-ganesha environment file

When these initial things have been taken care of, a configuration file needs to be created. The default configuration file mentioned in the environment file is /etc/ganesha.nfsd.conf. The sources of nfs-ganesha contain some examples, the vfs.conf is quite usable as a starting point. After copying the example and modifying the paths to something more suitable, starting the NFS-server should work:

# systemctl start nfs-ganesha

In case something failed, there should be a note about it in /var/log/ganesha.log.

Exporting the Ceph filesystem with NFS-Ganesha

This assumes you have a working Ceph Cluster which includes several MON, OSD and one or more MDS daemons. The FSAL_CEPH from NFS-Ganesha uses libcephfs which seems to be the same as the ceph-fuse package for Fedora. the easiest way to make sure that the Ceph filesystem is functional, is to try and mount it with ceph-fuse.

The minimal requirements to get a Ceph client system to access the Ceph Cluster, seem to be a /etc/ceph/ceph.conf with a [global] section and a suitable keyring. Creating the ceph.conf on the Fedora system that was done the ceph-deploy:

$ ceph-deploy config push $NFS_SERVER

In my setup I scp'd the /etc/ceph/ceph.client.admin.keyring from one of my Ceph servers to the $NFS_SERVER. There probably are better ways to create/distribute a keyring, but I'm new to Ceph and this worked sufficiently for my testing.

When the above configuration was done, it was possible to mount the Ceph filesystem on the Ceph client that is becoming the NFS-server. This command worked without issues:

# ceph-fuse /mnt
# echo 'Hello Ceph!' > /mnt/README
# umount /mnt

The first write to the Ceph filesystem took a while. This is likely due to the initial work the MDS and OSD daemons need to do (like creating pools for the Ceph filesystem).

After confirming that the Ceph Cluster and Filesystem work, the configuration for NFS-Ganesha can just be taken from the sources and saved as /etc/ganesha.nfsd.conf. With this configuration, and restarting the nfs-ganesha.service, the NFS-export becomes available:

# showmount -e $NFS_SERVER
Export list for $NFS_SERVER:
/ (everyone)

NFSv4 uses a 'pseudo root' as mentioned in the configuration file. This means that mounting the export over NFSv4 results in a virtual directory structure:

# mount -t nfs $NFS_SERVER:/ /mnt
# find /mnt
/mnt
/mnt/nfsv4
/mnt/nfsv4/pseudofs
/mnt/nfsv4/pseudofs/ceph
/mnt/nfsv4/pseudofs/ceph/README

Reading and writing to the mountpoint under /mnt/nfsv4/pseudofs/ceph works fine, as long as the usual permissions allow that. By default NFS-Ganesha enabled 'root squashing', so the 'root' user may not do a lot on the export. Disabling this security measure can be done by placing this option in the export section:

Squash = no_root_squash;

Restart the nfs-ganesha.service after modifying /etc/ganesha.nfsd.conf and writing files as 'root' should work too now.

Future Work

For me, this was a short "let's try it out" while learning about Ceph. At the moment, I have no intention on working on the FSAL_CEPH for NFS-Ganesha. My main interest of this experiment with exporting a Ceph filesystem though NFS-Ganesha on a plain Fedora 20 installation, was to learn about usability of a new NFS-Ganesha configuration/deployment. In order to improve the user experience with NFS-Ganesha, I'll try and fix some of the issues I run into. Progress can be followed in Bug 1144799.

In future, I will mainly use NFS-Ganesha for accessing Gluster Volumes. My colleague Soumya posted a nice explanation on how to download, build and run NFS-Ganesha with support for Gluster. We will be working on improving the out-of-the-box support in Fedora while stabilizing the FSAL_GLUSTER in the upstream NFS-Ganeasha project.

Thursday, July 31, 2014

GlusterFS 3.5.2 has been released!


GlusterFS 3.5.2 has been announced some minutes ago. These are the changes that have been included in this release. Known issues are documented below too.

Release Notes for GlusterFS 3.5.2

This is mostly a bugfix release. The Release Notes for 3.5.0 and 3.5.1 contain a listing of all the new features that were added and bugs fixed.

Bugs Fixed:

  • 1096020: NFS server crashes in _socket_read_vectored_request
  • 1100050: Can't write to quota enable folder
  • 1103050: nfs: reset command does not alter the result for nfs options earlier set
  • 1105891: features/gfid-access: stat on .gfid virtual directory return EINVAL
  • 1111454: creating symlinks generates errors on stripe volume
  • 1112111: Self-heal errors with "afr crawl failed for child 0 with ret -1" while performing rolling upgrade.
  • 1112348: [AFR] I/O fails when one of the replica nodes go down
  • 1112659: Fix inode leaks in gfid-access xlator
  • 1112980: NFS subdir authentication doesn't correctly handle multi-(homed,protocol,etc) network addresses
  • 1113007: nfs-utils should be installed as dependency while installing glusterfs-server
  • 1113403: Excessive logging in quotad.log of the kind 'null client'
  • 1113749: client_t clienttable cliententries are never expanded when all entries are used
  • 1113894: AFR : self-heal of few files not happening when a AWS EC2 Instance is back online after a restart
  • 1113959: Spec %post server does not wait for the old glusterd to exit
  • 1114501: Dist-geo-rep : deletion of files on master, geo-rep fails to propagate to slaves.
  • 1115369: Allow the usage of the wildcard character '*' to the options "nfs.rpc-auth-allow" and "nfs.rpc-auth-reject"
  • 1115950: glfsheal: Improve the way in which we check the presence of replica volumes
  • 1116672: Resource cleanup doesn't happen for clients on servers after disconnect
  • 1116997: mounting a volume over NFS (TCP) with MOUNT over UDP fails
  • 1117241: backport 'gluster volume status --xml' issues
  • 1120151: Glustershd memory usage too high
  • 1124728: SMB: CIFS mount fails with the latest glusterfs rpm's

Known Issues:

  • The following configuration changes are necessary for 'qemu' and 'samba vfs plugin' integration with libgfapi to work seamlessly:
    1. gluster volume set <volname> server.allow-insecure on
    2. restarting the volume is necessary
       gluster volume stop <volname>
       gluster volume start <volname>
      
    3. Edit /etc/glusterfs/glusterd.vol to contain this line:
       option rpc-auth-allow-insecure on
      
    4. restarting glusterd is necessary
       service glusterd restart
      
      More details are also documented in the Gluster Wiki on the Libgfapi with qemu libvirt page.
  • For Block Device translator based volumes open-behind translator at the client side needs to be disabled.
      gluster volume set <volname> performance.open-behind disabled
    
  • libgfapi clients calling glfs_fini before a successfull glfs_init will cause the client to hang as reported here. The workaround is NOT to call glfs_finifor error cases encountered before a successfull glfs_init.
  • If the /var/run/gluster directory does not exist enabling quota will likely fail (Bug 1117888).

Monday, July 21, 2014

Testers needed for GlusterFS 3.5.2beta1

GlusterFS 3.5.2beta1 has just been released. This is the first beta to allow users to verify the fixes for the bugs that were reported. See the bug reports below for more details on how to test and confirm the fix (or not).

This is a bugfix only release. The Release Notes for 3.5.0 and 3.5.1 contain a listing of all the new features that were added and bugs fixed.

Bugs Fixed:

  • 1096020: NFS server crashes in _socket_read_vectored_request
  • 1100050: Can't write to quota enable folder
  • 1103050: nfs: reset command does not alter the result for nfs options earlier set
  • 1105891: features/gfid-access: stat on .gfid virtual directory return EINVAL
  • 1111454: creating symlinks generates errors on stripe volume
  • 1112111: Self-heal errors with "afr crawl failed for child 0 with ret -1" while performing rolling upgrade.
  • 1112348: [AFR] I/O fails when one of the replica nodes go down
  • 1112659: Fix inode leaks in gfid-access xlator
  • 1112980: NFS subdir authentication doesn't correctly handle multi-(homed,protocol,etc) network addresses
  • 1113007: nfs-utils should be installed as dependency while installing glusterfs-server
  • 1113403: Excessive logging in quotad.log of the kind 'null client'
  • 1113749: client_t clienttable cliententries are never expanded when all entries are used
  • 1113894: AFR : self-heal of few files not happening when a AWS EC2 Instance is back online after a restart
  • 1113959: Spec %post server does not wait for the old glusterd to exit
  • 1114501: Dist-geo-rep : deletion of files on master, geo-rep fails to propagate to slaves.
  • 1115369: Allow the usage of the wildcard character '*' to the options "nfs.rpc-auth-allow" and "nfs.rpc-auth-reject"
  • 1115950: glfsheal: Improve the way in which we check the presence of replica volumes
  • 1116672: Resource cleanup doesn't happen for clients on servers after disconnect
  • 1116997: mounting a volume over NFS (TCP) with MOUNT over UDP fails
  • 1117241: backport 'gluster volume status --xml' issues
  • 1120151: Glustershd memory usage too high

Known Issues:

  • The following configuration changes are necessary for qemu and samba integration with libgfapi to work seamlessly:

    1. gluster volume set server.allow-insecure on
    2. restarting the volume is necessary

       gluster volume stop 
       gluster volume start 
      
    3. Edit /etc/glusterfs/glusterd.vol to contain this line:

       option rpc-auth-allow-insecure on
      
    4. restarting glusterd is necessary

       service glusterd restart
      

      More details are also documented in the Gluster Wiki on the Libgfapi with qemu libvirt page.

  • For Block Device translator based volumes open-behind translator at the client side needs to be disabled.

  • libgfapi clients calling glfs_fini before a successfull glfs_init will cause the client to hang as reported here. The workaround is NOT to call glfs_fini for error cases encountered before a successfull glfs_init.

  • If the /var/run/gluster directory does not exist enabling quota will likely fail (Bug 1117888).

Sunday, July 20, 2014

Change the default search engine in Epiphany, the GNOME Web application

When I'm enjoying the sun/wind/rain on the balcony, I tend to use my XO-1.75 for duties where most people would use a tablet. Reading/writing emails, browsing the internet, bug triaging or writing small fixes, release notes and all can be done fine on a small screen. My preference goes definitely towards physical keyboards, and less to their onscreen variants. Even when the keyboard is small, I like the typing on it much more than using a touchscreen for it. Of course, the space saving of not needing to display a keyboard helps too. But well, that aside...


My XO is is installed with the stock OLPC distribution, based on Fedora. Sometimes I use the Sugar desktop environment, on other days I'll switch to GNOME (Classic). With GNOME comes the Epiphany browser (recently renamed to Web). Unfortunately Epiphany uses Google as default search engine, and there is no option in the settings menu to change that. After a little DuckDuckGo'ing, I found a hint that the keyword-search-url can get set by gsettings:

$ gsettings set org.gnome.Epiphany keyword-search-url

Using the gsettings command works fine, but does not apply the option for all users on the system. I could not find a command to change the system-wide settings, which would help with automatically setting the option after a reinstall. More searching (now directly from the addressbar) suggested that I could use a special .gschema.override file. Indeed, the installation of the XO already has some of these .gschema.override files under /usr/share/glib-2.0/schemas/. Dropping the following file in the directory:

# filename: /usr/share/glib-2.0/schemas/50_use-duckduckgo.gschema.override
#
# use https://duckduckgo.com instead of Google for searches from the addressbar
#

[org.gnome.Epiphany]
keyword-search-url='https://duckduckgo.com/?q=%s'

After creating the file, it is needed to 'compile' the gschemas:

# glib-compile-schemas /usr/share/glib-2.0/schemas

Happy searching!

Tuesday, June 24, 2014

glusterfs-3.5.1 has been released


On Tue, Jun 24, 2014 at 03:15:58AM -0700, Gluster Build System wrote:
> 
> 
> SRC: http://bits.gluster.org/pub/gluster/glusterfs/src/glusterfs-3.5.1.tar.gz
> 
> This release is made off jenkins-release-73

Many thanks to everyone how tested the glusterfs-3.5.1 beta releases and 
gave feedback. There were no regressions reported compared to the 3.5.0 
release.

Many bugs have been fixed, and documentation for all new features in 3.5 
should be included now. Thanks to all the reporters, developers and 
testers for improving the 3.5 stable series.

Below you will find the release notes in MarkDown format for 
glusterfs-3.5.1, these are included in the tar.gz as
doc/release-notes/3.5.1.md. The mirror repository on GitHub provides 
a nicely rendered version:
- https://github.com/gluster/glusterfs/blob/v3.5.1/doc/release-notes/3.5.1.md

Packages for different Linux distributions will follow shortly.  
Notifications are normally sent to this list when the packages are 
available for download, and/or have reached the distributions update 
infrastructure.

Changes for a new 3.5.2 release are now being accepted. The list of 
proposed fixes is already growing:
- https://bugzilla.redhat.com/showdependencytree.cgi?hide_resolved=0&id=glusterfs-3.5.2

Anyone is free to request a bugfix or backport for the 3.5.2 release. In 
order to do so, file a bug and set the 'blocked' field to 
'glusterfs-3.5.2' so that we can track the requests. Use this link to 
make it a little easier for yourself:
- https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS&version=3.5.1&blocked=glusterfs-3.5.2

Cheers,
Niels

Release Notes for GlusterFS 3.5.1

This is mostly a bugfix release. The Release Notes for 3.5.0 contain a listing of all the new features that were added.
There are two notable changes that are not only bug fixes, or documentation additions:
  1. a new volume option server.manage-gids has been added This option should be used when users of a volume are in more than approximately 93 groups (Bug 1096425)
  2. Duplicate Request Cache for NFS has now been disabled by default, this may reduce performance for certain workloads, but improves the overall stability and memory footprint for most users

Bugs Fixed:

  • 765202: lgetxattr called with invalid keys on the bricks
  • 833586: inodelk hang from marker_rename_release_newp_lock
  • 859581: self-heal process can sometimes create directories instead of symlinks for the root gfid file in .glusterfs
  • 986429: Backupvolfile server option should work internal to GlusterFS framework
  • 1039544: [FEAT] "gluster volume heal info" should list the entries that actually required to be healed.
  • 1046624: Unable to heal symbolic Links
  • 1046853: AFR : For every file self-heal there are warning messages reported in glustershd.log file
  • 1063190: Volume was not accessible after server side quorum was met
  • 1064096: The old Python Translator code (not Glupy) should be removed
  • 1066996: Using sanlock on a gluster mount with replica 3 (quorum-type auto) leads to a split-brain
  • 1071191: [3.5.1] Sporadic SIGBUS with mmap() on a sparse file created with open(), seek(), write()
  • 1078061: Need ability to heal mismatching user extended attributes without any changelogs
  • 1078365: New xlators are linked as versioned .so files, creating .so.0.0.0
  • 1086743: Add documentation for the Feature: RDMA-connection manager (RDMA-CM)
  • 1086748: Add documentation for the Feature: AFR CLI enhancements
  • 1086749: Add documentation for the Feature: Exposing Volume Capabilities
  • 1086750: Add documentation for the Feature: File Snapshots in GlusterFS
  • 1086751: Add documentation for the Feature: gfid-access
  • 1086752: Add documentation for the Feature: On-Wire Compression/Decompression
  • 1086754: Add documentation for the Feature: Quota Scalability
  • 1086755: Add documentation for the Feature: readdir-ahead
  • 1086756: Add documentation for the Feature: zerofill API for GlusterFS
  • 1086758: Add documentation for the Feature: Changelog based parallel geo-replication
  • 1086760: Add documentation for the Feature: Write Once Read Many (WORM) volume
  • 1086762: Add documentation for the Feature: BD Xlator - Block Device translator
  • 1086766: Add documentation for the Feature: Libgfapi
  • 1086774: Add documentation for the Feature: Access Control List - Version 3 support for Gluster NFS
  • 1086781: Add documentation for the Feature: Eager locking
  • 1086782: Add documentation for the Feature: glusterfs and oVirt integration
  • 1086783: Add documentation for the Feature: qemu 1.3 - libgfapi integration
  • 1088848: Spelling errors in rpc/rpc-transport/rdma/src/rdma.c
  • 1089054: gf-error-codes.h is missing from source tarball
  • 1089470: SMB: Crash on brick process during compile kernel.
  • 1089934: list dir with more than N files results in Input/output error
  • 1091340: Doc: Add glfs_fini known issue to release notes 3.5
  • 1091392: glusterfs.spec.in: minor/nit changes to sync with Fedora spec
  • 1095256: Excessive logging from self-heal daemon, and bricks
  • 1095595: Stick to IANA standard while allocating brick ports
  • 1095775: Add support in libgfapi to fetch volume info from glusterd.
  • 1095971: Stopping/Starting a Gluster volume resets ownership
  • 1096040: AFR : self-heal-daemon not clearing the change-logs of all the sources after self-heal
  • 1096425: i/o error when one user tries to access RHS volume over NFS with 100+ GIDs
  • 1099878: Need support for handle based Ops to fetch/modify extended attributes of a file
  • 1101647: gluster volume heal volname statistics heal-count not giving desired output.
  • 1102306: license: xlators/features/glupy dual license GPLv2 and LGPLv3+
  • 1103413: Failure in gf_log_init reopening stderr
  • 1104592: heal info may give Success instead of transport end point not connected when a brick is down.
  • 1104915: glusterfsd crashes while doing stress tests
  • 1104919: Fix memory leaks in gfid-access xlator.
  • 1104959: Dist-geo-rep : some of the files not accessible on slave after the geo-rep sync from master to slave.
  • 1105188: Two instances each, of brick processes, glusterfs-nfs and quotad seen after glusterd restart
  • 1105524: Disable nfs.drc by default
  • 1107937: quota-anon-fd-nfs.t fails spuriously
  • 1109832: I/O fails for for glusterfs 3.4 AFR clients accessing servers upgraded to glusterfs 3.5
  • 1110777: glusterfsd OOM - using all memory when quota is enabled

Known Issues:

  • The following configuration changes are necessary for qemu and samba integration with libgfapi to work seamlessly:
    1. gluster volume set <volname> server.allow-insecure on
    2. restarting the volume is necessary
      gluster volume stop 
      gluster volume start 
      
    3. Edit /etc/glusterfs/glusterd.vol to contain this line:
      option rpc-auth-allow-insecure on
      
    4. restarting glusterd is necessary
      service glusterd restart
      
    More details are also documented in the Gluster Wiki on the Libgfapi with qemu libvirt page.
  • For Block Device translator based volumes open-behind translator at the client side needs to be disabled.
  • libgfapi clients calling glfs_fini before a successfull glfs_init will cause the client to hang has been reported by QEMU developers. The workaround is NOT to call glfs_fini for error cases encountered before a successfull glfs_init. Follow Bug 1091335 to get informed when a release is made available that contains a final fix.
  • After enabling server.manage-gids, the volume needs to be stopped and started again to have the option enabled in the brick processes
    gluster volume stop <volname>
    gluster volume start <volname>
    

Sunday, May 25, 2014

glusterfs-3.5.1beta released

Reposting the email to the Gluster Users and Developers mailinglists.
On Sat, 24 May, 2014 at 11:34:36PM -0700, Gluster Build System wrote:
> > SRC: http://bits.gluster.org/pub/gluster/glusterfs/src/glusterfs-3.5.1beta.tar.gz
This beta release is intended to verify the changes that should resolve the bugs listed below. We appreciate tests done by anyone. Please leave a comment in the respective bugreport with a short description of the success or failure. Visiting one of the bugreports is as easy as opening the bugzilla.redhat.com/$BUG URL, for the first in the list, this results in http://bugzilla.redhat.com/765202.

Bugs expected to be fixed (31 in total since 3.5.0):
#765202 - lgetxattr called with invalid keys on the bricks
#833586 - inodelk hang from marker_rename_release_newp_lock
#859581 - self-heal process can sometimes create directories instead of symlinks for the root gfid file in .glusterfs
#986429 - Backupvolfile server option should work internal to GlusterFS framework
#1039544 - [FEAT] "gluster volume heal info" should list the entries that actually required to be healed.
#1046624 - Unable to heal symbolic Links
#1046853 - AFR : For every file self-heal there are warning messages reported in glustershd.log file
#1063190 - [RHEV-RHS] Volume was not accessible after server side quorum was met
#1064096 - The old Python Translator code (not Glupy) should be removed
#1066996 - Using sanlock on a gluster mount with replica 3 (quorum-type auto) leads to a split-brain
#1071191 - [3.5.1] Sporadic SIGBUS with mmap() on a sparse file created with open(), seek(), write()
#1078061 - Need ability to heal mismatching user extended attributes without any changelogs
#1078365 - New xlators are linked as versioned .so files, creating <xlator>.so.0.0.0
#1086748 - Add documentation for the Feature: AFR CLI enhancements
#1086750 - Add documentation for the Feature: File Snapshots in GlusterFS
#1086752 - Add documentation for the Feature: On-Wire Compression/Decompression
#1086756 - Add documentation for the Feature: zerofill API for GlusterFS
#1086758 - Add documentation for the Feature: Changelog based parallel geo-replication
#1086760 - Add documentation for the Feature: Write Once Read Many (WORM) volume
#1086762 - Add documentation for the Feature: BD Xlator - Block Device translator
#1088848 - Spelling errors in rpc/rpc-transport/rdma/src/rdma.c
#1089054 - gf-error-codes.h is missing from source tarball
#1089470 - SMB: Crash on brick process during compile kernel.
#1089934 - list dir with more than N files results in Input/output error
#1091340 - Doc: Add glfs_fini known issue to release notes 3.5
#1091392 - glusterfs.spec.in: minor/nit changes to sync with Fedora spec
#1095775 - Add support in libgfapi to fetch volume info from glusterd.
#1095971 - Stopping/Starting a Gluster volume resets ownership
#1096040 - AFR : self-heal-daemon not clearing the change-logs of all the sources after self-heal
#1096425 - i/o error when one user tries to access RHS volume over NFS with 100+ GIDs
#1099878 - Need support for handle based Ops to fetch/modify extended attributes of a file

Before a final glusterfs-3.5.1 release is made, we hope to have all the blocker bugs fixed. There are currently 13 bugs marked that still need some work done:
#1081016 - glusterd needs xfsprogs and e2fsprogs packages
#1086743 - Add documentation for the Feature: RDMA-connection manager (RDMA-CM)
#1086749 - Add documentation for the Feature: Exposing Volume Capabilities
#1086751 - Add documentation for the Feature: gfid-access
#1086754 - Add documentation for the Feature: Quota Scalability
#1086755 - Add documentation for the Feature: readdir-ahead
#1086759 - Add documentation for the Feature: Improved block device translator
#1086766 - Add documentation for the Feature: Libgfapi
#1086774 - Add documentation for the Feature: Access Control List - Version 3 support for Gluster NFS
#1086781 - Add documentation for the Feature: Eager locking
#1086782 - Add documentation for the Feature: glusterfs and oVirt integration
#1086783 - Add documentation for the Feature: qemu 1.3 - libgfapi integration
#1095595 - Stick to IANA standard while allocating brick ports

A more detailed overview of the status of each of these bugs is here.

Tuesday, April 15, 2014

Configuring autofs for GlusterFS 3.5

GlusterFS 3.5 has not been released yet, but that should happen hopefully anytime soon (currently in beta). The RPM-packaging in this version has changed a little, and now offers a glusterfs-cli package. This package mainly contains the gluster commandline interface (and pulls in any dependencies).

On of the very useful things that is now made possible, is to list the available volumes on Gluster Storge Servers. This similar functionality is used by the /etc/auto.net script to list NFS-exports that are available for mounting. The auto.net script is by default enabled after installing and starting autofs:

# yum install autofs
# systemctl enable autofs.service
# systemctl start autofs.service

Checking, and mounting NFS-exports is made as easy as:

$ ls /net/nfs-server.example.net
archive  media  mock_cache  olpc
$ ls /net/nfs-server.example.net/mock_cache/fedora-rawhide-armhfp/
yum_cache

Making this functionality available for Gluster Volumes is simple, just follow these steps:

  1. install the gluster command

     # yum install glusterfs-cli
  2. save the file below as /etc/auto.glfs

     #!/bin/bash
     # /etc/auto.glfs -- based on /etc/auto.net
     #
     # This file must be executable to work! chmod 755!
     #
     # Look at what a host is exporting to determine what we can mount.
     # This is very simple, but it appears to work surprisingly well
     #
    
     key="$1"
    
     # add "nosymlink" here if you want to suppress symlinking local filesystems
     # add "nonstrict" to make it OK for some filesystems to not mount
     opts="-fstype=glusterfs,nodev,nosuid"
    
     for P in /usr/local/bin /usr/local/sbin /usr/bin /usr/sbin /bin /sbin
     do
         if [ -x ${P}/gluster ]
         then
                 GLUSTER_CLI=${P}/gluster
                 break
         fi
     done
    
     [ -x ${GLUSTER_CLI} ] || exit 1
    
     ${GLUSTER_CLI} --remote-host="${key}" volume list | \
         awk -v key="$key" -v opts="$opts" -- '
         BEGIN   { ORS=""; first=1 }
                 { if (first) { print opts; first=0 }; print " \\\n\t/" $1, key ":/" $1 }
         END     { if (!first) print "\n"; else exit 1 }' | \
         sed 's/#/\\#/g'
  3. make the script executable

     # chmod 0755 /etc/auto.glfs
  4. add an automount point to the autofs configuration

     # echo /glfs /etc/auto.glfs > /etc/auto.master.d/glfs.autofs
  5. reload the autofs configuration

     # systemctl reload autofs.service

After this, autofs should have created a new /glfs directory. The directory itself is empty, but a ls /glfs/gluster.example.net will show all the available volumes on the gluster.example.net server. These volumes can now be accessed through the autofs mountpoint. When the volumes are not used anymore, autofs will automatically unmount them after a timeout.

Sunday, February 23, 2014

Setting up a test-environment for Apache CloudStack and Gluster

This is an example of how to configure an environment where you can test CloudStack and Gluster. It uses two machines on the same LAN, one acts as a KVM hypervisor and the other as storage and management server. Because the (virtual) networking in the hypervisor is a little more complex than the networking on the management server, the hypervisor will be setup with an OpenVPN connection so that the local LAN is not affected with 'foreign' network traffic.

I am not a CloudStack specialist, so this configuration may not be optimal for real world usage. It is the intention to be able to test CloudStack and its Gluster integration in existing networks. The CloudStack installation and configuration done is suitable for testing and development systems, for production environments it is highly recommended to follow the CloudStack documentation instead.

 .----------------.                       .-------------------.
 |                |                       |                   |
 | KVM Hypervisor | <------- LAN -------> | Management Server |
 |                |    ^-- OpenVPN --^    |                   |
 '----------------'                       '-------------------'
agent.cloudstack.tld                      storage.cloudstack.tld

Both systems have one network interface with a static IP-address. In the LAN, other IP-addresses can not be used. This makes it difficult to access virtual machines, but that does not matter too much for this testing.

Both systems need a basic installation:

  • Red Hat Enterprise Linux 6.5 (CentOS 6.5 should work too)
  • Fedora EPEL enabled (howto install epel-release)
  • enable ssh access
  • SELinux in permissive mode (or disabled)
  • firewall enabled, but not restricting anything
  • Java 1.7 from the standard java-1.7.0-openjdk packages (not Java 1.6)

On the hypervisor, an additional (internal only) bridge needs to be setup. This bridge will be used for providing IP-addresses to the virtual machines. Each virtual machine seems to need at least 3 IP-addresses. This is a default in CloudStack. This example uses virtual networks 192.168.N.0/24, where N is 0 to 4.

Configuration for the main cloudbr0 device:

#file: /etc/sysconfig/network-scripts/ifcfg-cloudbr0
DEVICE=cloudbr0
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.0.1
NETMASK=255.255.255.0
NM_CONTROLLED=no

And the additional IP-addresses on the cloudbr0 bridge (create 4 files, replace N by 1, 2, 3 and 4):

#file: /etc/sysconfig/network-scripts/ifcfg-cloudbr0:N
DEVICE=cloudbr0:N
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.N.1
NETMASK=255.255.255.0
NM_CONTROLLED=no

Enable the new cloudbr0 bridge with all its IP-addresses:

# ifup cloudbr0
Any of the VMs that have a 192.168.*.* address, should be able to get to the real LAN, and ultimately also the internet. Enabling NAT for the internal virtual networks is the easiest:
# iptables -t nat -A POSTROUTING -o eth0 -s 192.168.0.0/24 -j MASQUERADE
# iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/24 -j MASQUERADE
# iptables -t nat -A POSTROUTING -o eth0 -s 192.168.2.0/24 -j MASQUERADE
# iptables -t nat -A POSTROUTING -o eth0 -s 192.168.3.0/24 -j MASQUERADE
# iptables -t nat -A POSTROUTING -o eth0 -s 192.168.4.0/24 -j MASQUERADE
# service iptables save

The hypervisor will need to be setup to act as a gateway to the virtual machines on the cloudbr0 bridge. In order to so do, a very basic OpenVPN service does the trick:

# yum install openvpn
# openvpn --genkey --secret /etc/openvpn/static.key
# cat << EOF > /etc/openvpn/server.conf
dev tun
ifconfig 192.168.200.1 192.168.200.2
secret static.key
EOF
# chkconfig openvpn on
# service openvpn start

On the management server, it is needed to configure OpenVPN as a client, so that routing to the virtual networks is possible:

# yum install openvpn
# cat << EOF > /etc/openvpn/client.conf
remote real-hostname-of-hypervisor.example.net
dev tun
ifconfig 192.168.200.2 192.168.200.1
secret static.key
EOF
# scp real-hostname-of-hypervisor.example.net:/etc/openvpn/static.key /etc/openvpn
# chkconfig opennvpn on
# service openvpn start

In /etc/hosts (on both the hypervisor and management server) the internal hostnames for the environment should be added:

#file: /etc/hosts
192.168.200.1 agent.cloudstack.tld
192.168.200.2 storage.cloudstack.tld

The hypervisor will also function as a DNS-server for the virtual machines. The easiest is to use dnsmasq which uses /etc/hosts and /etc/resolv.conf for resolving:

# yum install dnsmasq
# chkconfig dnsmasq on
# service dnsmasq start

The management server is also used as a Gluster Storage Server. Therefor it needs to have some Gluster packages:

# wget -O /etc/yum.repo.d/glusterfs-epel.repo \
http://download.gluster.org/pub/gluster/glusterfs/3.4/LATEST/RHEL/glusterfs-epel.repo
# yum install glusterfs-server
# vim /etc/glusterfs/glusterd.vol

# service glusterd restart

Create two volumes where CloudStack will store disk images. Before starting the volumes, apply the required settings too. Note that the hostname that holds the bricks should be resolvable by the hypervisor and the Secondary Storage VMs. This example does not show how to create volumes for production usage, do not create volumes like this for anything else than testing and scratch data.

# mkdir -p /bricks/primary/data
# mkdir -p /bricks/secondary/data
# gluster volume create primary storage.cloudstack.tld:/bricks/primary/data
# gluster volume set primary storage.owner-uid 36
# gluster volume set primary storage.owner-gid 36
# gluster volume set primary server.allow-insecure on
# gluster volume set primary nfs.disable true
# gluster volume start primary
# gluster volume create secondary storage.cloudstack.tld:/bricks/secondary/data
# gluster volume set secondary storage.owner-uid 36
# gluster volume set secondary storage.owner-gid 36
# gluster volume start secondary

When the preparation is all done, it is time to install Apache CloudStack. It is planned to have support for Gluster in CloudStack 4.4. At the moment not all required changes are included in the CloudStack git repository. Therefor, is is needed to build the RPM packages from the Gluster Forge repository where the development is happening. On a system running RHEL-6.5, checkout the sources and build the packages (this needs a standard CloudStack development environment, including java-1.7.0-openjdk-devel, Apache Maven and others):

$ git clone git://forge.gluster.org/cloudstack-gluster/cloudstack.git
$ cd cloudstack
$ git checkout -t -b wip/master/gluster
$ cd packaging/centos63
$ ./package.sh

In the end, these packages should have been build:

  • cloudstack-management-4.4.0-SNAPSHOT.el6.x86_64.rpm
  • cloudstack-common-4.4.0-SNAPSHOT.el6.x86_64.rpm
  • cloudstack-agent-4.4.0-SNAPSHOT.el6.x86_64.rpm
  • cloudstack-usage-4.4.0-SNAPSHOT.el6.x86_64.rpm
  • cloudstack-cli-4.4.0-SNAPSHOT.el6.x86_64.rpm
  • cloudstack-awsapi-4.4.0-SNAPSHOT.el6.x86_64.rpm

On the management server, install the following packages:

# yum localinstall cloudstack-management-4.4.0-SNAPSHOT.el6.x86_64.rpm \
cloudstack-common-4.4.0-SNAPSHOT.el6.x86_64.rpm \
cloudstack-awsapi-4.4.0-SNAPSHOT.el6.x86_64.rpm

Install and configure the database:

# yum install mysql-server
# chkconfig mysqld on
# service mysqld start
# vim /etc/cloudstack/management/classpath.conf

# cloudstack-setup-databases cloud:secret --deploy-as=root:

Install the systemvm templates:

# mount -t nfs storage.cloudstack.tld:/secondary /mnt
# /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \
-m /mnt \
-h kvm \
-u http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/lastSuccessfulBuild/artifact/tools/appliance/dist/systemvmtemplate-master-kvm.qcow2.bz2
# umount /mnt

The management server is now prepared, and the webui can get configured:

# cloudstack-setup-management

On the hypervisor, install the following additional packages:

# yum install qemu-kvm libvirt glusterfs-fuse
# yum localinstall cloudstack-common-4.4.0-SNAPSHOT.el6.x86_64.rpm \
cloudstack-agent-4.4.0-SNAPSHOT.el6.x86_64.rpm
# cloudstack-setup-agent

Make sure that in /etc/cloudstack/agent/agent.properties the right NICs are being used:

guest.network.device=cloudbr0
private.bridge.name=cloudbr0
private.network.device=cloudbr0
network.direct.device=cloudbr0
public.network.device=cloudbr0

Go to the CloudStack webinterface, this should be running on the management server: http://real-hostname-of-mgmt.example.net:8080/client The default username/password is: admin / password

It is easiest to skip the configuration wizard (not sure if that supports Gluster already). When the normal interface is shown, under 'Infrastructure' a new 'Zone' can get added. The Zone wizard will need the following input:

  • DNS 1: 192.168.0.1
  • Internal DNS 1: 192.168.0.1
  • Hypervisor: KVM

Under POD, use these options:

  • Reserved system gateway: 192.168.0.1
  • Reserved system netmask: 255.255.255.0
  • Start reserved system IP: 192.168.0.10
  • End reserved system IP: 192.168.0.250

Next the network config for the virtual machines:

  • Guest gateway: 192.168.1.1
  • Guest system netmask: 255.255.255.0
  • Guest start IP: 192.168.1.10
  • Guest end IP: 192.168.1.250

Primary storage:

  • Type: Gluster
  • Server: storage.cloudstack.tld
  • Volume: primary

Secondary Storage:

  • Type: nfs
  • Server: storage.cloudstack.tld
  • path: /secondary

Hypervisor agent:

  • hostname: agent.cloudstack.tld
  • username: root
  • password: password

If this all succeeded, the newly created Zone can get enabled. After a while, there should be two system VMs listed in the Infrastructure. It is possible to log in on these system VMs and check if all is working. To do so, log in over SSH on the hypervisor and connect to the VMs through libvirt:

# virsh list
 Id    Name                           State
----------------------------------------------------
 1     s-1-VM                         running
 2     v-2-VM                         running

# virsh console 1
Connected to domain s-1-VM
Escape character is ^]

Debian GNU/Linux 7 s-1-VM ttyS0

s-1-VM login: root
Password: password
...
root@s-1-VM:~# 

Log out from the shell, and press CTRL+] to disconnect from the console.

To verify that this VM indeed runs with the QEMU+libgfapi integration, check the log file that libvirt writes and confirm that there is a -drive with a glusterfs+tcp:// URL in /var/log/libvirt/qemu/s-1-VM.log:

... /usr/libexec/qemu-kvm -name s-1-VM ... -drive file=gluster+tcp://storage.cloudstack.tld:24007/primary/d691ac19-4ec1-47c1-b765-55f804b78bec,...