Thursday, July 31, 2014

GlusterFS 3.5.2 has been released!


GlusterFS 3.5.2 has been announced some minutes ago. These are the changes that have been included in this release. Known issues are documented below too.

Release Notes for GlusterFS 3.5.2

This is mostly a bugfix release. The Release Notes for 3.5.0 and 3.5.1 contain a listing of all the new features that were added and bugs fixed.

Bugs Fixed:

  • 1096020: NFS server crashes in _socket_read_vectored_request
  • 1100050: Can't write to quota enable folder
  • 1103050: nfs: reset command does not alter the result for nfs options earlier set
  • 1105891: features/gfid-access: stat on .gfid virtual directory return EINVAL
  • 1111454: creating symlinks generates errors on stripe volume
  • 1112111: Self-heal errors with "afr crawl failed for child 0 with ret -1" while performing rolling upgrade.
  • 1112348: [AFR] I/O fails when one of the replica nodes go down
  • 1112659: Fix inode leaks in gfid-access xlator
  • 1112980: NFS subdir authentication doesn't correctly handle multi-(homed,protocol,etc) network addresses
  • 1113007: nfs-utils should be installed as dependency while installing glusterfs-server
  • 1113403: Excessive logging in quotad.log of the kind 'null client'
  • 1113749: client_t clienttable cliententries are never expanded when all entries are used
  • 1113894: AFR : self-heal of few files not happening when a AWS EC2 Instance is back online after a restart
  • 1113959: Spec %post server does not wait for the old glusterd to exit
  • 1114501: Dist-geo-rep : deletion of files on master, geo-rep fails to propagate to slaves.
  • 1115369: Allow the usage of the wildcard character '*' to the options "nfs.rpc-auth-allow" and "nfs.rpc-auth-reject"
  • 1115950: glfsheal: Improve the way in which we check the presence of replica volumes
  • 1116672: Resource cleanup doesn't happen for clients on servers after disconnect
  • 1116997: mounting a volume over NFS (TCP) with MOUNT over UDP fails
  • 1117241: backport 'gluster volume status --xml' issues
  • 1120151: Glustershd memory usage too high
  • 1124728: SMB: CIFS mount fails with the latest glusterfs rpm's

Known Issues:

  • The following configuration changes are necessary for 'qemu' and 'samba vfs plugin' integration with libgfapi to work seamlessly:
    1. gluster volume set <volname> server.allow-insecure on
    2. restarting the volume is necessary
       gluster volume stop <volname>
       gluster volume start <volname>
      
    3. Edit /etc/glusterfs/glusterd.vol to contain this line:
       option rpc-auth-allow-insecure on
      
    4. restarting glusterd is necessary
       service glusterd restart
      
      More details are also documented in the Gluster Wiki on the Libgfapi with qemu libvirt page.
  • For Block Device translator based volumes open-behind translator at the client side needs to be disabled.
      gluster volume set <volname> performance.open-behind disabled
    
  • libgfapi clients calling glfs_fini before a successfull glfs_init will cause the client to hang as reported here. The workaround is NOT to call glfs_finifor error cases encountered before a successfull glfs_init.
  • If the /var/run/gluster directory does not exist enabling quota will likely fail (Bug 1117888).

Monday, July 21, 2014

Testers needed for GlusterFS 3.5.2beta1

GlusterFS 3.5.2beta1 has just been released. This is the first beta to allow users to verify the fixes for the bugs that were reported. See the bug reports below for more details on how to test and confirm the fix (or not).

This is a bugfix only release. The Release Notes for 3.5.0 and 3.5.1 contain a listing of all the new features that were added and bugs fixed.

Bugs Fixed:

  • 1096020: NFS server crashes in _socket_read_vectored_request
  • 1100050: Can't write to quota enable folder
  • 1103050: nfs: reset command does not alter the result for nfs options earlier set
  • 1105891: features/gfid-access: stat on .gfid virtual directory return EINVAL
  • 1111454: creating symlinks generates errors on stripe volume
  • 1112111: Self-heal errors with "afr crawl failed for child 0 with ret -1" while performing rolling upgrade.
  • 1112348: [AFR] I/O fails when one of the replica nodes go down
  • 1112659: Fix inode leaks in gfid-access xlator
  • 1112980: NFS subdir authentication doesn't correctly handle multi-(homed,protocol,etc) network addresses
  • 1113007: nfs-utils should be installed as dependency while installing glusterfs-server
  • 1113403: Excessive logging in quotad.log of the kind 'null client'
  • 1113749: client_t clienttable cliententries are never expanded when all entries are used
  • 1113894: AFR : self-heal of few files not happening when a AWS EC2 Instance is back online after a restart
  • 1113959: Spec %post server does not wait for the old glusterd to exit
  • 1114501: Dist-geo-rep : deletion of files on master, geo-rep fails to propagate to slaves.
  • 1115369: Allow the usage of the wildcard character '*' to the options "nfs.rpc-auth-allow" and "nfs.rpc-auth-reject"
  • 1115950: glfsheal: Improve the way in which we check the presence of replica volumes
  • 1116672: Resource cleanup doesn't happen for clients on servers after disconnect
  • 1116997: mounting a volume over NFS (TCP) with MOUNT over UDP fails
  • 1117241: backport 'gluster volume status --xml' issues
  • 1120151: Glustershd memory usage too high

Known Issues:

  • The following configuration changes are necessary for qemu and samba integration with libgfapi to work seamlessly:

    1. gluster volume set server.allow-insecure on
    2. restarting the volume is necessary

       gluster volume stop 
       gluster volume start 
      
    3. Edit /etc/glusterfs/glusterd.vol to contain this line:

       option rpc-auth-allow-insecure on
      
    4. restarting glusterd is necessary

       service glusterd restart
      

      More details are also documented in the Gluster Wiki on the Libgfapi with qemu libvirt page.

  • For Block Device translator based volumes open-behind translator at the client side needs to be disabled.

  • libgfapi clients calling glfs_fini before a successfull glfs_init will cause the client to hang as reported here. The workaround is NOT to call glfs_fini for error cases encountered before a successfull glfs_init.

  • If the /var/run/gluster directory does not exist enabling quota will likely fail (Bug 1117888).

Sunday, July 20, 2014

Change the default search engine in Epiphany, the GNOME Web application

When I'm enjoying the sun/wind/rain on the balcony, I tend to use my XO-1.75 for duties where most people would use a tablet. Reading/writing emails, browsing the internet, bug triaging or writing small fixes, release notes and all can be done fine on a small screen. My preference goes definitely towards physical keyboards, and less to their onscreen variants. Even when the keyboard is small, I like the typing on it much more than using a touchscreen for it. Of course, the space saving of not needing to display a keyboard helps too. But well, that aside...


My XO is is installed with the stock OLPC distribution, based on Fedora. Sometimes I use the Sugar desktop environment, on other days I'll switch to GNOME (Classic). With GNOME comes the Epiphany browser (recently renamed to Web). Unfortunately Epiphany uses Google as default search engine, and there is no option in the settings menu to change that. After a little DuckDuckGo'ing, I found a hint that the keyword-search-url can get set by gsettings:

$ gsettings set org.gnome.Epiphany keyword-search-url

Using the gsettings command works fine, but does not apply the option for all users on the system. I could not find a command to change the system-wide settings, which would help with automatically setting the option after a reinstall. More searching (now directly from the addressbar) suggested that I could use a special .gschema.override file. Indeed, the installation of the XO already has some of these .gschema.override files under /usr/share/glib-2.0/schemas/. Dropping the following file in the directory:

# filename: /usr/share/glib-2.0/schemas/50_use-duckduckgo.gschema.override
#
# use https://duckduckgo.com instead of Google for searches from the addressbar
#

[org.gnome.Epiphany]
keyword-search-url='https://duckduckgo.com/?q=%s'

After creating the file, it is needed to 'compile' the gschemas:

# glib-compile-schemas /usr/share/glib-2.0/schemas

Happy searching!

Tuesday, June 24, 2014

glusterfs-3.5.1 has been released


On Tue, Jun 24, 2014 at 03:15:58AM -0700, Gluster Build System wrote:
> 
> 
> SRC: http://bits.gluster.org/pub/gluster/glusterfs/src/glusterfs-3.5.1.tar.gz
> 
> This release is made off jenkins-release-73

Many thanks to everyone how tested the glusterfs-3.5.1 beta releases and 
gave feedback. There were no regressions reported compared to the 3.5.0 
release.

Many bugs have been fixed, and documentation for all new features in 3.5 
should be included now. Thanks to all the reporters, developers and 
testers for improving the 3.5 stable series.

Below you will find the release notes in MarkDown format for 
glusterfs-3.5.1, these are included in the tar.gz as
doc/release-notes/3.5.1.md. The mirror repository on GitHub provides 
a nicely rendered version:
- https://github.com/gluster/glusterfs/blob/v3.5.1/doc/release-notes/3.5.1.md

Packages for different Linux distributions will follow shortly.  
Notifications are normally sent to this list when the packages are 
available for download, and/or have reached the distributions update 
infrastructure.

Changes for a new 3.5.2 release are now being accepted. The list of 
proposed fixes is already growing:
- https://bugzilla.redhat.com/showdependencytree.cgi?hide_resolved=0&id=glusterfs-3.5.2

Anyone is free to request a bugfix or backport for the 3.5.2 release. In 
order to do so, file a bug and set the 'blocked' field to 
'glusterfs-3.5.2' so that we can track the requests. Use this link to 
make it a little easier for yourself:
- https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS&version=3.5.1&blocked=glusterfs-3.5.2

Cheers,
Niels

Release Notes for GlusterFS 3.5.1

This is mostly a bugfix release. The Release Notes for 3.5.0 contain a listing of all the new features that were added.
There are two notable changes that are not only bug fixes, or documentation additions:
  1. a new volume option server.manage-gids has been added This option should be used when users of a volume are in more than approximately 93 groups (Bug 1096425)
  2. Duplicate Request Cache for NFS has now been disabled by default, this may reduce performance for certain workloads, but improves the overall stability and memory footprint for most users

Bugs Fixed:

  • 765202: lgetxattr called with invalid keys on the bricks
  • 833586: inodelk hang from marker_rename_release_newp_lock
  • 859581: self-heal process can sometimes create directories instead of symlinks for the root gfid file in .glusterfs
  • 986429: Backupvolfile server option should work internal to GlusterFS framework
  • 1039544: [FEAT] "gluster volume heal info" should list the entries that actually required to be healed.
  • 1046624: Unable to heal symbolic Links
  • 1046853: AFR : For every file self-heal there are warning messages reported in glustershd.log file
  • 1063190: Volume was not accessible after server side quorum was met
  • 1064096: The old Python Translator code (not Glupy) should be removed
  • 1066996: Using sanlock on a gluster mount with replica 3 (quorum-type auto) leads to a split-brain
  • 1071191: [3.5.1] Sporadic SIGBUS with mmap() on a sparse file created with open(), seek(), write()
  • 1078061: Need ability to heal mismatching user extended attributes without any changelogs
  • 1078365: New xlators are linked as versioned .so files, creating .so.0.0.0
  • 1086743: Add documentation for the Feature: RDMA-connection manager (RDMA-CM)
  • 1086748: Add documentation for the Feature: AFR CLI enhancements
  • 1086749: Add documentation for the Feature: Exposing Volume Capabilities
  • 1086750: Add documentation for the Feature: File Snapshots in GlusterFS
  • 1086751: Add documentation for the Feature: gfid-access
  • 1086752: Add documentation for the Feature: On-Wire Compression/Decompression
  • 1086754: Add documentation for the Feature: Quota Scalability
  • 1086755: Add documentation for the Feature: readdir-ahead
  • 1086756: Add documentation for the Feature: zerofill API for GlusterFS
  • 1086758: Add documentation for the Feature: Changelog based parallel geo-replication
  • 1086760: Add documentation for the Feature: Write Once Read Many (WORM) volume
  • 1086762: Add documentation for the Feature: BD Xlator - Block Device translator
  • 1086766: Add documentation for the Feature: Libgfapi
  • 1086774: Add documentation for the Feature: Access Control List - Version 3 support for Gluster NFS
  • 1086781: Add documentation for the Feature: Eager locking
  • 1086782: Add documentation for the Feature: glusterfs and oVirt integration
  • 1086783: Add documentation for the Feature: qemu 1.3 - libgfapi integration
  • 1088848: Spelling errors in rpc/rpc-transport/rdma/src/rdma.c
  • 1089054: gf-error-codes.h is missing from source tarball
  • 1089470: SMB: Crash on brick process during compile kernel.
  • 1089934: list dir with more than N files results in Input/output error
  • 1091340: Doc: Add glfs_fini known issue to release notes 3.5
  • 1091392: glusterfs.spec.in: minor/nit changes to sync with Fedora spec
  • 1095256: Excessive logging from self-heal daemon, and bricks
  • 1095595: Stick to IANA standard while allocating brick ports
  • 1095775: Add support in libgfapi to fetch volume info from glusterd.
  • 1095971: Stopping/Starting a Gluster volume resets ownership
  • 1096040: AFR : self-heal-daemon not clearing the change-logs of all the sources after self-heal
  • 1096425: i/o error when one user tries to access RHS volume over NFS with 100+ GIDs
  • 1099878: Need support for handle based Ops to fetch/modify extended attributes of a file
  • 1101647: gluster volume heal volname statistics heal-count not giving desired output.
  • 1102306: license: xlators/features/glupy dual license GPLv2 and LGPLv3+
  • 1103413: Failure in gf_log_init reopening stderr
  • 1104592: heal info may give Success instead of transport end point not connected when a brick is down.
  • 1104915: glusterfsd crashes while doing stress tests
  • 1104919: Fix memory leaks in gfid-access xlator.
  • 1104959: Dist-geo-rep : some of the files not accessible on slave after the geo-rep sync from master to slave.
  • 1105188: Two instances each, of brick processes, glusterfs-nfs and quotad seen after glusterd restart
  • 1105524: Disable nfs.drc by default
  • 1107937: quota-anon-fd-nfs.t fails spuriously
  • 1109832: I/O fails for for glusterfs 3.4 AFR clients accessing servers upgraded to glusterfs 3.5
  • 1110777: glusterfsd OOM - using all memory when quota is enabled

Known Issues:

  • The following configuration changes are necessary for qemu and samba integration with libgfapi to work seamlessly:
    1. gluster volume set <volname> server.allow-insecure on
    2. restarting the volume is necessary
      gluster volume stop 
      gluster volume start 
      
    3. Edit /etc/glusterfs/glusterd.vol to contain this line:
      option rpc-auth-allow-insecure on
      
    4. restarting glusterd is necessary
      service glusterd restart
      
    More details are also documented in the Gluster Wiki on the Libgfapi with qemu libvirt page.
  • For Block Device translator based volumes open-behind translator at the client side needs to be disabled.
  • libgfapi clients calling glfs_fini before a successfull glfs_init will cause the client to hang has been reported by QEMU developers. The workaround is NOT to call glfs_fini for error cases encountered before a successfull glfs_init. Follow Bug 1091335 to get informed when a release is made available that contains a final fix.
  • After enabling server.manage-gids, the volume needs to be stopped and started again to have the option enabled in the brick processes
    gluster volume stop <volname>
    gluster volume start <volname>
    

Sunday, May 25, 2014

glusterfs-3.5.1beta released

Reposting the email to the Gluster Users and Developers mailinglists.
On Sat, 24 May, 2014 at 11:34:36PM -0700, Gluster Build System wrote:
> > SRC: http://bits.gluster.org/pub/gluster/glusterfs/src/glusterfs-3.5.1beta.tar.gz
This beta release is intended to verify the changes that should resolve the bugs listed below. We appreciate tests done by anyone. Please leave a comment in the respective bugreport with a short description of the success or failure. Visiting one of the bugreports is as easy as opening the bugzilla.redhat.com/$BUG URL, for the first in the list, this results in http://bugzilla.redhat.com/765202.

Bugs expected to be fixed (31 in total since 3.5.0):
#765202 - lgetxattr called with invalid keys on the bricks
#833586 - inodelk hang from marker_rename_release_newp_lock
#859581 - self-heal process can sometimes create directories instead of symlinks for the root gfid file in .glusterfs
#986429 - Backupvolfile server option should work internal to GlusterFS framework
#1039544 - [FEAT] "gluster volume heal info" should list the entries that actually required to be healed.
#1046624 - Unable to heal symbolic Links
#1046853 - AFR : For every file self-heal there are warning messages reported in glustershd.log file
#1063190 - [RHEV-RHS] Volume was not accessible after server side quorum was met
#1064096 - The old Python Translator code (not Glupy) should be removed
#1066996 - Using sanlock on a gluster mount with replica 3 (quorum-type auto) leads to a split-brain
#1071191 - [3.5.1] Sporadic SIGBUS with mmap() on a sparse file created with open(), seek(), write()
#1078061 - Need ability to heal mismatching user extended attributes without any changelogs
#1078365 - New xlators are linked as versioned .so files, creating <xlator>.so.0.0.0
#1086748 - Add documentation for the Feature: AFR CLI enhancements
#1086750 - Add documentation for the Feature: File Snapshots in GlusterFS
#1086752 - Add documentation for the Feature: On-Wire Compression/Decompression
#1086756 - Add documentation for the Feature: zerofill API for GlusterFS
#1086758 - Add documentation for the Feature: Changelog based parallel geo-replication
#1086760 - Add documentation for the Feature: Write Once Read Many (WORM) volume
#1086762 - Add documentation for the Feature: BD Xlator - Block Device translator
#1088848 - Spelling errors in rpc/rpc-transport/rdma/src/rdma.c
#1089054 - gf-error-codes.h is missing from source tarball
#1089470 - SMB: Crash on brick process during compile kernel.
#1089934 - list dir with more than N files results in Input/output error
#1091340 - Doc: Add glfs_fini known issue to release notes 3.5
#1091392 - glusterfs.spec.in: minor/nit changes to sync with Fedora spec
#1095775 - Add support in libgfapi to fetch volume info from glusterd.
#1095971 - Stopping/Starting a Gluster volume resets ownership
#1096040 - AFR : self-heal-daemon not clearing the change-logs of all the sources after self-heal
#1096425 - i/o error when one user tries to access RHS volume over NFS with 100+ GIDs
#1099878 - Need support for handle based Ops to fetch/modify extended attributes of a file

Before a final glusterfs-3.5.1 release is made, we hope to have all the blocker bugs fixed. There are currently 13 bugs marked that still need some work done:
#1081016 - glusterd needs xfsprogs and e2fsprogs packages
#1086743 - Add documentation for the Feature: RDMA-connection manager (RDMA-CM)
#1086749 - Add documentation for the Feature: Exposing Volume Capabilities
#1086751 - Add documentation for the Feature: gfid-access
#1086754 - Add documentation for the Feature: Quota Scalability
#1086755 - Add documentation for the Feature: readdir-ahead
#1086759 - Add documentation for the Feature: Improved block device translator
#1086766 - Add documentation for the Feature: Libgfapi
#1086774 - Add documentation for the Feature: Access Control List - Version 3 support for Gluster NFS
#1086781 - Add documentation for the Feature: Eager locking
#1086782 - Add documentation for the Feature: glusterfs and oVirt integration
#1086783 - Add documentation for the Feature: qemu 1.3 - libgfapi integration
#1095595 - Stick to IANA standard while allocating brick ports

A more detailed overview of the status of each of these bugs is here.

Tuesday, April 15, 2014

Configuring autofs for GlusterFS 3.5

GlusterFS 3.5 has not been released yet, but that should happen hopefully anytime soon (currently in beta). The RPM-packaging in this version has changed a little, and now offers a glusterfs-cli package. This package mainly contains the gluster commandline interface (and pulls in any dependencies).

On of the very useful things that is now made possible, is to list the available volumes on Gluster Storge Servers. This similar functionality is used by the /etc/auto.net script to list NFS-exports that are available for mounting. The auto.net script is by default enabled after installing and starting autofs:

# yum install autofs
# systemctl enable autofs.service
# systemctl start autofs.service

Checking, and mounting NFS-exports is made as easy as:

$ ls /net/nfs-server.example.net
archive  media  mock_cache  olpc
$ ls /net/nfs-server.example.net/mock_cache/fedora-rawhide-armhfp/
yum_cache

Making this functionality available for Gluster Volumes is simple, just follow these steps:

  1. install the gluster command

     # yum install glusterfs-cli
  2. save the file below as /etc/auto.glfs

     #!/bin/bash
     # /etc/auto.glfs -- based on /etc/auto.net
     #
     # This file must be executable to work! chmod 755!
     #
     # Look at what a host is exporting to determine what we can mount.
     # This is very simple, but it appears to work surprisingly well
     #
    
     key="$1"
    
     # add "nosymlink" here if you want to suppress symlinking local filesystems
     # add "nonstrict" to make it OK for some filesystems to not mount
     opts="-fstype=glusterfs,nodev,nosuid"
    
     for P in /usr/local/bin /usr/local/sbin /usr/bin /usr/sbin /bin /sbin
     do
         if [ -x ${P}/gluster ]
         then
                 GLUSTER_CLI=${P}/gluster
                 break
         fi
     done
    
     [ -x ${GLUSTER_CLI} ] || exit 1
    
     ${GLUSTER_CLI} --remote-host="${key}" volume list | \
         awk -v key="$key" -v opts="$opts" -- '
         BEGIN   { ORS=""; first=1 }
                 { if (first) { print opts; first=0 }; print " \\\n\t/" $1, key ":/" $1 }
         END     { if (!first) print "\n"; else exit 1 }' | \
         sed 's/#/\\#/g'
  3. make the script executable

     # chmod 0755 /etc/auto.glfs
  4. add an automount point to the autofs configuration

     # echo /glfs /etc/auto.glfs > /etc/auto.master.d/glfs.autofs
  5. reload the autofs configuration

     # systemctl reload autofs.service

After this, autofs should have created a new /glfs directory. The directory itself is empty, but a ls /glfs/gluster.example.net will show all the available volumes on the gluster.example.net server. These volumes can now be accessed through the autofs mountpoint. When the volumes are not used anymore, autofs will automatically unmount them after a timeout.

Sunday, February 23, 2014

Setting up a test-environment for Apache CloudStack and Gluster

This is an example of how to configure an environment where you can test CloudStack and Gluster. It uses two machines on the same LAN, one acts as a KVM hypervisor and the other as storage and management server. Because the (virtual) networking in the hypervisor is a little more complex than the networking on the management server, the hypervisor will be setup with an OpenVPN connection so that the local LAN is not affected with 'foreign' network traffic.

I am not a CloudStack specialist, so this configuration may not be optimal for real world usage. It is the intention to be able to test CloudStack and its Gluster integration in existing networks. The CloudStack installation and configuration done is suitable for testing and development systems, for production environments it is highly recommended to follow the CloudStack documentation instead.

 .----------------.                       .-------------------.
 |                |                       |                   |
 | KVM Hypervisor | <------- LAN -------> | Management Server |
 |                |    ^-- OpenVPN --^    |                   |
 '----------------'                       '-------------------'
agent.cloudstack.tld                      storage.cloudstack.tld

Both systems have one network interface with a static IP-address. In the LAN, other IP-addresses can not be used. This makes it difficult to access virtual machines, but that does not matter too much for this testing.

Both systems need a basic installation:

  • Red Hat Enterprise Linux 6.5 (CentOS 6.5 should work too)
  • Fedora EPEL enabled (howto install epel-release)
  • enable ssh access
  • SELinux in permissive mode (or disabled)
  • firewall enabled, but not restricting anything
  • Java 1.7 from the standard java-1.7.0-openjdk packages (not Java 1.6)

On the hypervisor, an additional (internal only) bridge needs to be setup. This bridge will be used for providing IP-addresses to the virtual machines. Each virtual machine seems to need at least 3 IP-addresses. This is a default in CloudStack. This example uses virtual networks 192.168.N.0/24, where N is 0 to 4.

Configuration for the main cloudbr0 device:

#file: /etc/sysconfig/network-scripts/ifcfg-cloudbr0
DEVICE=cloudbr0
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.0.1
NETMASK=255.255.255.0
NM_CONTROLLED=no

And the additional IP-addresses on the cloudbr0 bridge (create 4 files, replace N by 1, 2, 3 and 4):

#file: /etc/sysconfig/network-scripts/ifcfg-cloudbr0:N
DEVICE=cloudbr0:N
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.N.1
NETMASK=255.255.255.0
NM_CONTROLLED=no

Enable the new cloudbr0 bridge with all its IP-addresses:

# ifup cloudbr0
Any of the VMs that have a 192.168.*.* address, should be able to get to the real LAN, and ultimately also the internet. Enabling NAT for the internal virtual networks is the easiest:
# iptables -t nat -A POSTROUTING -o eth0 -s 192.168.0.0/24 -j MASQUERADE
# iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/24 -j MASQUERADE
# iptables -t nat -A POSTROUTING -o eth0 -s 192.168.2.0/24 -j MASQUERADE
# iptables -t nat -A POSTROUTING -o eth0 -s 192.168.3.0/24 -j MASQUERADE
# iptables -t nat -A POSTROUTING -o eth0 -s 192.168.4.0/24 -j MASQUERADE
# service iptables save

The hypervisor will need to be setup to act as a gateway to the virtual machines on the cloudbr0 bridge. In order to so do, a very basic OpenVPN service does the trick:

# yum install openvpn
# openvpn --genkey --secret /etc/openvpn/static.key
# cat << EOF > /etc/openvpn/server.conf
dev tun
ifconfig 192.168.200.1 192.168.200.2
secret static.key
EOF
# chkconfig openvpn on
# service openvpn start

On the management server, it is needed to configure OpenVPN as a client, so that routing to the virtual networks is possible:

# yum install openvpn
# cat << EOF > /etc/openvpn/client.conf
remote real-hostname-of-hypervisor.example.net
dev tun
ifconfig 192.168.200.2 192.168.200.1
secret static.key
EOF
# scp real-hostname-of-hypervisor.example.net:/etc/openvpn/static.key /etc/openvpn
# chkconfig opennvpn on
# service openvpn start

In /etc/hosts (on both the hypervisor and management server) the internal hostnames for the environment should be added:

#file: /etc/hosts
192.168.200.1 agent.cloudstack.tld
192.168.200.2 storage.cloudstack.tld

The hypervisor will also function as a DNS-server for the virtual machines. The easiest is to use dnsmasq which uses /etc/hosts and /etc/resolv.conf for resolving:

# yum install dnsmasq
# chkconfig dnsmasq on
# service dnsmasq start

The management server is also used as a Gluster Storage Server. Therefor it needs to have some Gluster packages:

# wget -O /etc/yum.repo.d/glusterfs-epel.repo \
http://download.gluster.org/pub/gluster/glusterfs/3.4/LATEST/RHEL/glusterfs-epel.repo
# yum install glusterfs-server
# vim /etc/glusterfs/glusterd.vol

# service glusterd restart

Create two volumes where CloudStack will store disk images. Before starting the volumes, apply the required settings too. Note that the hostname that holds the bricks should be resolvable by the hypervisor and the Secondary Storage VMs. This example does not show how to create volumes for production usage, do not create volumes like this for anything else than testing and scratch data.

# mkdir -p /bricks/primary/data
# mkdir -p /bricks/secondary/data
# gluster volume create primary storage.cloudstack.tld:/bricks/primary/data
# gluster volume set primary storage.owner-uid 36
# gluster volume set primary storage.owner-gid 36
# gluster volume set primary server.allow-insecure on
# gluster volume set primary nfs.disable true
# gluster volume start primary
# gluster volume create secondary storage.cloudstack.tld:/bricks/secondary/data
# gluster volume set secondary storage.owner-uid 36
# gluster volume set secondary storage.owner-gid 36
# gluster volume start secondary

When the preparation is all done, it is time to install Apache CloudStack. It is planned to have support for Gluster in CloudStack 4.4. At the moment not all required changes are included in the CloudStack git repository. Therefor, is is needed to build the RPM packages from the Gluster Forge repository where the development is happening. On a system running RHEL-6.5, checkout the sources and build the packages (this needs a standard CloudStack development environment, including java-1.7.0-openjdk-devel, Apache Maven and others):

$ git clone git://forge.gluster.org/cloudstack-gluster/cloudstack.git
$ cd cloudstack
$ git checkout -t -b wip/master/gluster
$ cd packaging/centos63
$ ./package.sh

In the end, these packages should have been build:

  • cloudstack-management-4.4.0-SNAPSHOT.el6.x86_64.rpm
  • cloudstack-common-4.4.0-SNAPSHOT.el6.x86_64.rpm
  • cloudstack-agent-4.4.0-SNAPSHOT.el6.x86_64.rpm
  • cloudstack-usage-4.4.0-SNAPSHOT.el6.x86_64.rpm
  • cloudstack-cli-4.4.0-SNAPSHOT.el6.x86_64.rpm
  • cloudstack-awsapi-4.4.0-SNAPSHOT.el6.x86_64.rpm

On the management server, install the following packages:

# yum localinstall cloudstack-management-4.4.0-SNAPSHOT.el6.x86_64.rpm \
cloudstack-common-4.4.0-SNAPSHOT.el6.x86_64.rpm \
cloudstack-awsapi-4.4.0-SNAPSHOT.el6.x86_64.rpm

Install and configure the database:

# yum install mysql-server
# chkconfig mysqld on
# service mysqld start
# vim /etc/cloudstack/management/classpath.conf

# cloudstack-setup-databases cloud:secret --deploy-as=root:

Install the systemvm templates:

# mount -t nfs storage.cloudstack.tld:/secondary /mnt
# /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \
-m /mnt \
-h kvm \
-u http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/lastSuccessfulBuild/artifact/tools/appliance/dist/systemvmtemplate-master-kvm.qcow2.bz2
# umount /mnt

The management server is now prepared, and the webui can get configured:

# cloudstack-setup-management

On the hypervisor, install the following additional packages:

# yum install qemu-kvm libvirt glusterfs-fuse
# yum localinstall cloudstack-common-4.4.0-SNAPSHOT.el6.x86_64.rpm \
cloudstack-agent-4.4.0-SNAPSHOT.el6.x86_64.rpm
# cloudstack-setup-agent

Make sure that in /etc/cloudstack/agent/agent.properties the right NICs are being used:

guest.network.device=cloudbr0
private.bridge.name=cloudbr0
private.network.device=cloudbr0
network.direct.device=cloudbr0
public.network.device=cloudbr0

Go to the CloudStack webinterface, this should be running on the management server: http://real-hostname-of-mgmt.example.net:8080/client The default username/password is: admin / password

It is easiest to skip the configuration wizard (not sure if that supports Gluster already). When the normal interface is shown, under 'Infrastructure' a new 'Zone' can get added. The Zone wizard will need the following input:

  • DNS 1: 192.168.0.1
  • Internal DNS 1: 192.168.0.1
  • Hypervisor: KVM

Under POD, use these options:

  • Reserved system gateway: 192.168.0.1
  • Reserved system netmask: 255.255.255.0
  • Start reserved system IP: 192.168.0.10
  • End reserved system IP: 192.168.0.250

Next the network config for the virtual machines:

  • Guest gateway: 192.168.1.1
  • Guest system netmask: 255.255.255.0
  • Guest start IP: 192.168.1.10
  • Guest end IP: 192.168.1.250

Primary storage:

  • Type: Gluster
  • Server: storage.cloudstack.tld
  • Volume: primary

Secondary Storage:

  • Type: nfs
  • Server: storage.cloudstack.tld
  • path: /secondary

Hypervisor agent:

  • hostname: agent.cloudstack.tld
  • username: root
  • password: password

If this all succeeded, the newly created Zone can get enabled. After a while, there should be two system VMs listed in the Infrastructure. It is possible to log in on these system VMs and check if all is working. To do so, log in over SSH on the hypervisor and connect to the VMs through libvirt:

# virsh list
 Id    Name                           State
----------------------------------------------------
 1     s-1-VM                         running
 2     v-2-VM                         running

# virsh console 1
Connected to domain s-1-VM
Escape character is ^]

Debian GNU/Linux 7 s-1-VM ttyS0

s-1-VM login: root
Password: password
...
root@s-1-VM:~# 

Log out from the shell, and press CTRL+] to disconnect from the console.

To verify that this VM indeed runs with the QEMU+libgfapi integration, check the log file that libvirt writes and confirm that there is a -drive with a glusterfs+tcp:// URL in /var/log/libvirt/qemu/s-1-VM.log:

... /usr/libexec/qemu-kvm -name s-1-VM ... -drive file=gluster+tcp://storage.cloudstack.tld:24007/primary/d691ac19-4ec1-47c1-b765-55f804b78bec,...