PVFS stands for Parallel Virtual File System. This filesystem behaves as a RAID stripe, using the disk space of multiple machines connected on a high speed network. With a fast enough network, its performance far exceeds that of a local scratch disk. PVFS2 is an experimental branch of PVFS that supports the Linux 2.6 kernel, among other improvements.

Please note that PVFS has no redundancy built-in, so if a node fails, the data that was kept in PVFS on that node will be unavailable. Also note that PVFS is designed for many nodes on a gigabit or faster network: If you configure it with two nodes on an 100Mb/s network you will see performance worse than writing to the drives locally.

Prerequisites

The following packages are required for compiling PVFS2:

db-devel
kernel-source

If you wish to use the Karma monitoring GUI, you must have the following:

gtk2-devel
pkgconfig

Installing PVFS

Installing PVFS2 userland

You can download PVFS from ftp://ftp.parl.clemson.edu/pub/pvfs2 . For this document, we will only cover 0.7 (current release) with the kernel module support. Installation documentation for PVFS2 is available at http://www.pvfs.org/pvfs2/pvfs2-quickstart.html

If you wish to use the Karma GUI, you must do this first so that the configure script finds gtk-config.

export PATH=/opt/gnome/bin:$PATH

Now for the actual configuration. The user you run this as must have write access to the kernel source tree, hence the sudo.

sudo ./configure --with-kernel=/usr/src/linux-`uname -r | cut -d- -f1,2` --enable-mmap-racache
sudo make
sudo make install

PVFS by default installs everything in /usr/local.

Installing PVFS2 kernel module

First, you must prepare the kernel source directory:

cd /usr/src/linux-`uname -r | cut -d- -f1,2`
sudo make cloneconfig
sudo make modules

Then go back to the pvfs2 source directory, compile, and install the module:

sudo make kmod
sudo make kmod_install

Getting PVFS running

Configuring PVFS

sudo pvfs2-genconfig /etc/pvfs2-fs.conf /etc/pvfs2-server.conf

Here are the questions it will ask you, and how you would normally answer it:

Protocol: default
Port: default
I/O Servers: A list of all of the hosts that will be providing disk space
Metadata Server: Your most reliable host.
Logfile: /var/log/pvfs2.log
Data store directory: /pvfs-storage

The resulting configuration will be saved in /etc/pvfs*. You will want to make sure this is in sync with the rest of the cluster.

Initializing the PVFS storage

The data store directory is where you should mount a hard disk partition that will be shared out for use with PVFS2. I just create a reiserfs partition there with YaST, and call it /pvfs-storage. Once this directory has been created, you must initialize it the following on each I/O node:

sudo /usr/local/sbin/pvfs2-server /etc/pvfs2-fs.conf /etc/pvfs2-server.conf-`hostname` -f

Setting up pvfs2tab

You will need to configure a pvfs2tab

tcp://metahost:3334/pvfs2-fs /pvfs2 pvfs2 default 0 0

Starting PVFS

sudo /usr/local/sbin/pvfs2-server /etc/pvfs2-fs.conf /etc/pvfs2-server.conf-`hostname`

Testing PVFS2

sudo pvfs2-ping -m /pvfs2

Make sure you do not see any freaky errors.

Using PVFS2 as a mounted partition

Starting the PVFS2 kernel client

Load the PVFS kernel module:

sudo modprobe pvfs2

Then start the PVFS client:

sudo /usr/local/sbin/pvfs2-client -p /usr/local/sbin/pvfs2-client-core

Mounting the filesystem

Wait a few seconds for the client to initialize, make sure you have a directory for pvfs to use, and then run the mount command:

sudo mkdir /pvfs
sudo mount -t pvfs2 tcp://metahost:3334/pvfs2-fs /pvfs

Troubleshooting

line 542: Error: malformed tcp address.

This means there is no port configured in /etc/pvfs2-server.conf-`hostname`. I'm pretty sure this is a bug in pvfs2-genconfig. The syntax should look like this:

HostID "tcp://beigetwin2:3334"

Invalid handle range for host tcp://host:3334 (alias (null)) specified in file system pvfs2-fs

You have a broken version of genconfig. Try this patch:

--- /usr/local/bin/pvfs2-genconfig~     2004-10-13 15:38:58.764938008 -0500
+++ /usr/local/bin/pvfs2-genconfig      2004-10-13 15:39:03.377236832 -0500
@@ -538,11 +538,11 @@
 $type = get_protocol();
 foreach (split(',', $type)) {
     if ($_ eq "tcp") {
-       $port{$_} = tcp_get_port();
+       $port{'tcp'} = tcp_get_port();
     } elsif ($_ eq "gm") {
-       $port{$_} = gm_get_port();
+       $port{'gm'} = gm_get_port();
     } elsif ($_ eq "ib") {
-       $port{$_} = ib_get_port();
+       $port{'ib'} = ib_get_port();
     } else {
        print "Sorry.  At this time, only the tcp,gm, and ib

mount: wrong fs type, bad option, bad superblock on tcp://host:3334/pvfs2-fs

You probably forgot to start the PVFS2 client before trying to mount the filesystem.

I've mounted PVFS, but df hangs when trying to display the information

I would run through the testing section just to make sure everything is set correct. Maybe you need pvfs2tab?

Failure: not all servers accepted fsid xxx

You may want to make sure that all of your PVFS2 storage volumes were prepared with the identical pvfs configuration files. Try removing all of the files under /pvfs-storage and reinitializing them.

mount hangs when I try to mount a PVFS volume

Make sure the client is running, and that you've used the correct hostname and port in the mount command.

Reference

http://wiki.chem.indiana.edu/Software/InstallingPVFSOnASUSE91Cluster

Rights

rw-rw-r--

tstrombe

ITG

Configuring Exceed for X11 Tunnelling

Software Index