Next Previous Contents

4. System Maintainance

4.1 Planing for the Future

System maintainance is a very important part of cluster design that most books leave out. When you build your cluster, as a rule of thumb, don't do anything "by hand" to any of the nodes when setting them up. Usually, following this rule will cause you to do things in a formal scripted or packaged manor, thus ensuring easy maintainance.

4.2 Always Use Package Managers

As I said before, I recommend installing every non-distribution package via encap, including scripts you write. This will make it easy to handle multiple versions of programs, and allow you to rapidly "roll back" if something goes wrong.

In a ArchType I system, you just install the RPM or encap, and then wait a day for rsync to take care of things. In an ArchType II system like Illigal, I recommend putting the RPM in somewhere exported to all nodes of the cluster, like /usr/src/redhat/RPMS/i386/, and then running "allnodes rpm -Uvh /usr/src/redhat/RPMS/i386/rpmname.rpm". Technically you are reinstaling most of the same files over and over again, but you do need to update the rpmdb on each node, and you also want to make sure that files in /lib and /bin get updated too.

4.3 Adding a Node

4.4 Other Notes

On the Illigal cluster, I found it handy to keep the kernel config files of the nodes in /usr/src/MachineType.kconfig. This makes upgrading the kernels easier, since you just copy the appropriate config file to .config, and run make oldconfig.

In addition, I also wrote a script called allexports which basically is a simple while loop that iterates on each of the NFS export directories sets a vol variable to each of the export directories. So to do something like copy in all the kernel modules to each directory, you would to allexports cp -a /lib/modules/2.2.17 \$vol/lib/modules/. You have to escape that $, or the shell will sub in $vol before the command runs. It also comes in handy if you need to edit a file on each of the nodes, or even launch another script (in which case you don't have to escape the $ from inside the script). $i is exported as well, and is simply the number 2->15.

Next Previous Contents