Installing Torque/Maui and parallel software¶

Installing torque-4.2.6¶

The following are some notes to have Torque 4.2.6 installed along with the MAUI scheduler.

Here is the official resource for the Torque resource manager:

http://www.adaptivecomputing.com/
http://www.adaptivecomputing.com/downloading/?file=/torque/torque-4.2.6.tar.gz

Export some handy variables:

# TORQUE_VER=4.2.6
# CLUSTER_NFS_SHARE=/opt

Install pam and xml2 libraries, needed to compile Torque:

# yum -y install pam-devel libxml2-devel

Now enter the directory and configure the package. Then build it:

# cd ./torque-$TORQUE_VER

> ./configure --prefix=$CLUSTER_NFS_SHARE/torque/$TORQUE_VER/ --enable-server --enable-mom --enable-clients --disable-gui --with-rcp=scp --with-default-server=master.nfs --with-pam=/lib64/security/

> make -j 4

> script ___install.log -c 'make install'

The value assigned to –with-default-server may change depending on the network configuration, “master.nfs” is the fully qualified domain name (FQDN) of our pbs/torque server.

Now modify the original init script for the server pbs_server and the pbs_mom (the lines commented out were needed in the past, latest version of torque automatically fix the paths):

> #sed "/^PBS_DAEMON=.*$/s||PBS_DAEMON=$CLUSTER_NFS_SHARE/torque/$TORQUE_VER/sbin/pbs_server|" -i contrib/init.d/pbs_server
> cp -va contrib/init.d/pbs_server /etc/init.d/
> chmod +x /etc/init.d/pbs_server

> #sed "/^PBS_DAEMON=.*$/s||PBS_DAEMON=$CLUSTER_NFS_SHARE/torque/$TORQUE_VER/sbin/pbs_mom|" -i contrib/init.d/pbs_mom
> cp -va contrib/init.d/pbs_mom $CLUSTER_NFS_SHARE/torque/$TORQUE_VER/pbs_mom.init
> chmod +x $CLUSTER_NFS_SHARE/torque/$TORQUE_VER/pbs_mom.init
> cp -va $CLUSTER_NFS_SHARE/torque/$TORQUE_VER/pbs_mom.init /etc/init.d/pbs_mom
> chmod +x /etc/init.d/pbs_mom

> cp -va contrib/init.d/trqauthd /etc/init.d/
> chmod +x /etc/init.d/trqauthd

Configure now the pam based authentication on the computing nodes (if required). This allows a user with a running job to login on the nodes where his/her job is actually running:

On computing nodes:
> test -f /lib64/security/pam_pbssimpleauth.so && sed '0,/^account/b ins;b;: ins /account/iaccount    sufficient   pam_pbssimpleauth.so' -i /etc/pam.d/sshd

Since torque and all its binaries have been installed in a non-standard location, the shell doesn’t know yet where to find these executables. Bash uses an environment variable for this purpose: $PATH. We have to append (or prepend in some cases) the ‘bin’ and ‘sbin’ directories for torque to this variable:

> echo $PATH
> export PATH=$PATH:$CLUSTER_NFS_SHARE/torque/$TORQUE_VER/bin:$CLUSTER_NFS_SHARE/torque/$TORQUE_VER/sbin
> echo $PATH

The main difference in appending or prepending a directory is that when a command name is written on the command line, the first executable found in the PATH list of directories is the one that will be executed. When there are no conflicts in binary names, the order is irrilevant, when more versions of the same software are installed, prepending is needed to select which version to use. This is also what the ‘module’ package does.

The commands above fix only the running shell, a new terminale/shell/login won’t be aware of these modifications. In order to make this permamnent, a new file must be created:

> echo "export PATH=\$PATH:$CLUSTER_NFS_SHARE/torque/$TORQUE_VER/bin:$CLUSTER_NFS_SHARE/torque/$TORQUE_VER/sbin" > /etc/profile.d/torque.sh

Please note the backslash (‘’) in front of $PATH. In this case we don’t want the current value of PATH to be used, we are just telling the script to use that variable at the time it will be used. The content of the file will be infact:

export PATH=$PATH:/opt/torque/4.2.6/bin:/opt/torque/4.2.6/sbin

At this point, before starting the services, we need to create at least the ‘nodes’ file, on the pbs server, which contains the list of hosts and resources available to the queue system:

> echo 'sgbd.acmad np=6 db' >> /var/spool/torque/server_priv/nodes
> echo 'model.acmad np=12 intel compute' >> /var/spool/torque/server_priv/nodes

Also Follow these steps on each compute node in the cluster: Edit /var/spool/torque/mom_priv/config to contain some basic info identifying the server:

$pbsserver      sgbd.acmad    # note: this is the hostname of the headnode
$logevent       255           # bitmap of which events to log

In this example, we’ll have an host named sgbd.acmad, with 2 processors that can be assigned by the queue system, with a feature named ‘db’. This last one is just a label (not a keyword) that can be used when submitting jobs to ask for nodes with specific characteristics or features. On a cluster, there will be one line for each node, and the features list allow to define pools of nodes with different characteristics (like the architecture, availability of high speed networks or specific hardware like gpus, parallel filesystem, and so forth). The second line tells the resource manager that there is a second node, with 12 cores, and a couple of features.

Please note that torque is especially picky about hostnames, sometimes using the FQDN or short hostname is not transparent and can lead to some issues. In case something doesn’t work, check whether the hostname is set correctly (command ‘hostname’ and HOSTNAME variable in /etc/sysconfig/network), if ‘hostname -f’ reports correctly the name+domain, if the file /etc/hosts (or the dns) is able to resolve the hosts/IP of your network (at least the hosts used by torque). The files that may give you some hints about the problems are:

on the machine running pbs_server (YYYYMMDD is the current date)

        /var/spool/torque/server_logs/YYYYMMDD

on the clients where pbs_mom is running

        /var/spool/torque/mom_logs/YYYYMMDD

on both
        /var/log/messages

The configuration, currently running on the server, reports these values:

[root@sgbd ~]# hostname
sgbd.acmad
[root@sgbd ~]> hostname -f
sgbd.acmad
[root@sgbd ~]> grep HOSTNAME /etc/sysconfig/network
HOSTNAME=sgbd.acmad
[root@sgbd ~]> cat /var/spool/torque/server_name
sgbd.acmad
[root@sgbd ~]> cat /var/spool/torque/server_priv/nodes
sgbd.acmad np=12
[root@sgbd ~]> cat /var/spool/torque/mom_priv/config
$clienthost sgbd
[root@sgbd ~]> cat /etc/hosts
127.0.0.1       localhost localhost.localdomain localhost4 localhost4.localdomain4
::1             localhost6 localhost6.localdomain6
192.168.88.20:8080   sgbd.acmad      sgbd
192.168.88.21   model.acmad     model
192.168.88.24   sp-sgbd.acmad   sp-sgbd
192.168.88.25   sp-model.acmad  sp-model

As you can see, “somewhere” (mom config) the short host name can be used, “somewhere else” (server name, nodes, and later will see qmgr configuration) the FQDN is strictly needed.

In order to start the services:

> service trqauthd start
> service pbs_server start
> service pbs_mom start

In order to configure them to start automatically at the next boot:

> for service in trqauthd pbs_server pbs_mom ; do chkconfig --add $service ; chkconfig $service on ; done

The following command can be used to verify whether the services are already configured to start:

> chkconfig --list trqauthd
> chkconfig --list pbs_server
> chkconfig --list pbs_mom

In case, for instance, that pbs_mom should not start automatically the –del option can be used (and –list to verify):

> chkconfig --list pbs_mom
> chkconfig --del pbs_mom
> chkconfig --list pbs_mom

Note that the service trqauthd is needed for both the pbs_server and the pbs_mom services (hence, on both the server and the computing node).

Verify:

> qmgr -c ‘print server’ > qstat -Q > pbsnodes -a

MAUI¶

This part will install the MAUI scheduler version 3.3.1.

The following are the official resources (registration is needed in order to download the package):

http://www.adaptivecomputing.com/
http://www.adaptivecomputing.com/products/open-source/maui/
http://www.clusterresources.com/product/maui/

These are some notes to install on CentOS operating system:

> yum -y install pcre pcre-devel

on centos 6.*
        configure will hang on mktime, solution:
                > yum -y install autoconf
                on C6.0 run "autoconf" and  then "configure" again
                on C6.2 up to C6.4 (to date) autoconf breaks the Makefile, this is a one-line fix:
                        > sed '/for (j = 1; 0 < j; j \*= 2)/s/j \*= 2/j <<= 1/' -i configure

make will complain about missing "-lpcreposix -lpcre", one-line fix:
                > sed 's/"//g' -i ./include/Makefile.inc.pcre.in

> ./configure --prefix=$CLUSTER_NFS_SHARE/maui/3.3.1/ --with-spooldir=/var/spool/maui --with-pbs=$CLUSTER_NFS_SHARE/torque/$TORQUE_VER/ --with-pcre
> make -j 4
> script ___install.log -c 'make install'

Please note that on the configure line, we are telling maui where to find pbs/torque. Since they’ll need to interact, maui need to be compiled against the running torque headers and libraries (be sure that $CLUSTER_NFS_SHARE and $TORQUE_VER are still set accordingly to the torque installation).

Copy the environment file:

> cp -va etc/maui.{c,}sh /etc/profile.d

Copy the startup script (fixing the path):

> sed "/^MAUI_PREFIX=/s|/opt/maui|$CLUSTER_NFS_SHARE/maui/3.3.1|" -i etc/maui.d
> cp -va etc/maui.d /etc/init.d/maui
> chmod +x /etc/init.d/maui

Configure maui to start at the next reboot:

> chkconfig --add maui
> chkconfig maui on

Startup the service:

> service maui start

Verify:

> showbf
> showq

Must haves¶

Make sure that /etc/hosts on all of the boxes in the cluster contains the hostnames of every PC in the cluster.

Be sure to open TCP for all machines using TORQUE. The pbs_server (server) and pbs_mom (client) by default use TCP and UDP ports 15001-15004. pbs_mom (client) also uses UDP ports 1023 and below if privileged ports are configured (the default).

An NFS share either on the server or another machine is highly recommended to simplify the process of sharing common build disk space.

Torque & Maui list of command¶

A short presentation on useful commands to use with the Torque resource manager and the MAUI scheduler.

torque:

Server configuration (superuser; pbs server):
> qmgr

Status of the queue system and running/queued jobs (anyone; server or client):
> qstat

Submit a job (users; server and, if server configured accordingly, client):
> qsub

Remove a job from the queue (superuser, user for his/her own jobs; anywhere):
> qdel

Force kill of a job (superuser, user for his/her own jobs; anywhere):
> qsig

maui (only the maui server, unless it’s configured to accept remote queries):

Server configuration (anyone)
> showconfig

Resource availability (anyone)
> showbf

Status of the queues (anyone)
> showq

Status of the queues (anyone)
> showstate

Set a static reservation (superuser)
> setres

See running reservations (anyone)
> showres

Release/remove a static reservation (superuser)
> releaseres

A few command to submit jobs¶

Resouce list (-l):

> qsub -l nodes=1:ppn=1 -l walltime=00:01:00 -q default

An example of a batch script (assuming file name /home/user/myfirstjob):

#!/bin/bash
#PBS -N myjob  #Declares a name for the job
#PBS -l nodes=1:ppn=1:db
#PBS -l walltime=00:01:00 #Specifies the resources that are required by the job and establishes a limit to the amount of resources that can be consumed.
#PBS -o myjob.stdout.log -e myjob.stderr.log   #Directs the standard output produced by the request to the stated file path
#PBS -q default   #Defines the destination queue of the job. 
hostname -f

Submitting a batch job:

qsub /home/user/myfirstjob

A Simple Script¶

edit a file name sleep.pbs and paste the following:

#!/bin/sh
 
for i in {1..120} ; do
       echo $i
       sleep 1
done

echo Master process running on `hostname`
echo Directory is `pwd`
echo PBS has allocated the following nodes:
echo `cat $PBS_NBODEFILE`