Slurm-Mail

Author: Neil Munday (neil at mundayweb.com)

Repository: https://github.com/neilmunday/slurm-mail

Contents

1. Introduction
2. Requirements
3. RPM Installation
4. Source Installation
5. Configuration
6. Upgrading from version 2.x to 3.x
7. SMTP Settings
8. Customising E-mails
9. Validating E-mails
10. Including Job Output in E-mails
11. Contributors

Introduction

E-mail notifications from Slurm are rather brief and all the information is contained in the subject of the e-mail - the body is empty.

Slurm-Mail aims to address this by providing a drop in replacement for Slurm's e-mails to give users much more information about their jobs via HTML e-mails which contain the following information:

• Start/End
• Job name
• Partition
• Work dir
• Elapsed time
• Exit code
• Std out file path
• Std err file path
• No. of nodes used
• Node list
• Requested memory per node
• Maximum memory usage per node
• CPU efficiency
• Wallclock
• Wallclock accuracy

E-mails can be easily customised to your needs using the provided templates (see below).

You can also opt to include a number of lines from the end of the job's output files in the job completion e-mails (see below).

Requirements

• Python 3
• Slurm 20, 21 or 22
• A working e-mail server

RPM Installation

Note: pre-built RPMs for RHEL7, RHEL8, SLES 15 and OpenSUSE 15 compatible operating systems are available at Slurm-Mail releases.

To create a Slurm-Mail RPM for your OS download the Slurm-Mail tar archive and then run:

rpmbuild -tb slurm-mail-3.4.tar.gz

The Slurm-Mail RPM will install to /opt/slurm-mail and will also create the required cron job for Slurm-Mail to function as well as providing a logrotate configuration for handling Slurm-Mail's log files.

Take note of where rpmbuild created the generated RPM and then install with your package manager (e.g. dnf, yum, zypper).

Source Installation

Download the latest release of Slurm-Mail and unpack it to a directory of your choosing on the server(s) running the Slurm controller daemon slurmctld, e.g. /opt/slurm-mail

tar xfz slurm-mail-3.4.tar.gz

Create the spool and log directories for Slurm-Mail on your Slurm controller(s):

mkdir -p /var/spool/slurm-mail /var/log/slurm-mail
chown slurm. /var/spool/slurm-mail /var/log/slurm-mail
chmod 0700 /var/spool/slurm-mail /var/log/slurm-mail

Create a cron job to run slurm-send-mail.py periodically to send HTML e-mails to users. As Slurm-Mail uses sacct to gather additional job information and may perform additional processing, the sending of e-mails was split into a separate application to prevent adding any overhead to slurmctld.

Example cron job, e.g./etc/cron.d/slurm-mail:

*    *    *    *    *    root    /opt/slurm-mail/bin/slurm-send-mail.py

Set-up logrotate:

cp /opt/slurm-mail/logrotate.d/slurm-mail /etc/logrotate.d/

Configuration

Edit /opt/slurm-mail/conf.d/slurm-mail.conf to suit your needs. For example, check that the location of sacct is correct. If you are installing from source check that the log and spool directories are set to your desired values.

Change the value of MailProg in your slurm.conf file to /opt/slurm-mail/bin/slurm-spool-mail.py. By default the Slurm config file will be located at /etc/slurm/slurm.conf.

Restart slurmctld:

systemctl restart slurmctld

Slurm-Mail will now log e-mail requests from Slurm users to the Slurm-Mail spool directory.

Upgrading from version 2.x to 3.x

Version 3.0 onwards uses a new location for the e-mail templates. Therefore for versions prior to this, please run the following commands:

cd /opt/slurm-mail/conf.d
mkdir templates
mv ./*.tpl templates/

After moving the templates please merge any of the changes from the latest 3.x release with your local copies.

SMTP Settings

By default Slurm-Mail will send e-mails to a mail server running on the same host as Slurm-Mail is installed on, i.e. localhost.

You can edit the smtp configuration options in conf.d/slurm-mail.conf. For example, to send e-mails via Gmail's SMTP server set the following settings:

smtpServer = smtp.gmail.com
smtpPort = 587
smtpUseTls = yes
smtpUserName = [email protected]
smtpPassword = your_gmail_password

NOTE: As this file will contain your Gmail password make sure that it has the correct owner, group and file access permissions.

For SMTP servers that use SSL rather than starttls please set smtpUseSsl = yes.

Customising E-mails

Slurm-Mail uses Python's string.Template class to create the e-mails it sends. Under Slurm-Mail's conf.d/templates directory you will find the following files that you can edit to customise e-mails to your needs.

You can adjust the font style, size, colours etc. by editing the Cascading Style Sheet (CSS) file conf.d/style.css used for generating the e-mails.

To change the date/time format used for job start and end times in the e-mails, change the datetimeFormat configuration option in conf.d/slurm-mail.conf. The format string used is the same as Python's datetime.strftime function.

To change the subject of the e-mails, change the emailSubject configuration option in conf.d/slurm-mail.conf. You use the following place holders in the string:

Validating E-mails

By default Slurm-Mail will not perform any checks on the destination e-mail address (i.e the value supplied to sbatch via --mail-user). If you would like Slurm-Mail to only send e-mails for jobs that correspond to a valid e-mail address (e.g. [email protected]) then you can set the validateEmail option in conf.d/slurm-mail.conf to true. E-mail addresses that failed this check will be logged in /var/log/slurm-mail/slurm-send-mail.log as an error.

Including Job Output in E-mails

In conf.d/slurm-mail.conf you can set the includeOutputLines to the number of lines to include from the end of each job's standard out and standard error files.

Notes:

• if the user has decided to use the same file for both standard output and standard error then there will be only one section of job output in the job completion e-mails.
• Job output can only be included if the process that is running slurm-send-mail.py is able to read the user's output files.

Job Arrays

Slurm-Mail will honour the behaviour of --mail-type option of sbatch for job arrays. If a user specifies --mail-type=ARRAY_TASKS then Slurm-Mail will send notification e-mails for all jobs in the array. If you want to limit the number of e-mails that will be sent in this scenario then change the arrayMaxNotifications parameter in slurm-mail.conf to a value greater than zero.

Contributors

Thank you to the following people who have contributed code improvements, features and aided the development of Slurm-Mail:

• Dan Barker (@danbarke): https://www.github.com/danbarke
• David Murray (@dajamu): https://www.github.com/dajamu
• drhey (@drhey): https://www.github.com/drhey
• hakasapl (@hakasapl): https://www.github.com/hakasapl
• Mehran Khodabandeh (@mkhodabandeh): https://www.github.com/mkhodabandeh
• sdx23 (@sdx23): https://www.github.com/sdx23
• Woody Chang (@jitkang): https://github.com/jitkang