Today I was reviewing the Moab documentation for an upcoming training and I ran across several feature gems that I thought were worth calling out. I’ll call them some of the “not-so-secret ingredients” that makes Moab great.
Scheduling with Partitions
Moab uses partitions to logical divide available resources in your environment. This allows you to separate geographically or by hardware configuration. For example, because a given job can only use resources within a single partition, you may want to create a partition to group nodes on the same local switch so that you can guarantee the fastest interprocess communication speed. Another cool benefit of partitions is the ability to set partition-specific policies, limits, priorities and scheduling algorithms, though this is rarely necessary.
In an effort to conserve power, Moab can automatically turn off idle nodes until they are needed again. This can be a significant savings if you’re not maxing your capacity all the time. To save on time, Moab can keep a pool of nodes on standby so that there is no delay when additional resources are needed. Moab comes with reference scripts for IPMI, but can be configured to work with iLO, DRAC and xCAT as well.
Using configured targets, you can use historical resource utilization as a factor for job priority. In essence, this would prevent a user with relatively infrequent workload to get buried in the backlog of jobs from much busier users. This kind of thinking extends to groups, accounts and other groupings as well. There are a number of configuration settings for fairshare that allow you to tweak performance to your needs, such as caps, evaluation timeframe, and consumption metrics.
You can associate multiple reservations through reservation groups. This allows each reservation to share the same variables. These variables can then be used in job triggers to automate tasks like email notification, script execution and internal Moab actions, such as creating another reservation. Of course, you can always override the inheritance by setting a variable locally on an individual reservation in the group.