There’s some debate still about these 20x developers out there, complete geniuses that gave rise to the “rock-star” moniker that every company seemed to advertise for. I’ve remained a touch skeptical–not that there aren’t amazing software engineers, I know several — but I’m skeptical because we still don’t have a really good, accessible ways to measure human contribution to software. Lines of code, commits and story point velocity are readily available. Regressions, debugging and runtime numbers can be gathered as well, but we know that all this still doesn’t paint the whole picture. Also add in the real-world impact that other factors outside of development have on performance as well: good product management, architecture and QA among them, but also deadlines, poor communication and bad management. With all of these variables, it’s going to be pretty tough to accurate gauge order-of-magnitude differences between programmers in the real world. And frankly, I don’t know that you need to because in your gut you already know.

“The differences arising from individuals in any given study will drown out any differences you might want to attribute to a change in methodology.” — Steve McConnell

Back in 2011 Steve McConnell responded to criticisms of his 10x programmer claim thoughtfully by pointing out what he was seeing in the studies that have already been done, most not even focused on individual contribution. What he found was that “the differences arising from individuals in any given study will drown out any differences you might want to attribute to a change in methodology.” In short, it nearly totally depends on who’s on your team. However, over-emphasis on this leads to ignoring methodology and tooling altogether and focusing instead on individual development and hiring, which would be short-sighted. My point here is that on the balance, the latter should constitute a more significant portion of team building efforts.

The trouble arises when you commoditize development. While there are many software tasks that require no planning or design, there is much more that requires careful thought. As Yevgeniy Brikman said in support of the 10x developer, “It’s not about writing more code; it’s about writing the right code. You become a 10x programmer not by doing an order of magnitude more work, but by making better decisions an order of magnitude more often.”

“It’s not about writing more code; it’s about writing the right code. You become a 10x programmer not by doing an order of magnitude more work, but by making better decisions an order of magnitude more often.” — Yevgeniy Brikman

This is why I prefer the title software engineer. Writing good applications is really about a mind well-suited for building: choices, compromises and the collective wisdom of personal experience and industry best practices. Don’t commoditize development. It’s a highly-talented and highly-compensated profession. It’s not about nuturing a team, it’s about providing the humans working around you with the best possible resources to be successful, innovative and frankly, fun to be around. Yes, tools matter, and methodology matters–they matter a lot and they can have a big impact on the quality and speed of your work, but investing in these while sacrificing investment in the professional development of the individuals on the team… well… that’s as bad as it sounds.

Today I was reviewing the Moab documentation for an upcoming training and I ran across several feature gems that I thought were worth calling out. I’ll call them some of the “not-so-secret ingredients” that makes Moab great.

Scheduling with Partitions
Moab uses partitions to logical divide available resources in your environment. This allows you to separate geographically or by hardware configuration. For example, because a given job can only use resources within a single partition, you may want to create a partition to group nodes on the same local switch so that you can guarantee the fastest interprocess communication speed. Another cool benefit of partitions is the ability to set partition-specific policies, limits, priorities and scheduling algorithms, though this is rarely necessary.

Green Computing
In an effort to conserve power, Moab can automatically turn off idle nodes until they are needed again. This can be a significant savings if you’re not maxing your capacity all the time. To save on time, Moab can keep a pool of nodes on standby so that there is no delay when additional resources are needed. Moab comes with reference scripts for IPMI, but can be configured to work with iLO, DRAC and xCAT as well.

Fairshare
Using configured targets, you can use historical resource utilization as a factor for job priority. In essence, this would prevent a user with relatively infrequent workload to get buried in the backlog of jobs from much busier users. This kind of thinking extends to groups, accounts and other groupings as well. There are a number of configuration settings for fairshare that allow you to tweak performance to your needs, such as caps, evaluation timeframe, and consumption metrics.

Reservation Groups
You can associate multiple reservations through reservation groups. This allows each reservation to share the same variables. These variables can then be used in job triggers to automate tasks like email notification, script execution and internal Moab actions, such as creating another reservation. Of course, you can always override the inheritance by setting a variable locally on an individual reservation in the group.