1 of 16

Software - Modules

Linux Cluster Institute - Intermediate

Alan Chapman�Systems Analyst - Software Specialist�RTO Research Computing�Arizona State University

2 of 16

Agenda

What are software modules and why do we use them

Usage of modules in HPC

Lua vs TCL modules

2

3 of 16

3

What Are Software Modules?

Software modules are scripts that configure the shell environment to access different versions of software packages and their dependencies. They modify environment variables such as PATH, LD_LIBRARY_PATH, CPATH, and others to ensure the correct software and its dependencies are used.

Modules allow users to load, unload, and switch between software packages easily without affecting the entire system. This is typically done using commands like module load, module unload, and module switch.

4 of 16

4

Why Use Software Modules in HPC?

Version Management: HPC systems often need multiple versions of the same software. Modules enable users to load the specific version they need without conflicts. For example, you might need Python 3.8 for one project and Python 3.9 for another.
Dependency Handling: Modules help manage software dependencies, ensuring that the correct libraries and tools are available for each application. This prevents the "dependency hell" that can occur when different applications require different versions of the same library.
User Environment Customization: Different users and projects might have unique requirements. Modules allow users to customize their environment to meet these needs without interfering with others on the same system.

Simplicity and Efficiency: Modules provide a simple way for users to manage their environment, reducing the complexity of configuring and maintaining software installations. This leads to more efficient use of the HPC resources.
Consistency: By using modules, system administrators can ensure that all users have a consistent and reproducible environment, which is crucial for scientific computing and collaboration.

5 of 16

5

Environment Modules

Environment Modules is a tool that helps manage and modify users' shell environments dynamically, allowing easy switching between different software versions and configurations in High-Performance Computing (HPC) systems. This tool is essential in HPC environments because it allows users to load, unload, and switch between different software packages and versions without manually adjusting environment variables like PATH, LD_LIBRARY_PATH, and others.

By using software modules, HPC environments can support diverse software needs and ensure that users have access to the right tools for their research and development tasks.

6 of 16

6

Types of Modules

Environment Modules:

These are the traditional modules written in the Tool Command Language (TCL).
Each module file is a script that modifies the environment variables as needed for software use.
TCL modules have been the standard for many years and are recognized for their simplicity and effectiveness in managing complex environments.

Lmod Modules:

A newer format for modules, using the Lua scripting language.
Lua modules are more expressive and allow more complex logic than TCL modules.
They are part of the Lmod system but can be used in the traditional Environment Modules system as well.

7 of 16

7

Example Workflow

Search for Modules: to find a module and not knowing the full name, a user can use avail to search for modules. To list all python modules do: module avail python
Loading a Module: To use a specific software, a user can load its module. For instance, to load Python 3.8, the user might run: module load python/3.8
Show loaded modules: If the user needs to know all modules currently loaded, they can do so easily: module list
Unloading a Module: When the user is done, they can unload the module to clean up their environment: module unload python/3.8
Switching Modules: If the user needs to switch from Python 3.8 to 3.9, they can do so easily: module switch python/3.8 python/3.9
Unload all Modules: If the user needs to unload all modules currently loaded, they can do so easily: module purge

8 of 16

8

Comparison and Benefits

Language: TCL, offering simplicity and ease of use.
Focus: Primarily adjusts environment paths; suitable for simpler or more static software environments.
Benefit: Ideal for straightforward module files where dependency management is either minimal or handled externally.

Language: Lua, which provides more flexibility and advanced features.
Focus: Offers dynamic dependency management with depends_on and environmental customization through functions like setenv.
Benefit: Superior for complex environments where software dependencies are intricately interwoven. Its hierarchical module management ensures that only compatible modules are loaded together, preventing conflicts.

Environment Modules

Lmod

9 of 16

9

Comparison of module file types

TCL

Lua

10 of 16

10

Charliecloud

11 of 16

11

Examples

local pkg = pathJoin("/apps",myModuleName(),myModuleVersion())

local bin = pathJoin(pkg,"bin"))

prepend_path("PATH",bin)

whatis("Name: ", myModuleName())

whatis("Version: ", myModuleVersion())

whatis("Description: ", "Git is a fast distributive version control system")

Introspection

12 of 16

12

Examples

local fn = myFileName() -- 1

local full = myModuleFullName() -- 2

local loc = fn:find(full,1,true)-2 -- 3

local mdir = fn:sub(1,loc) -- 4

local appsDir = mdir:gsub("(.*)/","%1") -- 5

local pkg = pathJoin(appsDir, full) -- 6

Relative Paths

13 of 16

13

Installation

Office Instructions - https://lmod.readthedocs.io/en/latest/030_installing.html
Simple instructions

# dnf config-manager --set-enabled powertools

# dnf install Lmod

# . /usr/share/lmod/lmod/init/profile

(The last line is only so you don’t have to reboot to get the module command to work.)

14 of 16

14

Resources

Documentation: https://lmod.readthedocs.io/en/latest/
GitHub: https://github.com/TACC/Lmod
SourceForge: https://lmod.sf.net
TACC Homepage: https://www.tacc.utexas.edu/research-development/tacc-projects/lmod
Lmod Test Suite: https://github.com/rtmclay/Lmod_test_suite
Join Lmod Mailing list: https://sourceforge.net/projects/lmod/lists/lmod-users
Monthly Zoom call: https://github.com/TACC/Lmod/wiki

15 of 16

15

Questions / Discussion