Build your own container image with Enroot

Enroot is a container runtime. It builds and runs the container images that Pyxis uses in your SLURM jobs.

If the pre-built images don't have what you need, build your own with Enroot.

Import a base image¶

Pull an image from a registry. Enroot converts it into a .sqsh file:

enroot import docker://nvcr.io/nvidia/pytorch:26.07-py3

This creates the .sqsh file in your current directory. A .sqsh file is a single, compressed, read-only image of the container's filesystem. Enroot uses this format to store and run container images.

Open a writable session¶

1. Create a container from the image:

   enroot create --name my-container nvidia+pytorch+26.07-py3.sqsh
   

2. Start it with a writable filesystem:

   NVIDIA_VISIBLE_DEVICES=void enroot start --rw my-container
   

   NVIDIA_VISIBLE_DEVICES=void starts the container even though the GPU isn't available in a login session. You'll still see warnings and errors about GPU functionality — ignore them:

   Expected warnings and errors

   [10-nvidia-mps.sh] WARNING: MPS control socket not found at /tmp/nvidia-mps/control
   [10-nvidia-mps.sh] Ensure nvidia-cuda-mps.service is running on the host
   ...
   ERROR: The NVIDIA Driver is present, but CUDA failed to initialize. GPU functionality will not be available.
   [[ Unable to initialize CUDA driver (error ???) ]]
   Failed to detect NVIDIA driver version.
   

   You'll see a prompt like this once you're inside the container:

   <username>@blazar:/workspace$
   

3. Make your changes inside the session. For example, install a package:

   pip install plotly
   exit
   

Save your changes¶

Export the container to a .sqsh file:

enroot export --output custom.sqsh my-container

Check that the custom image preserved your changes¶

1. Create a container from custom.sqsh:

   enroot create --name my-custom-container custom.sqsh
   

2. Start it, again with NVIDIA_VISIBLE_DEVICES=void:

   NVIDIA_VISIBLE_DEVICES=void enroot start --rw my-custom-container
   

3. Check that the package is available:

   python -c "import plotly; print(f'plotly version: {plotly.version}')"
   

   You should see:

   Expected output

   plotly version: <some version>
   

Remove containers you no longer need¶

Enroot doesn't remove containers when you exit them. They stay in ~/.local/share/enroot/ and keep using disk space until you remove them.

List your active containers:

enroot list

Remove one by name:

enroot remove <container-name>

For example, to remove the containers from this guide:

enroot remove my-container
enroot remove my-custom-container

This only removes the containers. Your custom image is still saved as a .sqsh file, ready to build a new container with Enroot or Pyxis.

Use your image in a job¶

Set --container-image to your saved image. Use it the same way as a pre-built image:

srun --gres=gpu:1 --container-image=./custom.sqsh python -c "import plotly; print(f'plotly version: {plotly.version}')"