Build your own container image with Enroot

Enroot is a container runtime. It builds and runs the container images that Pyxis uses in your SLURM jobs.

If the pre-built images don't have what you need, build your own with Enroot.

srun --partition=debug --mem=32G --pty bash

Import a base image¶

Pull an image from a registry. Enroot converts it into a .sqsh file:

enroot import docker://nvcr.io/nvidia/pytorch:26.07-py3

This creates a .sqsh file in your current directory, named after the image — in this example, nvidia+pytorch+26.07-py3.sqsh. A .sqsh file is a single, compressed, read-only image of the container's filesystem. Enroot uses this format to store and run container images.

Open a writable session¶

Creating a container from the image and starting it are two separate steps:

1. Create a container from the image:

   enroot create --name my-container nvidia+pytorch+26.07-py3.sqsh
   

2. Start it with a writable filesystem:

   NVIDIA_VISIBLE_DEVICES=void enroot start --rw my-container
   

   Add --root too if you plan to install an Ubuntu package with apt in step 3 — without it, you don't have permission to write to the system directories apt needs:

   NVIDIA_VISIBLE_DEVICES=void enroot start --rw --root my-container
   

   NVIDIA_VISIBLE_DEVICES=void starts the container even though the GPU isn't available in a login session. You'll still see warnings and errors about GPU functionality — ignore them:

   Expected warnings and errors

   [10-nvidia-mps.sh] WARNING: MPS control socket not found at /tmp/nvidia-mps/control
   [10-nvidia-mps.sh] Ensure nvidia-cuda-mps.service is running on the host
   ...
   ERROR: The NVIDIA Driver is present, but CUDA failed to initialize. GPU functionality will not be available.
   [[ Unable to initialize CUDA driver (error ???) ]]
   Failed to detect NVIDIA driver version.
   

   You'll see a prompt like this once you're inside the container:

   <username>@chispa:/workspace$
   

   With --root, the prompt shows root in place of your username.

3. Make your changes inside the session. For example, install a Python package with pip:

   pip install plotly
   exit
   

   Or install an Ubuntu package with apt (needs the container started with --root, as shown above):

   apt-get update
   apt-get install --yes graphviz
   exit
   

Save your changes¶

Export the container to a .sqsh file:

enroot export --output custom.sqsh my-container

Check that the custom image preserved your changes¶

1. Create a container from custom.sqsh:

   enroot create --name my-custom-container custom.sqsh
   

2. Start it, again with NVIDIA_VISIBLE_DEVICES=void:

   NVIDIA_VISIBLE_DEVICES=void enroot start --rw my-custom-container
   

3. Check that your changes are still there. If you installed plotly with pip:

   python -c "import plotly; print(f'plotly version: {plotly.version}')"
   

   You should see:

   Expected output

   plotly version: <some version>
   

   If you installed graphviz with apt, check:

   dot -V
   

   You should see:

   Expected output

   dot - graphviz version <some version> (<some build>)
   

Remove containers you no longer need¶

Enroot doesn't remove containers when you exit them. They stay in ~/.local/share/enroot/ and keep using disk space until you remove them.

List your active containers:

enroot list

Remove one by name:

enroot remove <container-name>

For example, to remove the containers from this guide:

enroot remove my-container
enroot remove my-custom-container

This only removes the containers. Your custom image is still saved as a .sqsh file — use it to create a new container with Enroot, or run a job with Pyxis.

Use your image in a job¶

Set --container-image to your saved image. Use it the same way as a pre-built image:

srun --gres=gpu:1 --container-image=./custom.sqsh python -c "import plotly; print(f'plotly version: {plotly.version}')"