FAQ

SSH sessions limit is 10. Close other connections and try again.

Occurs mainly when connecting from outside the campus. Make sure the VPN is connected and try to SSH to 132.68.39.200 instead.

Question

I can see free CPUs and GPUs on a cluster node, but my Slurm job remains in the PENDING (Resources) state. Why can’t my job start?

Answer

The most common reason is the server’s NUMA (Non-Uniform Memory Access) configuration.

To achieve better performance, the cluster associates groups of CPU cores with specific GPUs. As a result, CPUs and GPUs are not treated as one large pool of resources.

For example:

• NUMA Group 0

1. GPUs: All allocated
2. CPUs: 28 available

• NUMA Group 1

1. GPUs: 1 available
2. CPUs: 0 available

Although the node appears to have both free CPUs and a free GPU, there is no NUMA group that has both available simultaneously. Therefore, Slurm cannot satisfy the resource request and keeps the job in the PENDING (Resources) state.

Why is the cluster configured this way?

NUMA-aware scheduling improves performance by:

• reducing memory latency;
• improving CPU–GPU locality;
• increasing memory bandwidth;
• avoiding unnecessary cross-socket communication.

This configuration provides significantly better performance for GPU workloads than treating all CPUs as a single shared resource.

What can I do?

You have several options:

• Wait until resources become available within the required NUMA group.
• Use another node if one has a matching CPU/GPU combination available.
• Ask users running CPU-only jobs on the node whether they can move their workload to the Darwin cluster (our CPU-only cluster), freeing the CPUs required for GPU jobs.

How can I verify the NUMA allocation?

Start an interactive Slurm session on the target node (for example, requesting one CPU), and run:

check_cpu_numa

The command displays:

• CPU allocation by NUMA node;
• free CPUs in each NUMA group;
• GPU allocation associated with each NUMA group.

This information allows you to determine whether a compatible CPU/GPU combination is currently available.

Summary

Seeing free CPUs and free GPUs on a node does not necessarily mean they can be allocated together. Because the cluster uses NUMA-aware scheduling, Slurm requires the requested CPUs and GPUs to be available within the same NUMA locality. If they are not, the job will remain in the PENDING (Resources) state until a compatible allocation becomes available.

Resource over-scheduling (for the pending job) or cluster load. See the Workload management and Job and resource limits sections.

If you receive this error on newton (the login server) – launch the script on a node using Slurm. Newton is a virtual server, has no GPUs and doesn’t have CUDA installed.

If you receive the error on a node:

Add these lines at the end of the .bashrc file in your home folder (~/.bashrc):

export PATH=”/usr/local/cuda/bin:$PATH”
export LD_LIBRARY_PATH=”/usr/local/cuda/lib64:$LD_LIBRARY_PATH”

Save the file, exit the session (or job) and reconnect. Start bash on a node:

srun –pty /bin/bash
and run this command
nvcc -V

You should get CUDA’s version details.

Run this on some different Node (with flag -w) with different gpu and did you receive same error .

Set reserve memory for pytorch (clean cache by # torch.cuda.empty_cache() )

Release allocated memory : It can be done using the API functions free() and cudaFree(). Notice that, any memory declared using CUDA API, e.g. cudaMallocHost(), needs to be freed using cudaFree() as follows:

cudaFree(d_t1);
free(h_t1);

Check summary CUDA use :

torch.cuda.memory_summary()