By: N8 HPC Technical Management Group, Newcastle University
Why haven’t my jobs started yet? When will they start? I’ve checked available system resources, using the qhost command, which tells me that there are spare resources.
Waiting for a job to execute can be frustrating, especially when it is not clear why.
The qhost command shows the current status of hosts, queues and the jobs associated with the queues, that are available to Grid Engine, the queue-managing software. However, it is not able to give a rich picture of the factors that might affect job execution time.
Why haven’t my jobs started yet?
There are several reasons why the job might not have started yet:
- The available nodes might be large memory nodes
Polaris has two types of compute node. Both have 16 cores of the same CPUs, but differ in the quantity of the RAM. There are 16 nodes with 256G RAM (16G/core), all the others have 64G (4G/core).
By default, jobs pick up a request to run on the 64G nodes, leaving the 256G nodes free to run “capability” jobs that require the memory.
To run on the 256G nodes, you need to add “-l node_type=16core-256G” to your qsub command.
- The nodes might be reserved for a higher-priority job
Polaris operates a fair-share scheme, where each job attracts a priority based on how much its institution, project and user has recently used compared to the others. Each scheduling interval, the system runs through the waiting job list in order of priority, attempting to launch jobs in turn.
For the first few (currently 128) waiting jobs where there aren’t enough free resources to start, the scheduler will reserve the resources needed to start them.
The reserved resources can still be used by other jobs, but only if those other jobs are submitted with a sufficiently small time (“h_rt”) request that they will definitely end before the reservation is needed.
- The nodes might be disabled
Owing in large to a major flooding incident at the start of 2016, there are currently a number of hardware faults on compute nodes, awaiting investigation and repair.
- The nodes might be running jobs that asked for a lot of memory
On the 64G nodes, asking for more than 4Gb per core (16Gb for the highmemory nodes) will result in cores being idle.
The nodes might be running jobs that, for some reason, are not using the resources they asked for.
When will my job start?
You can ask Grid Engine for an estimated start-time for your jobs with the command:
Will show all reservations, which information could help you judge what resources to ask for, in order to try to slot a job into idle space.
Please note that qsched only reports on the jobs that the batch system is reserving resources for (i.e. jobs mentioned in section 2), basing its estimates on queue conditions at the time that it is run.