Skip to content

Slurm REST API

1. Introduction

The ORFEO HPC infrastructure provides a REST API for interacting with the Slurm workload manager programmatically.

The Slurm REST API allows users to retrieve cluster information, monitor jobs, submit workloads and manage their executions using standard HTTP requests.

Unlike the traditional Slurm command-line interface (squeue, sbatch, sinfo, etc.), the REST API can be integrated directly into external applications, automation scripts, web interfaces and custom monitoring tools.

What can you do with the API?

The API supports several operations, including:

  • Check the availability and status of compute nodes.
  • Retrieve information about cluster partitions.
  • List running and pending jobs.
  • Retrieve details about a specific job.
  • Submit new batch jobs.
  • Cancel jobs.
  • Retrieve historical job accounting information.

The API follows the same Slurm permissions and resource allocation policies as the command-line tools. Users can only perform operations permitted by their Slurm account and privileges.

2. API Endpoint

The ORFEO Slurm REST API is available at:

https://slurmrestd.hpc.rd.areasciencepark.it

The currently documented API version is:

v0.0.44

The main API paths are:

Base path Purpose
/slurm/v0.0.44/ Cluster status, nodes, partitions and job management
/slurmdb/v0.0.44/ Job accounting and historical information

For example, the following endpoint returns information about cluster nodes:

https://slurmrestd.hpc.rd.areasciencepark.it/slurm/v0.0.44/nodes

The API uses standard HTTP methods. The examples in this guide use JSON responses and JSON request bodies where applicable.

The examples below use curl to send requests and jq to format JSON responses.

3. Authentication

The REST API uses JSON Web Tokens (JWT) for authentication.

Each user must generate a personal token using their existing ORFEO Slurm account.

No separate REST API account is required.

3.1 Generate an authentication token

Connect to an ORFEO login node through SSH using your usual credentials.

Generate a JWT token:

scontrol token lifespan=3600

The command returns a token in the following format:

SLURM_JWT=eyJhbGciOiJIUzI1NiIs...

The lifespan parameter specifies the token lifetime in seconds.

In this example, the token is valid for one hour.

3.2 Store the token in an environment variable

For convenience, the token can be generated and exported directly:

export SLURM_USER="$(whoami)"

unset SLURM_JWT
export SLURM_API_TOKEN="$(scontrol token lifespan=3600 | cut -d= -f2-)"

Define the API URL and version:

export SLURM_API_URL="https://slurmrestd.hpc.rd.areasciencepark.it"
export SLURM_API_VERSION="v0.0.44"

These variables will be used in the following examples.

Important: scontrol token outputs a value named SLURM_JWT, but do not export your personal token as SLURM_JWT when using the standard Slurm command-line tools in the same shell. Slurm treats that variable specially and may switch sacct, squeue, sbatch, or srun to JWT authentication, causing failures. The independent variable SLURM_API_TOKEN is only used in the HTTP headers below. If you previously exported SLURM_JWT, run unset SLURM_JWT to restore the usual Slurm CLI authentication. See the official Slurm troubleshooting guide.

3.3 Authentication headers

Every authenticated API request must include the following HTTP headers:

Header Description
X-SLURM-USER-NAME Your ORFEO username
X-SLURM-USER-TOKEN Your JWT authentication token

Example:

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  "$SLURM_API_URL/slurm/$SLURM_API_VERSION/ping" \
  | jq

Security recommendations

  • Never share your JWT token with other users.
  • Do not include tokens in public repositories or application source code.
  • Do not expose tokens in application logs.
  • Always use HTTPS when communicating with the REST API.
  • Generate a new token when the previous one expires.

Tokens inherit the identity and permissions of the associated Slurm user.

4. Basic API Examples

4.1 Check API connectivity

Use the /ping endpoint to verify communication with the Slurm controller.

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  "$SLURM_API_URL/slurm/$SLURM_API_VERSION/ping" \
  | jq

The response contains a pings array describing Slurm controller status. Do not treat HTTP 200 alone as proof that the controller is healthy: also inspect pings[].responding and pings[].pinged.

On ORFEO, /ping has sometimes reported DOWN even while other REST endpoints and job operations were working. Therefore, use this endpoint as a diagnostic indicator, not as the sole service health check.

4.2 List cluster nodes

Equivalent Slurm CLI command:

sinfo -N

REST API request:

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  "$SLURM_API_URL/slurm/$SLURM_API_VERSION/nodes" \
  | jq

To display only node names and states:

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  "$SLURM_API_URL/slurm/$SLURM_API_VERSION/nodes" \
  | jq '.nodes[] | {name, state}'

This is useful for checking whether compute resources are available, allocated or unavailable.

4.3 List cluster partitions

Equivalent Slurm CLI command:

sinfo

REST API request:

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  "$SLURM_API_URL/slurm/$SLURM_API_VERSION/partitions" \
  | jq

To display only partition names:

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  "$SLURM_API_URL/slurm/$SLURM_API_VERSION/partitions" \
  | jq -r '.partitions[].name'

4.4 List jobs

Equivalent Slurm CLI command:

squeue

REST API request:

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  "$SLURM_API_URL/slurm/$SLURM_API_VERSION/jobs" \
  | jq

To display a simplified list:

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  "$SLURM_API_URL/slurm/$SLURM_API_VERSION/jobs" \
  | jq '.jobs[] | {job_id, name, user_name, partition, job_state}'

4.5 List only your jobs

Equivalent Slurm CLI command:

squeue --me

The following example filters the returned jobs by username:

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  "$SLURM_API_URL/slurm/$SLURM_API_VERSION/jobs" \
  | jq --arg user "$SLURM_USER" \
    '.jobs[] | select(.user_name == $user) |
    {job_id, name, partition, job_state}'

This retrieves the queue information and displays only the jobs belonging to the authenticated user.

Note that the filtering in this example is performed locally with jq, after retrieving the API response.

For frequent status checks, applications should avoid unnecessarily retrieving the entire job list.

4.6 Get information about a specific job

Equivalent Slurm CLI command:

scontrol show job JOB_ID

Set the job ID:

export JOB_ID=123456

Replace 123456 with an actual job ID.

Query the job:

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  "$SLURM_API_URL/slurm/$SLURM_API_VERSION/job/$JOB_ID" \
  | jq

To display selected fields:

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  "$SLURM_API_URL/slurm/$SLURM_API_VERSION/job/$JOB_ID" \
  | jq '.jobs[0] | {job_id, name, partition, job_state, nodes}'

5. Submitting a Job Through the API

The REST API can also submit batch jobs, similarly to the sbatch command.

Job submission uses the HTTP POST method and a JSON request body.

5.1 Prepare a simple job

Create a file named submit_job.json:

{
  "script": "#!/bin/bash\nhostname\ndate\nsleep 30\necho 'Job completed'\n",
  "job": {
    "name": "rest-api-test",
    "account": "YOUR_ACCOUNT",
    "partition": "EPYC",
    "tasks": 1,
    "cpus_per_task": 1,
    "memory_per_node": 1024,
    "time_limit": 5,
    "current_working_directory": "/path/to/your/home",
    "environment": [
      "PATH=/usr/local/bin:/usr/bin:/bin"
    ]
  }
}

Before submitting, replace:

  • YOUR_ACCOUNT with your authorized Slurm account.
  • /path/to/your/home with an existing directory accessible on the compute nodes.
  • EPYC with another partition if appropriate for your workload and account permissions.

The job requests one CPU, approximately 1 GiB of memory and a maximum execution time of five minutes.

5.2 Submit the job

curl -sS -X POST \
  -H "Content-Type: application/json" \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  --data-binary @submit_job.json \
  "$SLURM_API_URL/slurm/$SLURM_API_VERSION/job/submit" \
  | jq

If the submission is accepted, the response includes a job_id.

Example of a simplified successful response:

{
  "job_id": 123456,
  "step_id": "batch",
  "errors": [],
  "warnings": []
}

Use the returned job ID to query its status.

Important: a successful HTTP response does not always mean that Slurm accepted the job. Check errors and other operation-specific fields, and confirm that a valid job_id was returned.

Job submissions remain subject to the same account, partition, QoS, time and resource limits as submissions performed with sbatch.


6. Cancelling a Job

Jobs can be cancelled through the API using the HTTP DELETE method.

Equivalent Slurm CLI command:

scancel JOB_ID

Set the job ID:

export JOB_ID=123456

Cancel the job:

curl -sS -X DELETE \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  "$SLURM_API_URL/slurm/$SLURM_API_VERSION/job/$JOB_ID" \
  | jq

7. Accessing Job Accounting Information

In addition to current job information, the REST API provides access to the Slurm accounting database.

Accounting endpoints use the /slurmdb/ prefix instead of /slurm/.

For example, to retrieve accounting records associated with your username:

curl -sS \
  -H "X-SLURM-USER-NAME: $SLURM_USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_API_TOKEN" \
  "$SLURM_API_URL/slurmdb/$SLURM_API_VERSION/jobs/?users=$SLURM_USER" \
  | jq

This is useful for inspecting completed jobs, execution history and resource consumption information available through Slurm accounting.


8. Common Errors and Troubleshooting

8.1 Understanding API Responses and Error Codes

An HTTP 200 response does not necessarily mean that the requested Slurm operation was successful.

Some API endpoints report operation-specific errors inside the JSON response, even when the HTTP request itself completed successfully.

Applications should inspect both the HTTP status code and the relevant JSON fields, including:

  • errors: general errors reported by the API.
  • status: individual operation results, which may contain additional error information.
  • warnings: non-fatal warnings.

For example, attempting to cancel a job that is already completing or completed may return HTTP 200 with the following error inside the status array:

{
  "status": [
    {
      "error": {
        "code": 2021,
        "message": "Job/step already completing or completed"
      }
    }
  ],
  "errors": []
}

Always inspect the full response before considering an operation successful. In particular, do not rely exclusively on HTTP status codes or an empty errors array.

8.2 Slurm CLI authentication errors after using a JWT

If sacct, squeue, or other Slurm commands start returning errors such as Protocol authentication error, Failed to unpack SLURM_PERSIST_INIT message, or Problem talking to the database after you generated a token, check whether SLURM_JWT is exported in your shell.

Restore standard Slurm CLI authentication with:

unset SLURM_JWT

Alternatively, remove it for a single command:

env -u SLURM_JWT sacct -j JOB_ID

Always store the token for curl in SLURM_API_TOKEN, not in SLURM_JWT.

8.3 Refreshing an expired token

If your token has expired, generate a new one:

unset SLURM_JWT
export SLURM_API_TOKEN="$(scontrol token lifespan=3600 | cut -d= -f2-)"

9. Further Documentation

Slurm REST API — Methods and Models

Complete reference for API endpoints, HTTP methods, parameters, request bodies and response structures.

Slurm REST API Client Writing Guide

Guidance for developers building API clients, including OpenAPI schema generation and version compatibility.