Audience: software developers

Most of the time, SURF Research Cloud will be used interactively through the web portal. But in some cases, it makes sense to have your own program interact with Research Cloud.

The Research Cloud API is the interface that can be called to create and manage workspaces and all other resources.

The API is built following the REST paradigm.

Access Token

To obtain access to the Research Cloud API, you will create a personal access token through the Research Cloud portal.

  • In the portal, go to the "Profile" tab and select the "API tokens" tab in the top display tile.
  • You can give a name and a description for the token, for later reference. The generated token itself is displayed only once, ever. Make sure that you copy it and store it in a safe place.
The token is user based, should not be shared among people or hardcoded in any source code, and in general it should be handled securely - just like a password.

Exploring with Swagger

Swagger provides documentation for a given API. But Swagger is also a tool that allows you to explore an API interactively, without writing your own client (yet).

You can browse through the endpoints that are available. (An "endpoint" is a particular URL of a webservice that you can call.)

You can also authenticate with your API token and make calls to the API.

Be aware that you are actually interacting with your real Research Cloud user account. Any changes made by your calls in Swagger are for real.

But generally, any GET request will not make any changes - only retrieve information. GET is safe to execute while exploring.

Links to the Swagger documentation pages:

The Base URLs

As of now, four microservices of Research Cloud expose their endpoints through the API:

The API is versioned (currently v1). Changes to the API will be introduced in a backward compatible manner, in case of breaking changes we will create new versions and retire endpoints with a graceful retention period and proper notification beforehand.

A First Call with curl

This example uses a Linux terminal to send a simple call to the API.

First put your API token into a environment variable. (This is a good habit that keeps your token from leaking into saved code and ending up in a public git repository.)

Then use the curl command to make the actual call:


Using 'curl'
$ export RESEARCH_CLOUD_TOKEN="personal_access_token_obtained_through_the_portal"
 
$ curl -X 'GET' \
   'https://gw.live.surfresearchcloud.nl/v1/user/users/self/' \
   -H 'accept: application/json' \
   -H "authorization: $RESEARCH_CLOUD_TOKEN"

Explore with a Jupyter Notebook

To get close to writing your own client, you might want to use a Jupyter Notebook.

The examples here are in Python, but since this is all about making https requests, you should be able to transfer it to the language of your choice.

Create an .env file

In your working directory, create a file called ".env". It contains all the environment variables that your code will pick up using the "dotenv" package.

.env file
RESEARCH_CLOUD_TOKEN="your_personal_access_token_obtained_through_the_portal"
CATALOG_BASE_URL="https://gw.live.surfresearchcloud.nl/v1/application-market/"
USER_BASE_URL="https://gw.live.surfresearchcloud.nl/v1/user/"
WALLET_BASE_URL="https://gw.live.surfresearchcloud.nl/v1/wallet/"
WORKSPACE_BASE_URL="https://gw.live.surfresearchcloud.nl/v1/workspace/"

First Call with Python

This code can go into a cell of a Jupyter Notebook.

It prepares the environment and declares a generic method to make calls to API endpoints.

Then it executes a call to the user microservice to retrieve the available user data.

Copy-Button issue

The copy button on the top right of the code examples might insert invalid characters into the code. Not our fault. If the button gives you trouble, just copy the code with your mouse.


Get user data
%pip install python-dotenv
%pip install requests
 
import os
import json
import logging
import requests
 
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
 
from dotenv import load_dotenv
from urllib.parse import urljoin, quote_plus
 
load_dotenv()
RESEARCH_CLOUD_TOKEN = os.getenv("RESEARCH_CLOUD_TOKEN")
CATALOG_BASE_URL = os.getenv("CATALOG_BASE_URL")
USER_BASE_URL = os.getenv("USER_BASE_URL")
WALLET_BASE_URL = os.getenv("WALLET_BASE_URL")
WORKSPACE_BASE_URL = os.getenv("WORKSPACE_BASE_URL")
 
HEADERS = {
    "authorization": RESEARCH_CLOUD_TOKEN,
    "accept": "application/json"
}
 
def make_request(method, base_url, path="", headers=None, params=None, data=None):
    url = urljoin(base_url, path)
    logger.info(f"Making {method} request to {url} with params: {params} and data: {data}")
 
    fallback_error = {"message": ["Unable to decode error response"]}
    fallback_status = 500
 
    try:
        response = requests.request(method, url, headers=headers, params=params, json=data)
        response.raise_for_status()
        logger.info(f"Response status: {response.status_code}")
 
        content_type = response.headers.get("Content-Type", "")
        return response.status_code, response.json() if "application/json" in content_type else response.text
 
    except requests.exceptions.HTTPError as e:
        error_response = fallback_error
        try:
            error_response = response.json()
        except (json.JSONDecodeError, AttributeError):
            pass
 
        logger.error(f"HTTPError for URL {url}: {e}, Message: {error_response.get('message', [])}")
        return response.status_code, error_response
 
    except requests.exceptions.RequestException as e:
        logger.error(f"RequestException for URL {url}: {e}")
        return fallback_status, {"message": [str(e)]}
 
# print JSON response with indents
def pretty(response_data):
    print(json.dumps(response_data, indent=4))
 
# User - GET self
status, response_data = make_request("GET", USER_BASE_URL, "users/self/", HEADERS)
pretty(response_data)

Example: Creating a Workspace

Terminology

We are currently working on simplifying the content of the requests - the payload.

For creating a workspace you will need an "application" object which represents a "catalog item" from the portal.

"application offering" and "offering" are synonymous. Both terms refer to the object that links together a catalog item, a cloud provider ("subscription") and the OS- and size-flavours that are available.

For the time being the following steps are needed to construct the payload:

1.  get user collaborative organisations and select one
2. get user wallets and select one
3. get the catalog items with offerings: /v1/application-market/catalog_items/offerings/ and select one (parameters: co=selected co id, product based on wallet, type=Compute)
4. get the offerings (that can be used to create a workspace) for the selected catalog item: /v1/application-market/catalog_items/{catalog_item_id}/offerings/ (parameters: co=selected co id, product based on wallet)
5. Important: choose one os and one size flavour and pass all of their information (for now) in the payload

Optional: IP, Storage, Network attachment

Payload to Request a New Workspace

The following template shows how a new workspace can be requested from the API.

How to fill in the capitalized DETAILS is shown in the Python code examples hereafter.

If you are using a Jupyter Notebook, this would be the final cell to execute for creating a workspace.

The other cells below are working towards executing this one. 

META_DATA = {
    "application_offering_id": OFFERING["id"],
    "application_name": OFFERING["application"]["name"],
    "application_icon": CATALOG_ITEM_ICON, # pass the icon of the catalog item of choice
    "application_type": APPLICATION_TYPE, # Compute, Storage, IP, Network
    "subscription_tag": OFFERING["subscription"]["tag"],
    "subscription_name": OFFERING["subscription"]["name"],
    "subscription_group_id": OFFERING["subscription"]["subscription_group"]["id"],
    "co_name": CO_NAME,
    "host_name": HOST_NAME, # needs to be unique
    "subscription_resource_type": "VM",
    "flavours": [
        # Select one 'os' and one 'size' flavour from OFFERING['flavours']
        OS_FLAVOUR, # pass all details of the selected flavour with 'category': os
        SIZE_FLAVOUR # pass all details of the selected flavour with 'category': size
    ],
    "storages": [],
    "ips": [],
    "networks": [],
    "dataset_names": [],
    "dataset_ids": [],
    "interactive_parameters": [],
    "wallet_name": WALLET_NAME,
    "wallet_id": WALLET_ID
}
 
CREATE_DATA = {
    "co_id": CO_ID,
    "wallet_id": WALLET_ID,
    "description": WORKSPACE_DESCRIPTION,
    "name": WORKSPACE_NAME,
    "meta": META_DATA,
    "end_time": WORKSPACE_END_TIME # example: "2025-02-25T00:00:00.000Z"
}
 
status, response_data = make_request("POST", WORKSPACE_BASE_URL, "workspaces/", HEADERS, data=CREATE_DATA)
pretty(response_data)

API calls needed to construct the above payload

CO Details

# GET User details
status, user_response = make_request("GET", USER_BASE_URL, "users/self/", HEADERS)

all_cos = user_response['COs']
for ix in range(len(all_cos)):
    co = all_cos[ix]
    print(f"{ix:>2} {co['id']} {co['name']}")
# Select one CO's id and name to CO_ID and CO_NAME
my_selected_co_index = 8 # Fill in manually

co = all_cos[my_selected_co_index]
CO_ID = co['id']
CO_NAME = co['name']

Wallet Details

# GET Wallet(s)
status, wallet_response = make_request("GET", WALLET_BASE_URL, f"wallets/", HEADERS)

for ix in range(len(wallet_response)):
    wallet = wallet_response[ix]
    print(f"{ix:>2} {wallet['id']} {wallet['name']}")
# Select one of the wallets
my_selected_wallet_index = 3 # Fill in manually

wallet = wallet_response[my_selected_wallet_index]
PRODUCTS = wallet['budgets'][0]['products']
WALLET_ID = wallet['id']
WALLET_NAME = wallet['name']

Catalog Item

# List of catalog items with valid offerings (that can be used to create a workspace)
parameters = {
    "co": CO_ID,
    "type": "Compute",
    "product": PRODUCTS,
}
status, catalog_items_response = make_request("GET", CATALOG_BASE_URL, "catalog_items/offerings/", HEADERS, parameters)

catalog_items = catalog_items_response['results']
for ix in range(len(catalog_items)):
    catalog_item = catalog_items[ix]
    print(f"{ix:>2} {catalog_item['id']} {catalog_item['name']}")
# Select one of the catalog items
my_selected_catalog_item_index = 32 # Fill in manually 
catalog_item = catalog_items[my_selected_catalog_item_index]
CATALOG_ITEM_ID = catalog_item['id']
CATALOG_ITEM_ICON = catalog_item['icon']

Paginated Responses

The responses of some of the endpoints can be rather big. This is why those are paginated to 50 items, by default.

Endpoints that paginate, add a next and a previous element to the response. Those elements contain a URL for the same endpoint, with query string parameters like

<endpoint-URL>/?offset=50&limit=50
  1. You can keep calling the next URL to fetch items, building up a complete list.
  2. You can also, right with your first call, put an offset of zero and a very high number for the limit.

Option 1. might seem more cumbersome but guarantees a smooth performance.


Offerings

# Based on the PRODUCTS and the catalog item id its valid offerings (than can be used to create a workspace) can be queried
# GET catalog item offerings
parameters = {
    "co": CO_ID,
    "product": PRODUCTS,
}
status, offerings_response = make_request("GET", CATALOG_BASE_URL, f"catalog_items/{quote_plus(CATALOG_ITEM_ID)}/offerings/", HEADERS, parameters)

offerings = offerings_response['results']
for ix in range(len(offerings)):
    offering = offerings[ix]
    print(f"{ix:>2} {offering['id']} {offering['name']}")
my_selected_offering_id = 0 # Fill in manually 
OFFERING = offerings[my_selected_offering_id]

Flavours - OS and Size

flavours = OFFERING['flavours']
for ix in range(len(flavours)):
    flavour = flavours[ix]
    if flavour['category'] == 'os':
        print(f"{ix:>2} {flavour['category']:<8} {flavour['name']}")

print()

for ix in range(len(flavours)):
    flavour = flavours[ix]
    if flavour['category'] == 'size':
        print(f"{ix:>2} {flavour['category']:<8} {flavour['name']}")
my_selected_os_flavour_ix = 9 # Fill in manually  
my_selected_size_flavour_ix = 2 # Fill in manually  

OS_FLAVOUR = flavours[my_selected_os_flavour_ix]
SIZE_FLAVOUR = flavours[my_selected_size_flavour_ix]

APPLICATION_TYPE = 'Compute'

"Manual" Fields to Fill in

HOST_NAME = 'my1stapivm'
WORKSPACE_DESCRIPTION = """
    whatever ...
"""
WORKSPACE_NAME = "My 1st API Workspace"
WORKSPACE_END_TIME = "2025-03-22T00:00:00.000Z" # Pick a date in the (near) future.

(Finally) Submit the Complete Payload Request

Now all required variables for the workspace creation payload are set.

Go ahead and execute that earlier payload cell!

You should see the new workspace come up in the portal, too. (Give it a few seconds)

Further Actions/Assignments

Go back to the Swagger pages and find out how you can pause, resume and delete the workspace you just created.


  • No labels