Predictive Learning (PrL) API¶

This storage API is provided for users who require the ability to upload and download large files and have the files available to work with Predictive Learning. The PrL API is a simple remote storage repository that allows all operations using a temporary token and session.

Access to the API¶

Access to the storage path and API is provided for PrL clusters, environments, and jobs. AWS and Azure CLI can interact with the PrL API using these commands:

Access:¶

aws s3 ls s3://prl-storage-<digits>/<tenant>/data/mypath/

Uploading files:¶

aws s3 cp /tmp/myfile.txt s3://prl-storage-<digits>/data/<tenant>/data/mypath/myfile.txt

Downloading files:¶

aws s3 cp s3://prl-storage-<digits>/<tenant>/data/mypath/myfile.txt /tmp/myfile.txt

External Access to the API¶

To access the storage API from outside Predictive Learning, you must obtain a temporary token from the PrL API and use the provided credentials, through either a credentials file store, or through environment variables.

Use a POST request to obtain a token from the API:

curl --location --request POST $GATEWAY_ENDPOINT$'/prlstorage/v3/generateAccessToken' --header 'Content-Type: application/json'

Which results in:

    {
        "credentials": {
            "accessKeyId": "<access_key>",
            "secretAccessKey": "<secret>",
            "sessionToken": "<session>",
            "expirationTime": "2019-11-19T14:33:21.000"
        },
        "storageAccount": "prl-storage-<account>",
        "storagePath": "/<mytenant>/data/"
    }

Alternatively, you can extract the needed keys if you store the request result in a bash variable that can be parsed with the following commands:

    %%bash
    content=$(curl --location --request POST $GATEWAY_ENDPOINT$'/prlstorage/v3/generateAccessToken' --header 'Content-Type: application/json')
    secret=$(jq -r '.credentials.secretAccessKey' <<< "${content}")
    session=$(jq -r '.credentials.sessionToken' <<< "${content}")
    accesskey=$(jq -r '.credentials.accessKeyId' <<< "${content}")
    export AWS_ACCESS_KEY_ID=$(echo "${accesskey}")
    export AWS_SECRET_ACCESS_KEY=$(echo "${secret}")
    export AWS_SESSION_TOKEN=$(echo "${session}")

When PrL storage is used as an input or output source for a job, the root tenant path is available to the Docker container being executed. Both input and output is mounted in the container's /data/input and /data/output paths.

Except where otherwise noted, content on this site is licensed under the Development License Agreement.

Last update: February 22, 2023