Skip to content

Bulk Batch File Upload

Note

This section is applicable only for region Europe 1.

Bulk batch upload of objects in Integrated Data Lake adds the possibility to import or export data in bulk. It can be done from and to the external servers or programs to analyze or filter massive data. Bulk batch file upload uses the STS token provided by AWS with write access. It is required to specify the folder in which the bulk files need to be uploaded. Thereafter, the STS token with write access to that path (folder) is generated. Using this STS token, you can upload files in bulk. The validity for the STS token is up to 12 hours.

Set environment variables

Set the environment variables for your system:

Environment Paths on Windows

C:> setx AWS_ACCESS_KEY_ID XXXXXXXXXXXXXXXXXXXX C:> setx AWS_SECRET_ACCESS_KEY xxxxxxxxxxxxx/xxxxxxx/xxxxxxxxxxxxxxxxxx C:> setx AWS_SESSION_TOKEN=<>

Environment Paths on Linux

$ export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX $ export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxx/xxxxxxx/xxxxxxxxxxxxxxxxxx $ export AWS_SESSION_TOKEN=<>

Provide write permission

The tenantAdmin can access this API and provide write permission on folder or path, on the subtenants folder and on the root folder, except for the Time Series Import (TSI) folder.

POST /accessTokenPermissions
Content-Type: Content-Type: application/json

Request example:

{
  "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
  "path": "/myfolder1/myfolder2",
  "permission": "WRITE"
}

Response example:

{
  "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
  "id": "string",
  "path": "/myfolder1/myfolder2",
  "permission": "WRITE",
  "created": "2019-11-12T13:36:00.000Z"
}

Get a list of write permission folders

The tenantAdmin can get a list of all the folders that have the write permission. It is also possible to get a list of write permissions of the subtenants folders.

GET /accessTokenPermissions
Content-Type: application/json

Response example:

{
   "accessTokenPermissions": [
  {
    "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
    "id": "string",
    "path": "/myfolder1/myfolder2",
    "permission": "WRITE",
    "created": "2019-11-12T13:36:00.000Z"
  }
  ]
}

Get the list for write permission folders for the given id

You can get the list of folders that have the write permissions, for the given id. The tenantAdmin can also get the details of the request for subtenants. The subtenants can access this API to get details of the request belongs to their write folder.

GET /accessTokenPermissions/{id}
Content-Type: application/json

Response example:

{
  "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
  "id": "string",
  "path": "/myfolder1/myfolder2",
  "permission": "WRITE",
  "created": "2019-11-12T13:36:00.000Z""id": "0234sd34a23a-11e9-a2a3-2a2sdfw34ce4",
}

Revoke write permission

The TenantAdmin can access this API and provide delete permission on folders for the given id.

DELETE /accessTokenPermissions/{id}
Content-Type: application/json

Response example:

Status: 204 - deleted

Get the STS token with Write access

Request URL:

https://gateway.eu1.mindsphere.io/api/datalake/v3/generateAccessToken

Request example:

{
  "durationSeconds": 900,
  "permission": "WRITE"
}

Response example:

{
  "storageAccount": "dlbucketname",
  "storagePath": "data/ten=tenantname OR data/sub=subtenantId",
  "credentials": {
    "secretAccessKey": "AbCdEfGhIjKlMnOpQrStUvWxYz",
    "accessKeyId": "AbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYz",
    "sessionToken": "AbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYz"
  },
  "durationSeconds": 900,
  "permission": "WRITE"
}

Command and Output

The following command is used for Bulk upload:

aws s3 cp s3://ms-coldstorage-iot-integ-eu-central-1/000013bb5ed740f9950504854367e12a s3://estbuckettimeseries/ten=dide2/000013bb5ed740f9950504854367e12a --recursive

Completed 1.7 KiB/1.7 KiB (812 Bytes/s) with 1 file(s) remaining
copy: s3://ms-coldstorage-iot-integ-eu-central-1/000013bb5ed740f9950504854367e12a/propSetName=SPSNFRTestFields/date=2019-04-05/0900.parquet to s3://estbuckettimeseries/ten=dide2/000013bb5ed740f995050485436
7e12a/propSetName=SPSNFRTestFields/date=2019-04-05/0900.parquet

Note

The time required to upload files is dependent on the network bandwidth and location.

STS Token Additional API

Allows users to request temporary, limited-privilege AWS credentials to get read-only or write-only access on the URI returned in the response.

  • Generated access credentials can be used to access all the paths provided in the request with their respective permissions.
  • Path field is mandatory if any permission is provided in the request or vice-versa.
  • LIST permission is independent of the path and permission fields in the request. By default, LIST objects permission will be provided on the root prefix ("/") of tenant or subtenant.
  • If path and permission, both fields are not provided in the request then “READ” permission will be given on the root prefix ("/") of tenant or subtenant.
  • An access credential requested for a given path also automatically gives access to all sub-paths of the requested path. For example, if an access token is requested for path “/a” and there are sub-paths “/a/b” and “/a/b/c”, the token allows to access those too.
  • 1024 characters are supported as maximum length for path field. If the path length is shorter, then multiple accesses on different prefixes are supported in the same request. Otherwise, a single path prefix of max 1024 characters will only be supported. The number of prefixes supported will depend on the path length/permission specified in the request.
POST /generateAccessCredentials
Content-Type: application/json

Request example:

{
  "durationSeconds": 900,
  "accesses": [
    {
      "path": "/myfolder1/folder2",
      "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
      "permissions": [
        "READ",
        "WRITE",
        "DELETE"
      ]
    }
  ]
}

Response example:

{
  "credentials": {
    "secretAccessKey": "AbCdEfGhIjKlMnOpQrStUvWxYz",
    "accessKeyId": "AbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYz",
    "sessionToken": "AbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYzAbCdEfGhIjKlMnOpQrStUvWxYz"
  },
  "storageAccount": "dlbucketname",
  "durationSeconds": 900,
  "accesses": [
    {
      "path": "/myfolder1/folder2",
      "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4",
      "permissions": [
        "READ",
        "WRITE",
        "DELETE"
      ]
    }
  ]
}

Any questions left?

Ask the community


Except where otherwise noted, content on this site is licensed under the MindSphere Development License Agreement.


Last update: April 11, 2022