Skip to content

Metadata Management from Integrated Data Lake

Note

Integrated Data Lake Service API version 4.*.*is available only for Virtual Private cloud.

Metadata management is a crucial aspect of data management within any Data Lake application. It involves the systematic organization, storage, retrieval, and maintenance of metadata, which is essentially data about data. This includes information about the characteristics, origin, usage, and relationships of the actual data stored in the Data Lake.

Metadata collection operations

Metadata collection is the systematic process of collecting metadata keys that manage all the relevant metadata information associated with the specific dataset or system.

Metadata collections are classified into following categories:

  • Global collection
  • Custom collection
  • Reserved collection

Creating a Metadata collection

This API can be used to create metadata collection. Global collections are available by default in the system and cannot be created.

Create a collection using the following endpoint:

POST/metadataCollections

Sample Request:

{
  "metadataCollectionId": "document_review",
  "label": "Document Review",
  "description": "usage and purpose of custom Metadata Collection"
}

Note

  • Users cannot create collection with name 'global or 'system' as it is reserved for system use.

Sample Response:

{
  "metadataCollectionId": "document_review",
  "label": "Document Reeview",
  "description": "usage and purpose of custom Metadata Collection",
  "status": "DRAFT",
  "etag": 1
}

Retrieving the list of Metadata collections

This API can be used to get the list of all the metadata collections. Global metadata collection (though default) is listed only when a metadata key is added into it.

Retrieve collection list using the below endpoint:

GET/metadataCollections

Sample Response:

{
  "metadataCollections": [
    {
      "metadataCollectionId": "document_review",
      "label": "Document Reeview",
      "description": "usage and purpose of custom Metadata Collection",
      "status": "DRAFT",
      "etag": 1
    }
  ],
  "page": {
    "size": 0,
    "totalElements": 0,
    "totalPages": 0,
    "number": 0
  }
}

Retrieving the Metadata collection details

This API can be used to get the details of the collection for the given id.

Note

With this endpoint only collection details are fetched, not the details related to the keys inside the collection.

Retrieve collection details using the below endpoint:

GET/metadataCollections/{metadataCollectionId}

Sample Response:

{
  "metadataCollectionId": "document_review",
  "label": "Document Reeview",
  "description": "usage and purpose of custom Metadata Collection",
  "status": "DRAFT",
  "etag": 1
}

Update the Metadata collection details

This API can be used to update the details of the collection for the given id.

Note

It is only possible to update the collections created by the user.

Update collection details using the below endpoint:

PATCH/metadataCollections/{metadataCollectionId}

Sample Request:

{
  "label": "Document Review",
  "description": "usage and purpose of custom Metadata Collection",
  "status": "DRAFT"
}

Note

Collection status can be updated only from DRAFT to PUBLISHED.

Sample Response:

{
  "metadataCollectionId": "document_review",
  "label": "Document Reeview",
  "description": "usage and purpose of custom Metadata Collection",
  "status": "DRAFT",
  "etag": 1
}

Deleting the Collection

This API can be used to delete the collection of the given id.

Delete the collection using the following endpoint:

DELETE/metadataCollections/{metadataCollectionId}

Note

  • It is not possible to delete Global collection.
  • It is only possible to delete the custom collection in draft status.

Metadata Keys Operations

Metadata keys are the unique identifiers used to represent specific attributes or characteristics associated with the data in a key-value pair structure.

Creating a Metadata Key

This API can be used to create metadata key. This API allows admin users to add metadata keys to metadata collection. By default, admin users can add keys directly to Global metadata collection. There are various configuration parameters available while defining a metadata key. These configuration parameters helps admin users to define keys to accept specific metadata values.

The configuration parameters are defined below:

  • Value Type: This is the data type that defines the type of metadata value which can be accepted for this key.
  • Field name and Field values: These are the additional details that admin users can use to put constraints on metadata values.
  • applyOn: This parameter is used to define where it should be applied i.e., objects (files) or folders.
  • Rule Key: This indicated if this key can be used to create rules.

Note

Minimum value for key should be 2 characters.

Recommendations:

  • It is advised to keep the key in DRAFT status when created. Many details of the key cannot be updated once key is PUBLISHED.

"applyOn" can be use to organize metadata base on various requirements

| Options | Use Case | | OBJECTS_ONLY | Can be used for any metadata that is applicable only at object level | | FOLDERS_ONLY | Can be used to set the metadata in the entire hierarchy which should not be changed at any level e.g. Business Sensitivity or Classification Levels.
To be avoided for the metadata which is updated regularly | | OBJECTS_AND_FOLDERS | Can be used where metadata is applicable on entire hierarchy but can be changed if required |

Create a collection using the following endpoint:

POST/metadataCollections/{metadataCollectionId}/keys

Sample Request:

{
  "key": "country_of_origin",
  "label": "Country of Origin",
  "description": "description of metadata key and its usage",
  "valueType": "String",
  "additionalDetails": [
    {
      "fieldName": "StringType",
      "fieldValues": [
        "ALPHABETIC"
      ]
    }
  ],
  "applyOn": "OBJECTS_ONLY",
  "isMandatory": true,
  "isRuleKey": false,
  "isSearchable": false,
  "status": "DRAFT"
}

Sample Response:

{
  "key": "country_of_origin",
  "label": "Country of Origin",
  "description": "description of metadata key and its usage",
  "valueType": "String",
  "additionalDetails": [
    {
      "fieldName": "StringType",
      "fieldValues": [
        "ALPHABETIC"
      ]
    }
  ],
  "applyOn": "OBJECTS_ONLY",
  "isMandatory": true,
  "isRuleKey": false,
  "isSearchable": false,
  "status": "DRAFT",
  "etag": 1
}

Retrieving the list of Metadata keys

This API can be used to get the list of all the metadata keys in the provided metadata collection.

Retrieve list of keys using the below endpoint:

GET /metadataCollections/{metadataCollectionId}/keys

Sample Response:

{
  "metadataKeys": [
    {
      "key": "country_of_origin",
      "label": "Country of Origin",
      "description": "description of metadata key and its usage",
      "valueType": "String",
      "additionalDetails": [
        {
          "fieldName": "StringType",
          "fieldValues": [
            "ALPHABETIC"
          ]
        }
      ],
      "applyOn": "OBJECTS_ONLY",
      "isMandatory": true,
      "isRuleKey": false,
      "isSearchable": false,
      "status": "DRAFT",
      "etag": 1
    }
  ],
  "page": {
    "size": 0,
    "totalElements": 0,
    "totalPages": 0,
    "number": 0
  }
}

Retrieving the key details

This API can be used to get the details of the defined key in the collection.

Retrieve key details using the below endpoint:

GET /metadataCollections/{metadataCollectionId}/keys/{metadataKey}

Sample Response:

{
  "key": "country_of_origin",
  "label": "Country of Origin",
  "description": "description of metadata key and its usage",
  "valueType": "String",
  "additionalDetails": [
    {
      "fieldName": "StringType",
      "fieldValues": [
        "ALPHABETIC"
      ]
    }
  ],
  "applyOn": "OBJECTS_ONLY",
  "isMandatory": true,
  "isRuleKey": false,
  "isSearchable": false,
  "status": "DRAFT",
  "etag": 1
}

Update the Metadata key details

This API allows the admin users to update metadata key configuration. Update details for metadata key will depend on the status of the key

Following parameter can be updated based on key status:

| Parameter | Draft Status | Published Status | | label & description | Can be updated | Can be updated | | Additional Details | Can modify or remove existing constraints, or add new constraints | Can change existing fieldValues to extend the already defined range e.g. minValue can be reduced or maxValue can be increased or new Enum value can be added.
Cannot remove any existing constraints | | isMandatory | Can be updated to true or false | Can be updated to true or false | | isRuleKey | Can be updated to true or false | Can be updated only if metadata collection in which this ruleKey is used is in DRAFT status | | applyOn | Can be updated to any of the available Enum values | Cannot be updated | | isSearchable | Can be updated to true or false | Cannot be updated |

Update key details using the below endpoint:

PATCH /metadataCollections/{metadataCollectionId}/keys/{metadataKey}

Sample Request:

{
  "label": "Business Sensitivity",
  "description": "description of metadata key and its usage",
  "additionalDetails": [
    {
      "fieldName": "StringType",
      "fieldValues": [
        "ALPHABETIC"
      ]
    }
  ],
  "isMandatory": true,
  "isRuleKey": false,
  "status": "DRAFT"
}

Sample Response:

{
  "key": "country_of_origin",
  "label": "Country of Origin",
  "description": "description of metadata key and its usage",
  "valueType": "String",
  "additionalDetails": [
    {
      "fieldName": "StringType",
      "fieldValues": [
        "ALPHABETIC"
      ]
    }
  ],
  "applyOn": "OBJECTS_ONLY",
  "isMandatory": true,
  "isRuleKey": false,
  "isSearchable": false,
  "status": "DRAFT",
  "etag": 1
}

Deleting the Keys

This API allows tenant admin to delete metadata key under the provided metadata collection provided in the request.

Note

It is only possible to delete the key, which is in Draft status.

Delete the collection using the following endpoint:

DELETE /metadataCollections/{metadataCollectionId}/keys/{metadataKey}

Metadata Rules Operations

Metadata Rules are a set of predefined conditions that determines the metadata tied to a custom collection that is applicable to Integrated Data Lake resources.

Create a Metadata Rule

This API creates metadata rules for metadata collection corresponding to a metadata key.

Note

  • Status of the rule is governed by collection status
  • Rule can be created for collection in DRAFT as well as PUBLISHED status

    • If collection is in DRAFT status, rule is not applied and collection attributes will not be available to enter metadata values
    • If collection is in PUBLISHED status, rule gets applied immediately

Recommendations:

  • It is advised to Keep the collection in DRAFT status while creating rule. Do not publish collection unless certain about rule key and its value used in the rule.

Create a metadata rule using the following endpoint:

POST /metadataCollections/{metadataCollectionId}/metadataRules

Sample Request:

{
  "name": "Document Review Rule",
  "key": "business_senstivity",
  "value": "STRICTLY_PRIVATE"
}

Sample Response:

{
  "id": "0860f696-af41-4d8d-a104-c1dd508de97a",
  "name": "Document Review Rule",
  "key": "business_senstivity",
  "value": "STRICTLY_PRIVATE",
  "etag": 1
}

Retrieving the list of Metadata rules

This API lists all the metadata rules associated with a collection.

Retrieve the list of all the metadata rules using the following endpoint:

GET /metadataCollections/{metadataCollectionId}/metadataRules

Sample Response:

{
  "metadataRules": [
    {
      "id": "0860f696-af41-4d8d-a104-c1dd508de97a",
      "name": "Document Review Rule",
      "key": "business_senstivity",
      "value": "STRICTLY_PRIVATE",
      "etag": 1
    }
  ],
  "page": {
    "size": 0,
    "totalElements": 0,
    "totalPages": 0,
    "number": 0
  }
}

Retrieving the metadata rule details

This API provides the metadata rule details for the given id.

Retrieve the metadata rule details using the following endpoint:

GET /metadataCollections/{metadataCollectionId}/metadataRules/{id}

Sample Response:

{
  "id": "0860f696-af41-4d8d-a104-c1dd508de97a",
  "name": "Document Review Rule",
  "key": "business_senstivity",
  "value": "STRICTLY_PRIVATE",
  "etag": 1
}

Updating the metadata rules

This API updates the metadata rule details for the given id. Rule can be updated only in DRAFT status.

Update the metadata rule details using the below endpoint:

PATCH /metadataCollections/{metadataCollectionId}/metadataRules/{id}

Sample Request:

{
  "name": "Document Review Rule",
  "key": "business_senstivity",
  "value": "STRICTLY_PRIVATE"
}

Sample Response:

{
  "id": "0860f696-af41-4d8d-a104-c1dd508de97a",
  "name": "Document Review Rule",
  "key": "business_senstivity",
  "value": "STRICTLY_PRIVATE",
  "etag": 1
}

Delete the metadata rule

This API deletes the metadata rule provided in the request.

Note

It is only possible to delete the rule, which is in Draft status.

Delete the collection using the following endpoint:

DELETE /metadataCollections/{metadataCollectionId}/metadataRules/{id}

Last update: December 19, 2023

Except where otherwise noted, content on this site is licensed under the Development License Agreement.