Data lake interface header file. More...

#include "mcl_data_lake/mcl_data_lake_object.h"
#include "mcl_data_lake/mcl_data_lake_configuration.h"
#include "mcl_core/mcl_list.h"

Include dependency graph for mcl_data_lake.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Typedefs
typedef struct mcl_data_lake_t	mcl_data_lake_t

Functions
MCL_DATA_LAKE_EXPORT mcl_error_t	mcl_data_lake_initialize (mcl_data_lake_configuration_t configuration, mcl_data_lake_t *data_lake)

MCL_DATA_LAKE_EXPORT mcl_error_t	mcl_data_lake_generate_upload_url (mcl_data_lake_t data_lake, mcl_data_lake_object_t object)

MCL_DATA_LAKE_EXPORT mcl_error_t	mcl_data_lake_generate_upload_urls (mcl_data_lake_t data_lake, mcl_list_t object_list)

MCL_DATA_LAKE_EXPORT mcl_error_t	mcl_data_lake_generate_upload_url_for_subtenant (mcl_data_lake_t data_lake, mcl_data_lake_object_t object, const char *subtenant_id)

MCL_DATA_LAKE_EXPORT mcl_error_t	mcl_data_lake_generate_upload_urls_for_subtenant (mcl_data_lake_t data_lake, mcl_list_t object_list, const char *subtenant_id)

MCL_DATA_LAKE_EXPORT mcl_error_t	mcl_data_lake_upload (mcl_data_lake_t data_lake, mcl_data_lake_object_t object)

MCL_DATA_LAKE_EXPORT void	mcl_data_lake_destroy (mcl_data_lake_t **data_lake)

Detailed Description

Data lake interface header file.

Agents can store structured or unstructured data in MindSphere Data Lake using MCL's data lake component.

This module enables the agent to initialize a data structure for MCL data lake component using mcl_data_lake_initialize function. Following initialization, a signed URL to upload data to can be generated using #mcl_data_lake_get_upload_url, mcl_data_lake_generate_upload_urls, mcl_data_lake_generate_upload_url_for_subtenant or mcl_data_lake_generate_upload_urls_for_subtenant functions. Then, data lake object can be uploaded to the signed URL using mcl_data_lake_upload function.

Data structure for the data lake component is destroyed using mcl_data_lake_destroy funtion.

Copyright: Copyright (C) 2019 Siemens Aktiengesellschaft.
All rights reserved.

Definition in file mcl_data_lake.h.

Typedef Documentation

typedef struct mcl_data_lake_t mcl_data_lake_t

Handle for data lake.

Definition at line 34 of file mcl_data_lake.h.

Function Documentation

MCL_DATA_LAKE_EXPORT void mcl_data_lake_destroy ( mcl_data_lake_t ** data_lake )

This function destroys data lake structure.

Parameters

[in] data_lake Data lake handle which is going to be destroyed.

Definition at line 239 of file data_lake.c.

References MCL_DEBUG_ENTRY, MCL_DEBUG_LEAVE, MCL_FREE, and MCL_NULL.

Referenced by mcl_data_lake_initialize().

Here is the caller graph for this function:

MCL_DATA_LAKE_EXPORT mcl_error_t mcl_data_lake_generate_upload_url	(	mcl_data_lake_t *	data_lake,
		mcl_data_lake_object_t *	object
	)

This function gets the signed URL to upload data to.

Parameters

[in]	data_lake	Data lake handle.
[in]	object	Data lake object.

Returns

MCL_OK in case of success.
MCL_DATA_LAKE_SIGNED_URL_GENERATION_FAIL if signed URL could not be retrieved for the object.
MCL_TRIGGERED_WITH_NULL in case data_lake or object is null.
MCL_INVALID_PARAMETER if path of object is not set.
MCL_OUT_OF_MEMORY in case there is not enough memory in the system to proceed.
MCL_COULD_NOT_RESOLVE_PROXY if the proxy host name can not be resolved.
MCL_COULD_NOT_RESOLVE_HOST if the remote host name can not be resolved.
MCL_SSL_HANDSHAKE_FAIL if a problem occurs during SSL handshake.
MCL_COULD_NOT_CONNECT if there is a connection error with the server.
MCL_SERVER_CERTIFICATE_NOT_VERIFIED if the server can not be verified.
MCL_IMPROPER_CERTIFICATE in case the certificate provided is in improper format and it can not be parsed.
#MCL_CA_CERTIFICATE_AUTHENTICATION_FAIL if the server certificate can not be authenticated by the root certificate.
MCL_NETWORK_SEND_FAIL in case of an error in sending data to network.
MCL_NETWORK_RECEIVE_FAIL in case of an error in receiving data from the network.
MCL_BAD_REQUEST if response status code of server is 400.
MCL_UNAUTHORIZED if response status code of server is 401.
MCL_SERVER_FAIL if response status code of server is 500.

Definition at line 71 of file data_lake.c.

References data_lake_processor_generate_upload_url(), mcl_core_get_last_access_token(), MCL_DEBUG_ENTRY, MCL_DEBUG_LEAVE, MCL_ERROR, MCL_FREE, MCL_NULL, MCL_OK, and MCL_TRIGGERED_WITH_NULL.

Here is the call graph for this function:

MCL_DATA_LAKE_EXPORT mcl_error_t mcl_data_lake_generate_upload_url_for_subtenant	(	mcl_data_lake_t *	data_lake,
		mcl_data_lake_object_t *	object,
		const char *	subtenant_id
	)

This function gets the signed URL for subtenant to upload data to.

Parameters

[in]	data_lake	Data lake handle.
[in]	object	Data lake object.
[in]	subtenant_id	Subtenant ID.

Returns

MCL_OK in case of success.
MCL_DATA_LAKE_SIGNED_URL_GENERATION_FAIL if signed URL could not be retrieved for the object.
MCL_TRIGGERED_WITH_NULL in case data_lake or object is null.
MCL_INVALID_PARAMETER if path of object is not set.
MCL_OUT_OF_MEMORY in case there is not enough memory in the system to proceed.
MCL_COULD_NOT_RESOLVE_PROXY if the proxy host name can not be resolved.
MCL_COULD_NOT_RESOLVE_HOST if the remote host name can not be resolved.
MCL_SSL_HANDSHAKE_FAIL if a problem occurs during SSL handshake.
MCL_COULD_NOT_CONNECT if there is a connection error with the server.
MCL_SERVER_CERTIFICATE_NOT_VERIFIED if the server can not be verified.
MCL_IMPROPER_CERTIFICATE in case the certificate provided is in improper format and it can not be parsed.
#MCL_CA_CERTIFICATE_AUTHENTICATION_FAIL if the server certificate can not be authenticated by the root certificate.
MCL_NETWORK_SEND_FAIL in case of an error in sending data to network.
MCL_NETWORK_RECEIVE_FAIL in case of an error in receiving data from the network.
MCL_BAD_REQUEST if response status code of server is 400.
MCL_UNAUTHORIZED if response status code of server is 401.
MCL_SERVER_FAIL if response status code of server is 500.

Definition at line 141 of file data_lake.c.

References data_lake_processor_generate_upload_url(), mcl_core_get_last_access_token(), MCL_DEBUG_ENTRY, MCL_DEBUG_LEAVE, MCL_ERROR, MCL_FREE, MCL_NULL, MCL_OK, and MCL_TRIGGERED_WITH_NULL.

Here is the call graph for this function:

MCL_DATA_LAKE_EXPORT mcl_error_t mcl_data_lake_generate_upload_urls	(	mcl_data_lake_t *	data_lake,
		mcl_list_t *	object_list
	)

This function gets the signed URLs for a list of objects to upload data to.

Parameters

[in]	data_lake	Data lake handle.
[in]	object_list	List of data lake objects.

Returns

MCL_OK in case of success (all data lake objects with non-NULL paths have their signed urls).
MCL_DATA_LAKE_SIGNED_URL_GENERATION_FAIL if signed URL could not be retrieved for all data lake objects with non-NULL paths.
MCL_TRIGGERED_WITH_NULL in case processor or object_list is null.
MCL_OUT_OF_MEMORY in case there is not enough memory in the system to proceed.
MCL_INVALID_PARAMETER if all paths are NULL.
MCL_COULD_NOT_RESOLVE_PROXY if the proxy host name can not be resolved.
MCL_COULD_NOT_RESOLVE_HOST if the remote host name can not be resolved.
MCL_SSL_HANDSHAKE_FAIL if a problem occurs during SSL handshake.
MCL_COULD_NOT_CONNECT if there is a connection error with the server.
MCL_SERVER_CERTIFICATE_NOT_VERIFIED if the server can not be verified.
MCL_IMPROPER_CERTIFICATE in case the certificate provided is in improper format and it can not be parsed.
#MCL_CA_CERTIFICATE_AUTHENTICATION_FAIL if the server certificate can not be authenticated by the root certificate.
MCL_NETWORK_SEND_FAIL in case of an error in sending data to network.
MCL_NETWORK_RECEIVE_FAIL in case of an error in receiving data from the network.
MCL_BAD_REQUEST if response status code of server is 400.
MCL_UNAUTHORIZED if response status code of server is 401.
MCL_SERVER_FAIL if response status code of server is 500.

Definition at line 106 of file data_lake.c.

References data_lake_processor_generate_upload_urls(), mcl_core_get_last_access_token(), MCL_DEBUG_ENTRY, MCL_DEBUG_LEAVE, MCL_ERROR, MCL_FREE, MCL_NULL, MCL_OK, and MCL_TRIGGERED_WITH_NULL.

Here is the call graph for this function:

MCL_DATA_LAKE_EXPORT mcl_error_t mcl_data_lake_generate_upload_urls_for_subtenant	(	mcl_data_lake_t *	data_lake,
		mcl_list_t *	object_list,
		const char *	subtenant_id
	)

This function gets the signed URLs for subtenant for a list of objects to upload data to.

Parameters

[in]	data_lake	Data lake handle.
[in]	object_list	List of data lake objects.
[in]	subtenant_id	Subtenant ID.

Returns

MCL_OK in case of success (all data lake objects with non-NULL paths have their signed urls).
MCL_DATA_LAKE_SIGNED_URL_GENERATION_FAIL if signed URL could not be retrieved for all data lake objects with non-NULL paths.
MCL_TRIGGERED_WITH_NULL in case processor or object_list is null.
MCL_OUT_OF_MEMORY in case there is not enough memory in the system to proceed.
MCL_INVALID_PARAMETER if all paths are NULL.
MCL_COULD_NOT_RESOLVE_PROXY if the proxy host name can not be resolved.
MCL_COULD_NOT_RESOLVE_HOST if the remote host name can not be resolved.
MCL_SSL_HANDSHAKE_FAIL if a problem occurs during SSL handshake.
MCL_COULD_NOT_CONNECT if there is a connection error with the server.
MCL_SERVER_CERTIFICATE_NOT_VERIFIED if the server can not be verified.
MCL_IMPROPER_CERTIFICATE in case the certificate provided is in improper format and it can not be parsed.
#MCL_CA_CERTIFICATE_AUTHENTICATION_FAIL if the server certificate can not be authenticated by the root certificate.
MCL_NETWORK_SEND_FAIL in case of an error in sending data to network.
MCL_NETWORK_RECEIVE_FAIL in case of an error in receiving data from the network.
MCL_BAD_REQUEST if response status code of server is 400.
MCL_UNAUTHORIZED if response status code of server is 401.
MCL_SERVER_FAIL if response status code of server is 500.

Definition at line 176 of file data_lake.c.

References data_lake_processor_generate_upload_urls(), mcl_core_get_last_access_token(), MCL_DEBUG_ENTRY, MCL_DEBUG_LEAVE, MCL_ERROR, MCL_FREE, MCL_NULL, MCL_OK, and MCL_TRIGGERED_WITH_NULL.

Here is the call graph for this function:

MCL_DATA_LAKE_EXPORT mcl_error_t mcl_data_lake_initialize	(	mcl_data_lake_configuration_t *	configuration,
		mcl_data_lake_t **	data_lake
	)

This function creates and initializes a data structure of type mcl_data_lake_t.

Parameters

[in]	configuration	Data structure holding data lake configuration parameters.
[out]	data_lake	Data lake handle which is going to be initialized.

Returns

MCL_OK in case of success.
MCL_TRIGGERED_WITH_NULL in case configuration or data_lake is null.
MCL_INVALID_PARAMETER if provided configuration is invalid.
MCL_OUT_OF_MEMORY in case there is not enough memory in the system to proceed.
MCL_NOT_ONBOARDED if the agent is not onboarded using core.
MCL_NO_ACCESS_TOKEN_EXISTS if core does not have an access token.
MCL_FAIL in case of an internal error in MCL.

Definition at line 15 of file data_lake.c.

References data_lake_configuration_validate(), MCL_ASSERT_NOT_NULL, mcl_core_get_client_id(), mcl_core_get_http_client(), mcl_data_lake_destroy(), MCL_DEBUG_ENTRY, MCL_DEBUG_LEAVE, MCL_FUNCTION_LEAVE_LABEL, mcl_http_client_add_certificate(), MCL_NEW, MCL_NULL, MCL_OK, and MCL_OUT_OF_MEMORY.

Here is the call graph for this function:

MCL_DATA_LAKE_EXPORT mcl_error_t mcl_data_lake_upload	(	mcl_data_lake_t *	data_lake,
		mcl_data_lake_object_t *	object
	)

This function uploads data object to the signed URL.

Parameters

[in]	data_lake	Data lake handle.
[in]	object	Data lake object.

Returns

MCL_OK in case of success.
MCL_OUT_OF_MEMORY if there is not enough memory in the system to proceed.
MCL_TRIGGERED_WITH_NULL in case data_lake or object is null.
MCL_INVALID_PARAMETER in case item has one or more missing mandatory parameter.
MCL_COULD_NOT_RESOLVE_PROXY in case the proxy host name can not be resolved.
MCL_COULD_NOT_RESOLVE_HOST in case the remote host name can not be resolved.
MCL_SSL_HANDSHAKE_FAIL in case a problem occurs during SSL handshake.
MCL_COULD_NOT_CONNECT in case there is a connection error.
MCL_SERVER_CERTIFICATE_NOT_VERIFIED in case the server can not be verified.
MCL_IMPROPER_CERTIFICATE in case the certificate provided is in improper format and it can not be parsed.
#MCL_CA_CERTIFICATE_AUTHENTICATION_FAIL in case the server certificate can not be authenticated by the root certificate.
MCL_NETWORK_SEND_FAIL in case of an error in sending data to network.
MCL_NETWORK_RECEIVE_FAIL in case of an error in receiving data from the network.
MCL_BAD_REQUEST if the server returns HTTP 400 status code.
MCL_UNAUTHORIZED if the server returns HTTP 401 status code.
MCL_SERVER_FAIL if the server returns HTTP 500 status code.
MCL_FAIL in case of an internal error in MCL.

Definition at line 211 of file data_lake.c.

References data_lake_processor_upload(), MCL_DEBUG_ENTRY, MCL_DEBUG_LEAVE, MCL_ERROR, MCL_NULL, MCL_OK, and MCL_TRIGGERED_WITH_NULL.

Here is the call graph for this function:

Typedefs

Functions

Detailed Description

Typedef Documentation

Function Documentation