Deploying and Upgrading LPC Environment¶
This section explains the process for upgrading and deploying the LPC system in a customer's environment. The goal is to provide an organized framework for performing an upgrade.
Prerequisites¶
The following requirements must be met before proceeding with the deployment:
- Kubernetes Version: v1.26.13
- Longhorn Version: v1.4.2
- Operating System Version: Rocky Linux 8 / CentOS Linux 8 / RedHat Linux 8
Upgrade preparation¶
- Prepare "Helm Charts" and "ArgoCD/Deploy" scripts for the customer environment.
- Prepare "jumphost", VPN or network deployment admission conditions.
- Backup of current configurations.
- Validate system status and verify prerequisites.
Upgrading dependencies¶
Component | Task Details |
---|---|
Kafka Upgrade | Upgrade PVC name, download, and edit Kafka charts. |
Upgrade Kafka version. | |
PGO for UTS | Separate UTS database. |
(No resources to perform this, ignore for now.) | |
PGO for IoT | Separate read/write operations for IoT. |
RabbitMQ Configuration | Delete and unbind specified queues. |
Restarting services¶
Restart necessary services that use Kafka, including the "Kafka connector" and "InsightsHub services".
Upgrading Insights Hub services¶
Upgrade "InsightsHub services" and ensure that services are running correctly and perform troubleshooting if needed.
Testing Post-Upgrade¶
Perform tests to ensure that the upgrade is successful, focus on system stability, data integrity and functional validation of Insights Hub.
Preparation and environment validation¶
- Specify the required CPU, memory and disk space resources for the system.
- Ensure that the Helm Charts and ArgoCD/Deploy scripts are correctly set up for the customer environment.
- Check that ArgoCD auto-sync is disabled.
- Check that there is sufficient memory and CPU to support the upgrade process. Example: PGO: + (16C/32G)*3
- Verify that images are pre-pulled.
Environment validation¶
Perform the necessary validation steps, including checks for network connectivity, DNS configuration and time synchronization:
- Verify network connectivity from Siemens site to the customer Kubernetes (k8s) API server.
- Ensure customer site can access the InsightsHub image registry.
- Validate time synchronization across all Kubernetes nodes.
- Validate the running status of all Kubernetes (k8s) pods.
- Check the InsightsHub UI status.
Backup instructions¶
This explains how to back up existing data and configurations before starting the deployment process.
- Backup
PGO
data. - Backup
CoreDNS configmap
configuration. - Backup
resolv.conf
configuration. - Backup
ingress controller configmap
configuration.
Backend services upgrade details¶
Overview of required upgrades¶
Identify the backend services for InsightsHub that require upgrades, including their version changes.
Service Name | Version Change | Comments |
---|---|---|
bitnami/kafka | 20.0.0 -> 28.2.5 | Restart all services using Kafka. |
uts/pgo | pgo: 5.3.0 crunchy-postgres:ubi8-14.6-2 | UTS database will be separated. |
iot/pgo | pgo: 5.3.0 crunchy-postgres:ubi8-14.6-2 | IoT read/write operations are separated. |
Kafka Upgrade¶
Upgrade bitnami Kafka from version 20.0.0 to 28.2.5.
PGO Upgrade¶
Node Expansion for PGO¶
Node Name | Current CPU | Current Memory | Future CPU | Future Memory |
---|---|---|---|---|
pgo-node1 | 16 | 32 | 32 | 64 |
pgo-node2 | 16 | 32 | 32 | 64 |
pgo-node3 | 16 | 32 | 32 | 64 |
Migrating UTS Database to new PGO cluster¶
- Set up a new PGO cluster (
mdsp-pg-uts
) on the same nodes as the existingmdsp-pg-all
cluster. -
Deploy UTS services using the new UTS database endpoint.
Note: The credentials for the new UTS PGO cluster are not yet updated, hence, the UTS services will not connect to the new cluster at this stage.
-
Backup the UTS databases from the
mdsp-pg-all
PGO cluster, including the following databases:- utsdb
- utsdatalakedb
- ruleenginedb
-
Restore the backup databases into the
mdsp-pg-uts
PGO cluster.Note: The restoration time depends on the size of the databases.
-
Update the
postgres
user password in themdsp-pg-uts
PGO cluster. After the password is updated, the UTS services can connect to the new cluster. - Restart the UTS services to ensure the new database credentials are properly applied.
Infra secrets for new UTS/PGO¶
Secret Name | Variable |
---|---|
mdsp-secret-infra-credentials | uts_pg_sql_host |
mdsp-secret-infra-credentials | uts_pg_sql_password |
mdsp-secret-infra-credentials | uts_pg_sql_port |
mdsp-secret-infra-credentials | uts_pg_sql_user |
Infra Secrets for new IoT/PGO¶
Secret Name | Variable |
---|---|
mdsp-secret-infra-credentials | iot_pg_sql_ro_user |
mdsp-secret-infra-credentials | iot_pg_sql_ro_host |
mdsp-secret-infra-credentials | iot_pg_sql_ro_password |
mdsp-secret-infra-credentials | iot_pg_sql_rw_user |
mdsp-secret-infra-credentials | iot_pg_sql_rw_host |
mdsp-secret-infra-credentials | iot_pg_sql_rw_password |
Configuring updates¶
Service Name | Configuration Updates |
---|---|
bitnami/rabbitmq | Delete specific queues and unbind exchanges |
Operations¶
The queue management and exchange unbinding components are detailed below:
Action | Entity Name |
---|---|
Delete Queue | asset-notifications-topic.iot-bulk-data-stream |
asset-notifications-topic.iot-bulk-data-svc-asset-cache | |
notification-job-exchange.notification-job-queue | |
Unbind Exchanges | conn -> provisioning-notification-topic.dpmservice |
conn -> provisioning-notification-topic.emmservice |
Deprecated configurations¶
Note
Identify any deprecated configurations that need to be removed, along with cleanup instructions.
Updating business services¶
This section explains the steps required to upgrade InsightsHub's business services. It includes Helm chart upgrades, rolling updates for workloads, pod restarts and applying new configurations.
- Upgrade core services
- Upgrade IOT services
- Upgrade advanced services
- Upgrade connection services
- Upgrade other services
- Validate and fix issues in affected environments (
faw-vw,sunny-optical,maco…
)
Verifying Post-Upgrade checklist¶
Provide a checklist to validate that all services are functioning correctly after the update:
- Run the Docker suite.
- Validate tenant login on the Insights Hub UI.
- Review the Docker suite running result report.
Release Notes¶
Refer to the official documentation for detailed release notes: Insights Hub Release Notes.