Path B: Kubernetes and Telemetry¶

Overview¶

Use this deployment path to provision a service Kubernetes cluster and deploy Omnia Telemetry without deploying Slurm.

The workflow prepares the Omnia Infrastructure Manager (OIM), synchronizes catalog content with Repository Manager, and builds the Kubernetes node images with Image Build Manager. After you provide a reviewed PXE mapping, Orchestrator provisions the cluster and generates the inventory required by Telemetry. The final stages deploy the enabled telemetry components and verify the Kubernetes cluster and Telemetry deployment.

Kubernetes and Telemetry deployment workflow¶

Required module sequence and output handoffs

Start on the OIM

Configure and set up the OIM

omnia.env → omnia.sh --setup-venv

Learn more: OIM setup >>

Synchronize catalog content

Make catalog repositories available for image builds

Learn more: Repository Manager >>

Build Kubernetes node images

Create images for the selected Kubernetes functional groups

Learn more: Build OS images >>

Provide the PXE mapping

OME Discovery or a manual CSV

Learn more: Create the PXE mapping >>

Provision service Kubernetes

Provision the cluster and generate its inventory

Learn more: Deploy Kubernetes >>

Select and deploy telemetry sources

Deploy the enabled sources and Kubernetes sinks

Learn more: Deploy Telemetry >>

Verify Kubernetes and Telemetry

kubectl get nodes and telemetry pod state

Learn more: Kubernetes >> · Telemetry >>

Kubernetes telemetry platform ready

Prerequisites¶

Use an Omnia source checkout on the OIM.
For the Telemetry module, use Python 3.12 or later, Ansible 2.20 or later, and the documented validated RHEL 10.0 baseline on the OIM. See the Operating Systems Matrix.
Set SYSTEM_ADMIN_NIC_IPV4 in src/main/omnia.env to an IPv4 address assigned to an OIM interface. Review the project name, shared data path, hostname, domain, Omnia version, and catalog path in the same file.
Select a catalog whose functional layers include service Kubernetes. For an offline Telemetry deployment, ensure its package manifest artifacts are available through the Pulp repository configured in telemetry_packages.yml.
Prepare the admin-network values required by Orchestrator and a shared NFS mount for Kubernetes configuration and Telemetry packages.
Prepare at least one supported Telemetry source: iDRAC, PowerScale, UFM, VAST, or OME. This path intentionally excludes LDMS because the source requires Slurm nodes in the Orchestrator inventory.
For OME discovery, have the OME address and credentials available. For automated PXE boot, the mapping must contain the applicable BMC information and Orchestrator must be able to collect the BMC credentials.

Procedure¶

1. Configure and set up the OIM¶

Edit the environment configuration from the Omnia source tree:
Run on: OIM host
```
cd src/main
vi omnia.env
```
Create the shared virtual environment, install module dependencies, stage the module input templates, and copy the catalog samples:
Run on: OIM host
```
./omnia.sh --setup-venv
```
This command runs each selected module's domain-init.sh. Do not run the individual initialization scripts again unless a module was skipped or setup used --deps-only.

Load the installed environment and activate the shared virtual environment in the current shell:

Run on: OIM host

source /etc/profile.d/omnia-env.sh
source "$OMNIA_DATA_PATH/activate-omnia.sh"
repo_manager_path="${OMNIA_DATA_PATH}/repo_manager"
image_build_manager_path="${OMNIA_DATA_PATH}/image_build_manager"
discovery_path="${OMNIA_DATA_PATH}/discovery"
orchestrator_path="${OMNIA_DATA_PATH}/orchestrator"
telemetry_path="${OMNIA_DATA_PATH}/telemetry"

Run these commands in each new shell before using the paths based on environment variables in this guide.

For all environment and setup options, see Configure the environment and Set up the OIM.

2. Configure and run Repository Manager¶

Review these staged inputs:

$repo_manager_path/input/$OMNIA_PROJECT_NAME/repo_manager_config.yml
$repo_manager_path/input/$OMNIA_PROJECT_NAME/repo_manager_endpoint_config.yml
The catalog JSON identified by CATALOG_FILE_PATH

Ensure the selected catalog includes the service Kubernetes functional layers and that each selected package source resolves through the configured RPM repository, container registry, or artifact URL.

At minimum, this Kubernetes and Telemetry path requires both of these x86_64 functional layers for the selected operating-system version:

Required functional layer	Required Kubernetes components
`service_kube_control_plane_rhel_<major>_<minor>_x86_64`	`service_k8s_common_group`, `service_k8s_telemetry_group`, `service_k8s_cluster_group`, and `service_kube_control_plane_group`
`service_kube_node_rhel_<major>_<minor>_x86_64`	`service_k8s_common_group`, `service_k8s_telemetry_group`, and `service_kube_node_group`

For RHEL 10.x, replace <minor> with the minor version supplied by the selected catalog and use that same version in both functional-layer names. For example, a RHEL 10.2 catalog supplies service_kube_control_plane_rhel_10_2_x86_64 and service_kube_node_rhel_10_2_x86_64. Do not create a catalog containing only the groups shown in this table; the selected catalog must include the complete base OS, dependency, and package definitions.

Warning

Without the service Kubernetes functional layers, Orchestrator does not enable service Kubernetes or generate the Kubernetes inventory required by this Telemetry deployment path.

Verify the selected catalog before continuing:

Run on: OIM host

python3 - "$CATALOG_FILE_PATH" <<'PY'
import json
import sys

try:
    with open(sys.argv[1], encoding="utf-8") as catalog_file:
        layers = json.load(catalog_file)["catalog"]["functionallayer"]
    for layer in layers:
        print(f"{layer['name']}\t{','.join(layer['components'])}")
except (OSError, json.JSONDecodeError, KeyError, TypeError) as error:
    print(f"Catalog validation failed: {error}", file=sys.stderr)
    raise SystemExit(1)
PY

Run the complete standard Repository Manager flow:
Using omnia.sh (recommended)Using ansible-playbook
Run on: OIM host
cd <OMNIA_SOURCE_PATH>/src/main ./omnia.sh --run repo_manager
Run on: OIM host
source /opt/omnia/activate-omnia.sh cd <OMNIA_SOURCE_PATH>/src/repo_manager/playbooks ansible-playbook repo_manager.yml
The flow validates the environment and inputs, collects or reuses credentials, deploys Pulp, synchronizes the selected content, and writes:
```
$repo_manager_path/output/$OMNIA_PROJECT_NAME/repo_status.yml
```
Do not continue until overall_status is success.

For the configuration and credential procedure, see Create Local Repositories.

3. Configure and run Image Build Manager¶

Review the staged image_build_config.yml under:
```
$image_build_manager_path/input/$OMNIA_PROJECT_NAME/
```
Its repo_manager_output_path must identify the successful repo_status.yml. With functional_groups_source: "catalog", the Image Build Manager resolves the image package sets from CATALOG_FILE_PATH. With functional_groups_source: "config", also configure package_groups.yml in the same project input directory.
Run the complete standard image-build flow:
Using omnia.sh (recommended)Using ansible-playbook
Run on: OIM host
cd <OMNIA_SOURCE_PATH>/src/main ./omnia.sh --run image_build_manager
Run on: OIM host
source /opt/omnia/activate-omnia.sh cd <OMNIA_SOURCE_PATH>/src/image_build_manager/playbooks ansible-playbook image_build_manager.yml
The flow validates the configuration, collects or reuses the applicable S3 and aarch64 credentials, prepares MinIO when selected, deploys the local registry, builds the selected functional-group images, and writes:
```
$image_build_manager_path/output/$OMNIA_PROJECT_NAME/build_status.yml
```
Confirm that overall_status is success and that every service Kubernetes functional group used in the PXE mapping has a corresponding image.

For configuration, build modes, and direct playbook alternatives, see Build Images.

4. Provide the PXE mapping¶

Choose one method. Orchestrator consumes the reviewed file as $orchestrator_path/input/$OMNIA_PROJECT_NAME/pxe_mapping_file.csv.

Discover nodes through OMECreate the mapping manually

Configure discovery_config.yml and network_spec.yml under $discovery_path/input/$OMNIA_PROJECT_NAME/. Set ome_ip to a valid, non-loopback OME IPv4 address.

Run Discovery:

Using omnia.sh (recommended)Using ansible-playbook

Run on: OIM host

cd <OMNIA_SOURCE_PATH>/src/main
./omnia.sh --run discovery

Run on: OIM host

source /opt/omnia/activate-omnia.sh
cd <OMNIA_SOURCE_PATH>/src/discovery/playbooks
ansible-playbook discovery.yml

Review the timestamped mapping and discovery report under $discovery_path/output/$OMNIA_PROJECT_NAME/. Then copy the latest mapping to the Orchestrator input directory. With the installed environment active, run:
Run on: OIM host
```
cp "$discovery_path/output/$OMNIA_PROJECT_NAME/bmc_pxe_mapping_file.csv" \
  "$orchestrator_path/input/$OMNIA_PROJECT_NAME/pxe_mapping_file.csv"
```

Discovery intentionally leaves this handoff to the operator so that node hostnames, functional groups, and group assignments can be reviewed before provisioning.

Edit the staged Orchestrator mapping directly:

Run on: OIM host

vi "$orchestrator_path/input/$OMNIA_PROJECT_NAME/pxe_mapping_file.csv"

Preserve the source-defined header:

FUNCTIONAL_GROUP_NAME,GROUP_NAME,SERVICE_TAG,PARENT_SERVICE_TAG,HOSTNAME,ADMIN_MAC,ADMIN_IP,BMC_MAC,BMC_IP,IB_NIC_NAME,IB_IP

Assign the intended control-plane and worker nodes to functional groups beginning with service_kube_control_plane and service_kube_node. Do not add Slurm functional groups for this deployment path.

The following example shows RHEL 10.0 selected in the catalog. When the catalog selects another supported RHEL 10.x minor version, replace every _rhel_10_0_ segment with the matching catalog version. For example, a RHEL 10.2 selection uses _rhel_10_2_:

Example: pxe_mapping_file.csv

FUNCTIONAL_GROUP_NAME,GROUP_NAME,SERVICE_TAG,PARENT_SERVICE_TAG,HOSTNAME,ADMIN_MAC,ADMIN_IP,BMC_MAC,BMC_IP,IB_NIC_NAME,IB_IP
service_kube_control_plane_rhel_10_0_x86_64,grp3,KUBE001,,nid001,02:00:00:00:11:01,172.16.107.61,02:00:00:00:12:01,172.17.107.61,,
service_kube_control_plane_rhel_10_0_x86_64,grp3,KUBE002,,nid002,02:00:00:00:11:02,172.16.107.62,02:00:00:00:12:02,172.17.107.62,,
service_kube_control_plane_rhel_10_0_x86_64,grp3,KUBE003,,nid003,02:00:00:00:11:03,172.16.107.63,02:00:00:00:12:03,172.17.107.63,,
service_kube_node_rhel_10_0_x86_64,grp4,KUBE004,,nid004,02:00:00:00:11:04,172.16.107.64,02:00:00:00:12:04,172.17.107.64,,
service_kube_node_rhel_10_0_x86_64,grp4,KUBE005,,nid005,02:00:00:00:11:05,172.16.107.65,02:00:00:00:12:05,172.17.107.65,,

Use one Kubernetes minor version

Use the single RHEL 10.x minor version selected by the catalog for all Kubernetes control-plane and worker nodes. Do not combine different RHEL minor versions in the same Kubernetes cluster. Replace every sample service tag, MAC address, IP address, and hostname with values from the target servers. Keep the exact 11-column header; leave optional fields empty with consecutive commas. Leave PARENT_SERVICE_TAG, IB_NIC_NAME, and IB_IP empty when they are not used. Use unique lowercase hostnames without a domain suffix; when dns_enabled is true, use nid001 through nid999. Ensure the admin addresses belong to a configured admin subnet and every functional group exists in the selected catalog and successful image-build output.

For the complete mapping schema and OME procedure, see Discover Nodes and Create a Mapping File.

5. Configure and run Orchestrator¶

Review the staged files under $orchestrator_path/input/$OMNIA_PROJECT_NAME/:

Input	Kubernetes requirement
`orchestrator_config.yml`	Confirm the mapping, Repository Manager, Image Build Manager, catalog, and PXE-boot settings.
`network_spec.yml`	Configure the OIM interface, admin subnet, DHCP range, router, and any optional additional network.
`omnia_config.yml`	Select exactly one `service_k8s_cluster` entry with `deployment: true`; configure its cluster networks, CNI, and storage name. Set `enable_powerscale_csi: true` only when CSI is required; both CSI file paths then become mandatory.
`high_availability_config.yml`	Provide a `service_k8s_cluster_ha` entry whose `cluster_name` matches the selected Kubernetes cluster.
`storage_config.yml`	Define the NFS mount named by `nfs_storage_name` and make it writable from the OIM where configured.
`pxe_mapping_file.csv`	Assign the intended nodes to service Kubernetes functional groups and ensure corresponding images exist in `build_status.yml`.
`security_config.yml`	Configure this file when the selected catalog enables OpenLDAP.
`additional_cloud_init.yml` (optional)	Add validated common and per-functional-group `write_files` and `runcmd` directives during provisioning. Set `additional_cloud_init_config_file` in `orchestrator_config.yml` to this file's absolute path to enable it.
`set_pxe_boot_config.yml` (optional)	Override server restart behavior, the PXE boot mode and target, and node-registration verification timing. When omitted, Orchestrator uses the documented defaults.

Orchestrator derives Kubernetes support and cluster OS metadata from the catalog.

Run the complete standard Orchestrator flow:
Using omnia.sh (recommended)Using ansible-playbook
Run on: OIM host
cd <OMNIA_SOURCE_PATH>/src/main ./omnia.sh --run orchestrator
Run on: OIM host
source /opt/omnia/activate-omnia.sh cd <OMNIA_SOURCE_PATH>/src/orchestrator/playbooks ansible-playbook orchestrator.yml
The untagged flow performs prechecks, collects or reuses credentials, prepares OpenCHAMI and catalog-selected services, provisions the service Kubernetes groups, validates provisioning, and performs iDRAC PXE boot when enable_pxe_boot: true. Do not run a second PXE-boot command after this flow unless you intentionally need to repeat that operation.
Confirm that Orchestrator generated the inventory Telemetry consumes:
```
$orchestrator_path/output/$OMNIA_PROJECT_NAME/orchestrator_inventory.yml
```
The file must contain kube_vip_group and populated functional groups beginning with service_kube_control_plane and service_kube_node.

For the detailed Kubernetes and provisioning settings, see Deploy Service Kubernetes, Configure Kubernetes HA, Configure Storage, and Provision Nodes.

6. Configure and run Telemetry¶

Review all three staged files under $telemetry_path/input/$OMNIA_PROJECT_NAME/:

Input	Kubernetes-only requirement
`telemetry_config.yml`	Set `cluster_inventory` to Orchestrator's generated inventory. Enable only sources available in this environment and configure their source-specific values.
`telemetry_storage_config.yml`	Size the replicas, CPU, memory, and persistent storage for the selected sinks, sources, and bridges.
`telemetry_packages.yml`	Select `online` or `offline`, configure the repository URL when offline, confirm `k8s_cluster_mount`, and review package and image references.

In telemetry_config.yml, use the generated Orchestrator inventory. Resolve its absolute path for the active project:
Run on: OIM host
```
printf '%s\n' "$orchestrator_path/output/$OMNIA_PROJECT_NAME/orchestrator_inventory.yml"
```
Copy the printed path into the configuration:
telemetry_config.yml
```
cluster_inventory: "<absolute path printed above>"
```
Disable LDMS and its Vector bridge because this path has no Slurm nodes:
telemetry_config.yml — edit the existing keys
```
telemetry_sources:
  ldms:
    metrics_enabled: false

telemetry_bridges:
  vector_ldms:
    metrics_enabled: false
```
Do not replace the full source file with this excerpt. Retain every key and set the metrics_enabled and logs_enabled fields to false for every other source that is not available or not configured. When OME metrics or logs are disabled, disable the corresponding vector_ome fields as well.
Complete the values required by each enabled source. In particular, when iDRAC is enabled, set idrac_telemetry_configurations.bmc_group_data_path to the bmc_group_data.csv produced by Orchestrator. Resolve the path for the active project and copy the printed value into that field:
Run on: OIM host
```
printf '%s\n' "$orchestrator_path/output/$OMNIA_PROJECT_NAME/bmc_group_data.csv"
```
The Telemetry credential role creates encrypted telemetry_credentials.yml and prompts only for empty credentials needed by the enabled sources.
In telemetry_packages.yml, keep k8s_cluster_mount aligned with the NFS mount configured for service Kubernetes. For the default offline mode, set repo_url to the Pulp content base that contains the packages named in the Telemetry package manifest. For online mode, leave repo_url empty.
Run the Telemetry environment precheck:
Using omnia.sh (recommended)Using ansible-playbook
Run on: OIM host
cd <OMNIA_SOURCE_PATH>/src/main ./omnia.sh --run telemetry --tags precheck
Run on: OIM host
source /opt/omnia/activate-omnia.sh cd <OMNIA_SOURCE_PATH>/src/telemetry/playbooks ansible-playbook telemetry.yml --tags precheck
The precheck validates access to the Kubernetes VIP, control-plane and worker readiness, non-Telemetry pod health, and the applicable PowerScale prerequisites. With LDMS disabled, its Slurm checks are skipped.
Run the complete standard Telemetry flow:
Using omnia.sh (recommended)Using ansible-playbook
Run on: OIM host
cd <OMNIA_SOURCE_PATH>/src/main ./omnia.sh --run telemetry
Run on: OIM host
source /opt/omnia/activate-omnia.sh cd <OMNIA_SOURCE_PATH>/src/telemetry/playbooks ansible-playbook telemetry.yml
The untagged flow validates the three runtime input files, deploys the sinks required by the selected collection targets, deploys each enabled source and bridge, checks pod state, and writes:
```
$telemetry_path/output/$OMNIA_PROJECT_NAME/telemetry_status.yml
```

For source-specific configuration and verification guides, see the Telemetry landing page.

Verification¶

Confirm that all four module contracts report success:

Run on: OIM host

grep '^overall_status:' "$repo_manager_path/output/$OMNIA_PROJECT_NAME/repo_status.yml"
grep '^overall_status:' "$image_build_manager_path/output/$OMNIA_PROJECT_NAME/build_status.yml"
grep '^overall_status:' "$orchestrator_path/output/$OMNIA_PROJECT_NAME/orchestrator_status.yml"
grep '^overall_status:' "$telemetry_path/output/$OMNIA_PROJECT_NAME/telemetry_status.yml"

Review the provisioning summary and generated inventory:

Run on: OIM host

cat "$orchestrator_path/output/$OMNIA_PROJECT_NAME/provisioning_report.yml"
cat "$orchestrator_path/output/$OMNIA_PROJECT_NAME/orchestrator_inventory.yml"

On the first Kubernetes control-plane node, confirm node and workload state:
Run on: first Kubernetes control-plane node
```
kubectl get nodes -o wide
kubectl get pods --all-namespaces -o wide
```
All mapped nodes should appear. Workloads should reach Running or Completed.
On the Kubernetes VIP, inspect the Telemetry namespace:
Run on: Kubernetes VIP
```
kubectl get pods -n telemetry
```
In telemetry_status.yml, confirm overall_status: success and verify that every requested sink, source, and bridge is deployed. Components disabled in telemetry_config.yml are reported as skipped.

Next steps¶

Use the source-specific guides on the Telemetry landing page to configure and verify additional available data sources.
Export Kafka connection details when an external producer or consumer needs access.
Export VictoriaMetrics connection details or VictoriaLogs connection details for external queries and integrations.
Deploy the PowerScale CSI driver when enable_powerscale_csi: true is set on the deployed service_k8s_cluster entry in omnia_config.yml and its input files have been prepared. The catalog supplies the required artifacts but does not enable the driver.
Use Add Nodes for supported Kubernetes node additions with matching source templates and image artifacts.
Follow Full Deployment when adding a Slurm cluster and LDMS to the environment.

Troubleshooting¶

If a later module reports a missing upstream contract, verify that the preceding status file exists and has overall_status: success.
If Kubernetes configuration is skipped, verify that the catalog contains a service Kubernetes functional layer and that the mapping includes both service_kube_control_plane and service_kube_node functional groups.
If image validation fails, compare every mapping functional group with functional_group_images in build_status.yml.
If physical nodes do not PXE boot, confirm enable_pxe_boot: true, review failed_nodes.json, and inspect orchestrator_status.yml in the Orchestrator project output directory.
If Telemetry cannot resolve the Kubernetes VIP, verify that cluster_inventory names the generated orchestrator_inventory.yml and that the file contains all.children.kube_vip_group.hosts.
If Telemetry input validation fails, correct all reported schema and cross-field errors across the three Telemetry input files. Source and bridge enablement must agree; for example, a Vector-OME metrics bridge requires OME metrics to be enabled.
If a Telemetry pod is in an error state, run kubectl logs -n telemetry <pod-name> on the Kubernetes VIP and correct the reported image, secret, storage, or source configuration problem.
Review module logs under /var/log/omnia/<domain>/ and project logs under <OMNIA_DATA_PATH>/<domain>/log/.
See Kubernetes troubleshooting and Telemetry deployment troubleshooting for component-specific investigations.