Configure UFM Telemetry

Configure NVIDIA Unified Fabric Manager (UFM) to securely stream Telemetry metrics and logs to the Service Kubernetes cluster.

Overview

UFM Telemetry collects InfiniBand fabric metrics and logs from an existing NVIDIA UFM appliance.

Components

• UFM Prometheus Exporter -- Exposes InfiniBand metrics over a Prometheus-compatible HTTPS endpoint. The default port is 9001.
• vmagent (shared) -- Scrapes the UFM exporter over TLS and forwards the metrics to VictoriaMetrics.
• VMServiceScrape -- Defines the UFM scrape target, authentication, TLS, interval, and timeout for vmagent.
• VLAgent -- Receives RFC 3164 or RFC 5424 syslog messages from UFM and sends them to VictoriaLogs.
• Kubernetes Service and Endpoints -- Represent the external UFM appliance as the ufm-external service in the telemetry namespace.

Omnia does not deploy or configure the UFM appliance.

Data flow

UFM Fabric Manager -> UFM Prometheus Exporter -> vmagent (shared) -> VictoriaMetrics
UFM Fabric Manager -> syslog -> VLAgent -> VictoriaLogs

Supported metrics and logs

Metrics category	Metrics collected
Port state	InfiniBand port operational state, including up, down, and disabled
Traffic counters	Transmit and receive rates, bytes per second, and packets per port
Error counters	Symbol errors, link recovery and link-down events, VL15 drops, and excessive buffer overruns
Fabric topology	Switch information, port mappings, node GUIDs, and LIDs
Telemetry health	Scrape success, scrape duration, and ingestion latency

For the complete list, see the UFM Metrics reference.

Log category	Logs collected
Fabric events	Topology changes, port transitions, errors, and warnings
Manager events	Subnet Manager, SHARP, and UFM health events
Labels	Hostname, severity, and facility metadata

UFM metrics and logs are controlled independently by metrics_enabled and logs_enabled.

Prerequisites

• Complete the common Telemetry deployment prerequisites.
• Ensure the service Kubernetes cluster has sufficient resources to run vmagent (shared instance) and VLAgent.
• Ensure network connectivity between the service Kubernetes cluster and the NVIDIA UFM appliance.
• Ensure that an NVIDIA UFM appliance is running. Omnia does not deploy UFM.
• Ensure that telemetry_config.yml has UFM telemetry entries enabled. For details, see the telemetry configuration reference.
• UFM telemetry must bind and listen on a specific designated IP address rather than the default localhost (127.0.0.1).
• Allow ports 9000, 9001, and 9002 through the firewalld service on the UFM appliance. Create and enable the required firewall rules before deployment.
• Deploy UFM with self-signed SSL/TLS certificates and configure it to listen on a specific IP address and designated port for secure communication.
• Provide a UFM IP address that the Kubernetes cluster can reach.
• Enable the UFM Prometheus endpoint on the appliance and know its port.
• For basic authentication, provide ufm_username and ufm_password when prompted.
• For CA-signed TLS, place the PEM CA certificate on the OIM and record its path.

Procedure

Step 1: Configure the UFM Appliance

Enable UFM Telemetry in the gv.cfg configuration file on the UFM appliance:

[Telemetry]
telemetry_provider = telemetry

(Optional) Configure SSL certificates -- If using CA-signed TLS, set up SSL and CA certificates in UFM. For detailed steps, see Setting Up SSL and CA Certificates in UFM.

Step 2: Configure UFM Log Forwarding

To collect UFM logs, configure syslog forwarding on the UFM appliance. First, retrieve the VLAgent LoadBalancer IP:

Run on K8s control plane

kubectl get svc -n telemetry | grep vlagent

Using the UFM Web UI:

1. From the left navigation menu, select Settings > Data Streaming.
2. Select System log and complete the fields:
   • Destination: Enter the VLAgent LoadBalancer IP address
   • Syslog Port: Enter 514 (default)
   • System logs Level: Select syslog level from the dropdown based on your requirements
   • Streaming Data: Select UFM logs
3. Click Save.

Using the UFM CLI:

Modify the [Logging] section in /opt/ufm/conf/gv.cfg:

[Logging]
syslog = true
syslog_addr = <external vlagent loadbalancer IP>:514
ufm_syslog = true
event_syslog = true
syslog_level = WARNING

For detailed information on UFM syslog configuration parameters, see NVIDIA UFM Enterprise User Manual - Configuring Syslog.

Step 3: Configure UFM Telemetry

1. Enable UFM metrics and VictoriaMetrics in telemetry_config.yml:

   telemetry_sources:
     ufm:
       metrics_enabled: true
       logs_enabled: false
       collection_targets:
         - victoria_metrics
   
   ufm_configuration:
     ufm_endpoint: "172.20.44.180"
     ufm_metrics_port: 9001
     scrape_interval: "30s"
     scrape_timeout: "15s"
     tls_mode: "self_signed"
     ufm_ca_cert_path: ""
     auth_mode: "basic"
   

   tls_mode accepts self_signed or ca_signed; auth_mode accepts basic or none. When ca_signed is selected, set ufm_ca_cert_path to the PEM file.

Step 4: Validate and Deploy UFM Telemetry

1. Run the Telemetry precheck. Choose one execution method; do not run both commands for the same operation.

   Using omnia.sh (recommended)Using ansible-playbook

   Run on: OIM

   cd src/main
   ./omnia.sh --run telemetry --tags precheck
   

   Run on: OIM

   source /opt/omnia/activate-omnia.sh
   cd <OMNIA_SOURCE_PATH>/src/telemetry/playbooks
   ansible-playbook telemetry.yml --tags precheck
   

2. Validate the Telemetry inputs and collect the required credentials:

   Using omnia.sh (recommended)Using ansible-playbook

   Run on: OIM

   cd src/main
   ./omnia.sh --run telemetry --tags validate
   

   Run on: OIM

   source /opt/omnia/activate-omnia.sh
   cd <OMNIA_SOURCE_PATH>/src/telemetry/playbooks
   ansible-playbook telemetry.yml --tags validate
   

3. Deploy the enabled Telemetry configuration:

   Using omnia.sh (recommended)Using ansible-playbook

   Run on: OIM

   cd src/main
   ./omnia.sh --run telemetry --tags deploy
   

   Run on: OIM

   source /opt/omnia/activate-omnia.sh
   cd <OMNIA_SOURCE_PATH>/src/telemetry/playbooks
   ansible-playbook telemetry.yml --tags deploy
   

4. To run validation and deployment in one invocation, omit the tag:

   Using omnia.sh (recommended)Using ansible-playbook

   Run on: OIM

   cd src/main
   ./omnia.sh --run telemetry
   

   Run on: OIM

   source /opt/omnia/activate-omnia.sh
   cd <OMNIA_SOURCE_PATH>/src/telemetry/playbooks
   ansible-playbook telemetry.yml
   

   The untagged flow does not run the opt-in precheck. Run step 1 separately when an environment precheck is required.

Verification

Verify UFM Telemetry pods

1. Verify that the VictoriaMetrics pods are running:

   Run on: Kubernetes control plane

   kubectl get pods -n telemetry -o wide | grep vm
   

   

2. Verify that the VictoriaMetrics services are running:

   Run on: Kubernetes control plane

   kubectl get service -n telemetry -o wide | grep vm
   

   

3. Check the shared vmagent logs for successful UFM scrapes:

   Run on: Kubernetes control plane

   VMAGENT_POD=$(kubectl get pods -n telemetry \
     -l app.kubernetes.io/name=vmagent \
     -o jsonpath='{.items[0].metadata.name}')
   kubectl logs "$VMAGENT_POD" -n telemetry -c vmagent --tail=50
   

   

View UFM metrics in VictoriaMetrics UI

1. Identify the external vmselect service:

   Run on: Kubernetes control plane

   kubectl get svc -n telemetry | grep vmselect
   

   

2. Export the VictoriaMetrics connection details:

   Using omnia.sh (recommended)Using ansible-playbook

   Run on: OIM

   cd src/main
   ./omnia.sh --run telemetry --tags external_victoria
   

   Run on: OIM

   source /opt/omnia/activate-omnia.sh
   cd <OMNIA_SOURCE_PATH>/src/telemetry/playbooks
   ansible-playbook telemetry.yml --tags external_victoria
   

3. Open the URL recorded in victoria_metrics.endpoints.vmselect.ui_url in the following file:

   $OMNIA_DATA_PATH/telemetry/output/$OMNIA_PROJECT_NAME/external_victoria/external_victoria_connect_details.yml
   

4. Query a UFM metric, such as infiniband_CBW, to confirm that UFM metrics are reaching VictoriaMetrics. To filter InfiniBand metrics by their source labels, use:

   {source="ufm", subsystem="infiniband"}
   

   

View UFM logs in VictoriaLogs

Complete these steps only when UFM log collection is enabled.

1. Verify that VLAgent and VictoriaLogs services are running:

   Run on: Kubernetes control plane

   kubectl get svc -n telemetry | grep -E '(vlagent|victoria-logs)'
   

   

2. Identify the external vlselect service:

   Run on: Kubernetes control plane

   kubectl get svc -n telemetry | grep vlselect
   

   

3. If needed, export the Victoria connection details by using either command shown in step 2 of the previous section. Open the URL recorded in victoria_logs.endpoints.vlselect.ui_url in:

   $OMNIA_DATA_PATH/telemetry/output/$OMNIA_PROJECT_NAME/external_victoria/external_victoria_connect_details.yml
   

4. Query ufm to confirm that UFM logs are reaching VictoriaLogs.

   

Confirm sources.ufm.metrics: deployed in telemetry_status.yml. This status records the integration resource state; successful metric queries provide the end-to-end validation. When logs are enabled, sources.ufm.logs: deployed confirms that VLAgent is running; a successful log query confirms that the UFM appliance has begun sending logs.

Next steps

• Optional: To disable UFM Telemetry metrics and logs without removing its infrastructure, set telemetry_sources.ufm.metrics_enabled: false and telemetry_sources.ufm.logs_enabled: false in telemetry_config.yml. Then rerun deployment:

  Using omnia.sh (recommended)Using ansible-playbook

  Run on: OIM

  cd <OMNIA_SOURCE_PATH>/src/main
  ./omnia.sh --run telemetry --tags deploy
  

  Run on: OIM

  source /opt/omnia/activate-omnia.sh
  cd <OMNIA_SOURCE_PATH>/src/telemetry/playbooks
  ansible-playbook telemetry.yml --tags deploy
  

The UFM Telemetry pods are scaled down to zero. Its services and ConfigMaps remain available for later use. - To re-enable UFM Telemetry, set the required metrics_enabled or logs_enabled values to true, and rerun the same deploy command. - To remove UFM Telemetry resources and source-owned persistent volumes, see Clean Up Telemetry. - Use Export VictoriaMetrics Connection Details to obtain the query endpoint and UI URL.

Troubleshooting

• The endpoint is rejected: Set a non-empty UFM IP address and a port from 1 through 65535.
• Credentials are missing: Supply UFM credentials when auth_mode: basic.
• Credentials are requested with auth_mode: none: The current credential collection is gated by enabled UFM metrics, not by auth_mode. Complete the prompt while this source behavior remains in place.
• The CA file is rejected: With tls_mode: ca_signed, provide an existing PEM certificate path on the OIM.
• Deployment fails with auth_mode: none and self-signed TLS: The current source can render an empty Secret while still attempting to apply it. Use basic authentication or CA-signed TLS until that source limitation is fixed.
• No metrics arrive: Confirm the UFM endpoint is reachable from Kubernetes and that its authentication, TLS mode, scrape interval, and timeout are correct.
• Logs do not arrive: Confirm that UFM is sending to the exported VLAgent syslog endpoint and that victoria_logs remains in its collection targets.