Collect Telemetry Data from External Client Nodes to Victoria DB (Cluster Mode)

This section describes how to create a new metric in VictoriaMetrics and configure an external telemetry producer to stream metrics securely into the Service Kubernetes cluster using TLS.

This procedure assumes that VictoriaMetrics is deployed in cluster mode inside the telemetry namespace of the Service Kubernetes cluster. For more details, see the VictoriaMetrics Cluster Mode documentation.

For collecting telemetry data from Smart Fabric Manager (SFM) to Victoria DB, see Collect telemetry data from Smart Fabric Manager to Victoria DB (cluster mode).

Prerequisites

Ensure the following prerequisites are met:

  • A Service Kubernetes cluster is running with VictoriaMetrics deployed in the telemetry namespace.

  • External access to VictoriaMetrics is available through:

    • LoadBalancer port 8480 for ingesting (inserting) data.

    • LoadBalancer port 8481 for querying data.

Retrieve the Victoria Select and Insert LoadBalancer IP Addresses

Run the following playbook to retrieve the VictoriaMetrics connection details and TLS certificate from the Service Kubernetes cluster:

cd /omnia/utils
ansible-playbook external_victoria_connect_details.yml
The external_victoria_connect_details.yml playbook does the following:

  • Retrieves the VictoriaMetrics vminsert and vmselect LoadBalancer IPs.

  • Extracts the server CA certificate for TLS.

  • Writes the connection details to /opt/omnia/telemetry/external_victoria_connect_details.yml.

  • Saves the CA certificate at /opt/omnia/telemetry/victoria-certs/ca.crt.

Push Sample metrics from Omnia Core Container in the OIM

  1. Add the LoadBalancer insert and select IP addresses to /etc/hosts:

    echo "<vminsert-IP> vminsert.telemetry.svc.cluster.local" >> /etc/hosts
echo "<vmselect-IP> vmselect.telemetry.svc.cluster.local" >> /etc/hosts

For vminsert and vmselect IP, use the values retrieved by the external_victoria_connect_details.yml playbook.

Note

The /etc/hosts update must be repeated if the SFM Prometheus pod restarts.

  1. Create a new test metric using the following command:

    curl --cacert ca.crt -X POST "https://vminsert.telemetry.svc.cluster.local:8480/insert/0/prometheus/api/v1/import/prometheus" \
-H "Content-Type: text/plain" \
-d "test_metric{source=\"external\"} 42"

Note

Use https://vminsert.telemetry.svc.cluster.local:8480//insert/0/prometheus/api/v1/write to push the metrics from an external client, such as SmartFabric Manager (SFM). To know more about SFM, refer to SFM.

Note

For collecting telemetry data from Smart Fabric Manager (SFM) to Victoria DB, see Collect telemetry data from Smart Fabric Manager to Victoria DB (cluster mode).

  1. Push the sample test metrics to Victoria DB using the following command:

    curl --cacert ca.crt -X POST   "https://vminsert.telemetry.svc.cluster.local:8480/insert/0/prometheus/api/v1/import/prometheus"   -H "Content-Type: text/plain"   -d 'cpu_usage{host="server1",job="new"} 75.5
memory_usage{host="server1",job="new"} 1024
disk_usage{host="server1",job="new"} 512
network_rx{host="server1",interface="eth0"} 1000000
network_tx{host="server1",interface="eth0"} 500000'

  2. Use the following commands to query the inserted data from Victoria DB:

    curl --cacert ca.crt -s "https://vmselect.telemetry.svc.cluster.local:8481/select/0/prometheus/api/v1/query?query=new_metric"
curl --cacert ca.crt -s "https://vmselect.telemetry.svc.cluster.local:8481/select/0/prometheus/api/v1/query_range?query=cpu_usage&start=$(date -d '1 hour ago' +%s)&end=$(date +%s)&step=600s"

If you have any feedback about Omnia documentation, please reach out at omnia.readme@dell.com.