Skip to content

Configure Cluster DNS

Enable Cluster DNS to provide dynamic hostname resolution for Slurm, login, and Kubernetes nodes using CoreDNS instead of static /etc/hosts file management.

Overview

Cluster DNS replaces per-node /etc/hosts synchronization with coresmd, a CoreDNS instance on the OIM that generates DNS records automatically from the OpenCHAMI SMD inventory. For a full explanation of the architecture, DNS ownership boundaries, and failure behavior, see Cluster DNS.

Prerequisites

  • Omnia is deployed on the OIM node with OpenCHAMI services running.
  • /opt/omnia/input/project_default/provision_config.yml exists and is validated.
  • The OIM node is accessible on the admin network.

Procedure

Enable Cluster DNS

  1. Edit the provision configuration file on the OIM node:

    Run on: OIM host
    vi /opt/omnia/input/project_default/provision_config.yml

  2. Set the dns_enabled parameter to true:

    File: /opt/omnia/input/project_default/provision_config.yml
    dns_enabled: true

    Note

    The default value is false, which preserves the legacy /etc/hosts behavior.

  3. Deploy or redeploy OpenCHAMI with coresmd (if not already deployed):

    Run on: omnia_core container
    cd /omnia/prepare_oim
ansible-playbook prepare_oim.yml

  4. Run the provisioning playbook so nodes receive cloud-init with /etc/resolv.conf configured:

    Run on: omnia_core container
    cd /omnia/provision
ansible-playbook provision.yml

  5. Reprovision (reboot) all compute nodes to apply the new cloud-init configuration.

    Important

    Nodes must be reprovisioned after setting dns_enabled: true for the change to take effect. Existing nodes retain their previous configuration until reprovisioned.

Disable Cluster DNS (Revert to /etc/hosts)

  1. Edit provision_config.yml and set dns_enabled to false:

    File: /opt/omnia/input/project_default/provision_config.yml
    dns_enabled: false

  2. Re-run the provisioning playbook to regenerate cloud-init configuration:

    Run on: omnia_core container
    cd /omnia/provision
ansible-playbook provision.yml

  3. Reprovision (reboot) all compute nodes to apply the new cloud-init configuration.

    Note

    No coresmd or OpenCHAMI changes are needed for rollback. coresmd continues running but compute nodes no longer query it.

Verification

  1. Verify the compute node resolver configuration:

    Run on: compute node
    cat /etc/resolv.conf
    Expected output
    search <domain_name>
nameserver <admin_nic_ip>
options timeout:1 attempts:2

  2. Verify no peer entries exist in /etc/hosts (only localhost entries should be present):

    Run on: compute node
    cat /etc/hosts

  3. Verify forward DNS resolution for a cluster hostname:

    Run on: compute node
    getent hosts <hostname>.<domain>
    Expected output
    172.16.0.1 nid001.hpc.cluster

  4. Query coresmd directly from the OIM node or any node with network access to it:

    Run on: OIM host
    dig <hostname>.<domain> @<admin_nic_ip>

    Expected output includes an ANSWER SECTION with the node's admin IP address.

  5. Verify Kubernetes CoreDNS patching (if Kubernetes is deployed). Confirm the ConfigMap contains the forward zone:

    Run on: kube_control_plane
    kubectl -n kube-system get configmap coredns -o yaml
    Expected output
    hpc.cluster:53 {
    errors
    cache 30
    forward . 172.16.107.254
}

  6. Verify Kubernetes pod resolution (if Kubernetes is deployed):

    Run on: kube_control_plane
    kubectl exec -it <pod> -- getent hosts <hostname>.<domain>

  7. Verify Slurm and MPI functionality:

    Run on: OIM host
    sinfo
srun -N <N> hostname
mpirun -np 4 -host <host1>,<host2> hostname

    All nodes should show as IDLE or ALLOCATED in sinfo, and jobs should complete without DNS errors or timeouts.

  8. Verify new node auto-resolution. After adding a node via provision.yml, wait up to 30 seconds for coresmd to refresh its cache, then confirm resolution without any playbook re-run:

    Run on: compute node
    getent hosts <new_hostname>

Best Practices

  • Plan DNS mode before deployment -- Decide on DNS mode before the initial cluster deployment. Changing mode afterward requires reprovisioning all nodes.
  • Monitor coresmd health -- Track coresmd container status and logs, and use Prometheus metrics (port 9153) to monitor DNS query performance.
  • Configure reliable upstream DNS -- Configure at least two reliable upstream DNS servers in admin_network.dns and test connectivity before enabling Cluster DNS.
  • Test resolution before production -- Verify DNS resolution, Slurm/MPI job execution, and Kubernetes pod resolution before running production workloads.
  • Document domain configuration -- Record the cluster domain name and hostname pattern (cluster_shortname, cluster_nidlength) for reference.
  • Plan for high availability -- The OIM node is a single point of failure for DNS in the current implementation. Plan for OIM HA deployment and monitor OIM node health.
  • Use short-name resolution -- Leverage the search <domain_name> directive so users can reference short hostnames instead of FQDNs.
  • Validate after node changes -- After adding or removing nodes, verify DNS resolution within 30 seconds using dig or getent hosts.

Next Steps

Troubleshooting

Custom hostnames not resolving

Custom hostnames from the PXE mapping file resolve via /etc/hosts, not CoreDNS. If a custom hostname does not resolve, check that it is present in /etc/hosts:

Run on: compute node
grep <hostname> /etc/hosts

If missing, re-run provision.yml to repopulate /etc/hosts.

Mixed-state cluster

If some nodes resolve via DNS while others use /etc/hosts, only some nodes were reprovisioned after changing dns_enabled. Check /etc/resolv.conf on the affected nodes to determine which mode they are using, then reprovision and reboot all nodes for a consistent configuration.

Related pages