Skip to content

Provisioning Issues

Issues related to PXE booting, node discovery, cloud-init configuration, and the discovery.yml playbook.

PXE Boot Failures

Node Hangs at nm-wait-online-initrd.service

Symptom

Node hangs during boot at the nm-wait-online-initrd.service stage.

Cause

IP address conflict with an old node.

Resolution
  1. Ensure the old node is powered off or disconnected.
  2. Verify the IP address is unused on the network.
  3. Re-run provision.yml.

PXE Boot Timeout (TFTP/Service Timeout)

Symptom

PXE boot process times out with TFTP or service timeout errors:

Expected output
PXE-E32: TFTP open timeout
PXE-T02: TFTP packet timeout
Cause
  • PXE NIC not configured in BIOS.
  • Extra NIC interfering with the boot process.
  • Multiple PXE servers on the same network.
Resolution
  1. Configure BIOS: navigate to Network Settings > PXE Device and assign the correct active NIC.
  2. Remove or disable any extra NIC until after boot completion.
  3. Verify no rogue PXE/DHCP servers exist on the admin network.

Target Server Unreachable After PXE Boot

Symptom

Target server becomes unreachable after PXE boot completes.

Cause
  • POST errors on the target server.
  • F1 hardware prompts blocking boot.
  • Boot stalls due to hardware issues.
Resolution
  1. Log in to iDRAC and check console output.
  2. Clear errors or disable POST prompts.
  3. Hard reboot the server.
  4. Disable PXE temporarily if needed to bypass boot loops.

Root Login Fails After Provisioning

Symptom

Unable to log in as root via SSH. Error messages include:

  • WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
  • Permission denied (publickey,gssapi-keyex,gssapi-with-mic)
  • ssh: connect to host <ip> port 22: Connection refused
Cause
  • Outdated SSH key in ~/.ssh/known_hosts.
  • cloud-init not rendered on the target node.
Resolution

  1. Remove the stale SSH key:

    Run on: OIM host
    ssh-keygen -R <hostname>

  2. Retry login or reprovision the node.

Connecting directly to OpenCHAMI

Symptom

Unable to issue OpenCHAMI commands. Error includes:

  • Environment variable OIM_ACCESS_TOKEN unset for reading token for cluster "oim"
Cause
  • OIM_ACCESS_TOKEN environment variable has not been set.
Resolution

  1. Set the OIM_ACCESS_TOKEN:

    export OIM_ACCESS_TOKEN=$(sudo bash -lc 'gen_access_token')

  2. Retry the OpenCHAMI command.

Cloud-Init Issues

Symptom

Nodes boot the OS successfully but post-boot configuration fails. The node is accessible via console but network settings, hostname, or SSH keys are not configured correctly.

Cause
  • The cloud-init data source is not configured.
  • The cloud-init configuration file has syntax errors.
  • The cloud-init service was disabled or removed from the OS image.
Resolution

  1. Check cloud-init status on the affected node:

    Run on: compute node
    cloud-init status --long

  2. Review cloud-init logs:

    Run on: compute node
    cat /var/log/cloud-init.log
cat /var/log/cloud-init-output.log

  3. If cloud-init was disabled, re-enable it:

    Run on: compute node
    systemctl enable cloud-init
cloud-init clean
reboot

Boot Issues on Provisioned Nodes

Symptom

A provisioned node fails to boot correctly, or post-boot configuration (hostname, network, SSH keys) is incomplete or missing.

Cause
Resolution

  1. Check the cloud-init output log on the affected node:

    Run on: compute node
    cat /var/log/cloud-init-output.log

  2. Review the provisioning log on the OIM:

    Run on: omnia_core
    cat /opt/omnia/log/provision.log

  3. If cloud-init completed with errors, re-run provision.yml after fixing the root cause.

  4. If the hostname or root password is not configured because cloud-init was not loaded in time, wait 5 minutes and retry provisioning the node. If the issue persists, redeploy the cluster after running the oim_cleanup.yml playbook.

IP Route Conflict After Provisioning

Symptom

After provisioning, nodes lose connectivity on the admin network or cannot reach the OIM, while the public/internet NIC works (or vice versa).

Cause

An IP route conflict exists between the admin network and an additional NIC (for example, an internet-facing NIC). Both NICs may have overlapping default routes.

Resolution

  1. List current routes on the affected node:

    Run on: compute node
    ip route show

  2. Delete the conflicting admin route or adjust route priority:

    Run on: compute node
    # Delete conflicting route
ip route del <conflicting_route>

# Or set metric to prioritize one route over another
ip route add <network> via <gateway> dev <nic> metric <priority>

  3. To make the change persistent, update the network configuration files for the appropriate NIC.

Info