KVM OpenStack Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]

Few times during KVM based OpenStack (Mitaka, Newton) automated installations using packstack we encountered DB synchronization errors.

It turned out, that these installation errors appeared due to slow network and/or poor hardware performance of KVM virtualized hardware used to build OpenStack virtual nodes (Controller, Network, Compute).

After running OpenStack installation using packstack:

[root@controller ~]# packstack --answer-file=/root/answers.txt

…the installation failed with the following timeout errors:

ERROR : Error appeared during Puppet run: 192.168.2.21_api_nova.pp
Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]: Failed to call refresh: Command exceeded timeout

ERROR : Error appeared during Puppet run: 192.168.2.21_neutron.pp
Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Failed to call refresh: Command exceeded timeout

To fix db sync errors during KVM based OpenStack installations, you can perform the following steps:

1. Execute packstack automated installer with extended timeout

Packstack uses Puppet for OpenStack automated node deployment. Packstack, by default, sets timeout for Puppet’s manifests execution to 300 seconds. If you install OpenStack on old hardware, which has poor performance or the nodes are connected to the network with poor throughput, Puppet may encounter execution timeouts when applying it’s manifests across all OpenStack nodes. In other words, Openstack nodes are not able to respond to Puppet (running on Controller node) within the default timeout, this causes Packstack installation to fail.

We can execute packstack installation with timeout parameter to extend Puppet’s manifests execution timeout (i.e.: up to 600 seconds):

[root@controller ~]# packstack --answer-file=/root/answers.txt --timeout=600

2. Set CPU pinning for OpenStack KVM nodes (KVM performance tuning for OpenStack)

CPU pinning significantly improves performance of KVM virtual machines and all KVM virtual/isolated networks where virtual machines are hooked up together. KVM CPU pinning reserves particular KVM hypervisor’s CPU(s) for particular VM to secure it’s demand for CPU resources.

We can use virt-manager to pin hypervisor’s CPUs to particular VMs.

Example pinning scheme:
Hypervisor’s total CPU resources: 4 CPUs (CPU0, CPU1, CPU2, CPU3)
– Compute node: 1 CPU (CPU0)
– Controller node: 2 CPUs (CPU1, CPU2)
– Network node: 1 CPU (CPU3)

Compute node CPU pinning:

Controller node CPU pinning:

Network node CPU pinning:

3. Provide sufficient amount of RAM for OpenStack nodes

If your nodes (especially Controller node) run out of RAM during packstack installation, they will start swapping and their performance will significantly decrease. Equip your nodes with the appropriate amount of RAM to avoid performance drops.

Below we propose the minimum RAM requirements for OpenStack nodes:
– Controller node: 3GB RAM
– Network node: 2GB RAM
– Compute node: 3GB+ RAM (depends on the amount of Instances)