KVM OpenStack Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]

Aug 7, 2016 Cloud Computing

Packstack Error: Command exceeded timeout
Few times during KVM based OpenStack (Mitaka, Newton) automated installations using packstack we encountered DB synchronization errors.

It turned out, that these installation errors appeared due to slow network and/or poor hardware performance of KVM virtualized hardware used to build OpenStack virtual nodes (Controller, Network, Compute).

After running OpenStack installation using packstack:

[root@controller ~]# packstack --answer-file=/root/answers.txt

…the installation failed with the following timeout errors:

ERROR : Error appeared during Puppet run:
Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]: Failed to call refresh: Command exceeded timeout
ERROR : Error appeared during Puppet run:
Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Failed to call refresh: Command exceeded timeout

To fix db sync errors during KVM based OpenStack installations, you can perform the following steps:

1. Execute packstack automated installer with extended timeout

Packstack uses Puppet for OpenStack automated node deployment. Packstack, by default, sets timeout for Puppet’s manifests execution to 300 seconds. If you install OpenStack on old hardware, which has poor performance or the nodes are connected to the network with poor throughput, Puppet may encounter execution timeouts when applying it’s manifests across all OpenStack nodes. In other words, Openstack nodes are not able to respond to Puppet (running on Controller node) within the default timeout, this causes Packstack installation to fail.

We can execute packstack installation with timeout parameter to extend Puppet’s manifests execution timeout (i.e.: up to 600 seconds):

[root@controller ~]# packstack --answer-file=/root/answers.txt --timeout=600

2. Set CPU pinning for OpenStack KVM nodes (KVM performance tuning for OpenStack)

CPU pinning significantly improves performance of KVM virtual machines and all KVM virtual/isolated networks where virtual machines are hooked up together. KVM CPU pinning reserves particular KVM hypervisor’s CPU(s) for particular VM to secure it’s demand for CPU resources.

We can use virt-manager to pin hypervisor’s CPUs to particular VMs.

Example pinning scheme:
Hypervisor’s total CPU resources: 4 CPUs (CPU0, CPU1, CPU2, CPU3)
– Compute node: 1 CPU (CPU0)
– Controller node: 2 CPUs (CPU1, CPU2)
– Network node: 1 CPU (CPU3)

Compute node CPU pinning:
cpu pinning compute node openstack kvm installation

Controller node CPU pinning:
cpu pinning controller node openstack kvm installation

Network node CPU pinning:
cpu pinning network node openstack kvm installation

3. Provide sufficient amount of RAM for OpenStack nodes

If your nodes (especially Controller node) run out of RAM during packstack installation, they will start swapping and their performance will significantly decrease. Equip your nodes with the appropriate amount of RAM to avoid performance drops.

Below we propose the minimum RAM requirements for OpenStack nodes:
– Controller node: 3GB RAM
– Network node: 2GB RAM
– Compute node: 3GB+ RAM (depends on the amount of Instances)

One thought on “KVM OpenStack Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]”
  1. Got the same error on Hyper-V.
    Have increased VM configuration to 4 cores and 8 GB of Ram but – still the same. Any ideas?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.