OpenStack networking topology is pretty complicated structure, even with it’s, let’s say, default configuration based on OVS (OpenVSwitch), not even mentioning about additional overlays, like OpenDaylight or Nuage.
If you are troubleshooting network issues in regard to a particular Instance (VM), it’s good to locate network overlay interfaces (qbr, qvo) assigned to that Instance to have a starting point to begin your network analysis from. You can then analyze inbound/outbound traffic on those interfaces using network protocol analyzers, like tcpdump/wireshark.
Below you can find few steps, how to locate qbr,qvo interfaces belonging to a particular Instance:
1. Locate Nova Compute host including faulty Instance
Login to your controller node and source keystone file (import admin credentials):
[root@controller ~]# source /root/keystonerc_admin
[root@controller ~(keystone_admin)]#Display Instances for all tenants:
[root@controller ~(keystone_admin)]# openstack server list --all-projects --long
+--------------------+-------------------+---------+------------+-------------+--------------------+-------------+--------------------+-------------------+---------+------------+
| ID | Name | Status | Task State | Power State | Networks | Image Name | Image ID | Availability Zone | Host | Properties |
+--------------------+-------------------+---------+------------+-------------+--------------------+-------------+--------------------+-------------------+---------+------------+
| 94b233cb-c377-4b94 | kali | SHUTOFF | None | Shutdown | priv_net=192.168.2 | kali_2017.3 | 60602fe5-caaa-4d0f | nova | compute | |
| -ab5f-f4982ffc0e23 | | | | | 2.13, 192.168.2.57 | | -a681-a9648e5e3f6b | | | |
| a3164da8-9af8-4000 | cirros_instance-2 | SHUTOFF | None | Shutdown | priv_net=192.168.2 | | | nova | compute | |
| -90d2-9d9d35880ed1 | | | | | 2.8, 192.168.2.60 | | | | | |
| c022869a-acf0 | cirros_instance-1 | ACTIVE | None | Running | priv_net=192.168.2 | | | nova | compute | |
| -432f-8e5a- | | | | | 2.5, 192.168.2.51 | | | | | |
| b4439ca7f5d0 | | | | | | | | | | |
+--------------------+-------------------+---------+------------+-------------+--------------------+-------------+--------------------+-------------------+---------+------------+Let’s say cirros_instance-1 is having networking issues and we need to find qbr,qvo interfaces that belong to this Instance.
From the above example we can see, that Instance is located on compute node, so we need to log in to the compute node.
In our case compute node is a KVM vHost (hypervisor), so all the Instances running on compute node are in fact KVM Virtual Machines.
Login to the compute node, which hosts faulty Instance and list KVM VMs:
[root@compute ~]# virsh list --all
Id Name State
----------------------------------------------------
2 instance-00000001 running
- instance-00000002 shut off
- instance-00000009 shut off2. Locate faulty Instance
VM names at the end of the string include IDs of corresponding Instances, that means instance-00000001 has ID=1 in OpenStack database.
Let’s check Instance’s ID in MariaDB database on controller node:
[root@controller ~(keystone_admin)]# mysql -u root
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 293
Server version: 10.1.17-MariaDB MariaDB Server
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> use nova;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [nova]> select * from instance_id_mappings where id='1';
+---------------------+------------+------------+----+--------------------------------------+---------+
| created_at | updated_at | deleted_at | id | uuid | deleted |
+---------------------+------------+------------+----+--------------------------------------+---------+
| 2016-10-22 22:28:12 | NULL | NULL | 1 | c022869a-acf0-432f-8e5a-b4439ca7f5d0 | 0 |
+---------------------+------------+------------+----+--------------------------------------+---------+
1 row in set (0.00 sec)Now on controller node verify Instance using above uuid:
[root@controller ~(keystone_admin)]# openstack server show c022869a-acf0-432f-8e5a-b4439ca7f5d0
+--------------------------------------+----------------------------------------------------------+
| Field | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-10-22T22:29:19.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | priv_net=192.168.22.5, 192.168.2.51 |
| config_drive | |
| created | 2016-10-22T22:28:12Z |
| flavor | m2.tiny (31e77a52-e81e-46e0-989f-f95044ee1a0a) |
| hostId | ea2e42e26aac9ab745b35dcbc01120db9630e29e16fb212cb3ce5a52 |
| id | c022869a-acf0-432f-8e5a-b4439ca7f5d0 |
| image | |
| key_name | None |
| name | cirros_instance-1 |
| os-extended-volumes:volumes_attached | [{u'id': u'7c70f9cb-9936-4f1e-8c4e-bb6731cde0cb'}] |
| progress | 0 |
| project_id | d8b0f52bdb2741b6a4e1cf3d8082a8f2 |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | ACTIVE |
| updated | 2017-11-22T10:21:32Z |
| user_id | 2c540c9c6ba541d1be6ef02afba5b004 |
+--------------------------------------+----------------------------------------------------------+The above listing proves that instance-00000001 in KVM is a cirros_instance-1 in OpenStack database with id=c022869a-acf0-432f-8e5a-b4439ca7f5d0.
3. Find qbr,qvo interfaces of the faulty Instance
Once we are sure that instance-00000001 is the faulty Instance, we can try to locate its interfaces on compute host:
[root@compute ~]# virsh domiflist instance-00000001
Interface Type Source Model MAC
-------------------------------------------------------
tapc5dafda7-1f bridge qbrc5dafda7-1f virtio fa:16:3e:68:8f:30The output shows, that instance-00000001 is assigned to qbrc5dafda7-1f interface.
Knowing qbr interface, it’s fairly easy to find corresponding qvo interface, which in fact has the same suffix c5dafda7-1f, so the corresponding qvo interface is qvoc5dafda7-1f.
qvoc5dafda7-1f interface is connected, as a port, to OVS integration bridge br-int on Nova Compute host, just like other qvo interfaces from other existing Instances:
[root@compute ~]# ovs-vsctl list-ports br-int
int-br-eth1
patch-tun
qvo489f9dbe-7c
qvoc14c83e0-c0
qvoc5dafda7-1f[root@compute ~]# ovs-vsctl show
6b8ed57a-4e2a-4ddd-ae5f-cc6872a83067
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "qvoc5dafda7-1f"
tag: 1
Interface "qvoc5dafda7-1f"
Port "qvo489f9dbe-7c"
tag: 1
Interface "qvo489f9dbe-7c"
Port br-int
Interface br-int
type: internal
Port "qvoc14c83e0-c0"
tag: 1
Interface "qvoc14c83e0-c0"
Port "int-br-eth1"
Interface "int-br-eth1"
type: patch
options: {peer="phy-br-eth1"}
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
...
..
.Once we know both qbr and qvo interface, we can start analyzing traffic of the Instance using tcpdump/wireshark:
[root@compute ~]# tcpdump -i qbrc5dafda7-1f[root@compute ~]# tcpdump -i qvoc5dafda7-1fI think this is a good starting point in troubleshooting networking issues of Instances in OpenStack environment.
