RDO是在RHEL，Fedora, CentOS等系统上部署OpenStack的方案。简单的试用或研究OpenStack可以选择all-in-one模式安装。这种模式将所有的OpenStack组件都部署在同一台机器上。本文来分析这种模式下虚拟网络的架构。
all-in-one模式安装完成时，RDO自动完成了以下工作:
创建了外部网络public: 172.24.4.224/28
创建了租户demo
为租户demo创建了私有网络private: 10.0.0.0/24
为租户demo创建路由器router1, 将private和public连接在一起
我们登录demo租户，创建两个虚拟机，网络拓朴图如下:
本文基于该拓朴分析虚拟网络架构。
首先导入租户demo的keystone RC文件:
1 source ./keystonerc_demo
查看虚拟机列表:
1 2 3 4 5 6 7 [root@localhost ~(keystone_demo)] +--------------------------------------+------+--------+------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------+--------+------------+-------------+------------------+ | bd6cc17d-9128-49aa-8834-734e9cac0b80 | vm0 | ACTIVE | - | Running | private=10.0.0.5 | | 1b61b896-3c10-4237-85c1-0ee46a79036b | vm1 | ACTIVE | - | Running | private=10.0.0.6 | +--------------------------------------+------+--------+------------+-------------+------------------+
查看Port列表:
1 2 3 4 5 6 7 8 9 [root@localhost ~(keystone_demo)] +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+ | id | name | mac_address | fixed_ips | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+ | 55e885f5-932f-4a24-bedd-c08fc6a3aeb3 | | fa:16:3e:89:a4:48 | {"subnet_id" : "f805a609-da7c-4a79-98a4-1aefb6aca2a2" , "ip_address" : "10.0.0.6" } | | aa048a76-f7bf-4d92-ad0c-538ea54c4080 | | fa:16:3e:ae:16:12 | {"subnet_id" : "f805a609-da7c-4a79-98a4-1aefb6aca2a2" , "ip_address" : "10.0.0.5" } | | af6b8f51-4650-4a13-b31c-5d12d2939d8e | | fa:16:3e:f7:d8:ef | {"subnet_id" : "f805a609-da7c-4a79-98a4-1aefb6aca2a2" , "ip_address" : "10.0.0.1" } | | be7511c5-68f6-4a0d-a1b3-3c5ae225789a | | fa:16:3e:e3:eb:66 | {"subnet_id" : "f805a609-da7c-4a79-98a4-1aefb6aca2a2" , "ip_address" : "10.0.0.2" } | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
VM0的Port为: aa048a76-f7bf-4d92-ad0c-538ea54c4080, VM1的Port为: 55e885f5-932f-4a24-bedd-c08fc6a3aeb3
查看TAP设备:
1 2 3 [root@localhost ~(keystone_demo)] 21: tapaa048a76-f7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master qbraa048a76-f7 state UNKNOWN mode DEFAULT qlen 500 25: tap55e885f5-93: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master qbr55e885f5-93 state UNKNOWN mode DEFAULT qlen 500
TAP设备根据Port的UUID前11位命名，VM0的虚拟网络接口为tapaa048a76-f7, VM1的虚拟网络接口为tap55e885f5-93。
接着，查看Linux bridge信息:
1 2 3 4 5 6 [root@localhost ~(keystone_demo)] bridge name bridge id STP enabled interfaces qbr55e885f5-93 8000.e68f141893fa no qvb55e885f5-93 tap55e885f5-93 qbraa048a76-f7 8000.d6c6d2e7f411 no qvbaa048a76-f7 tapaa048a76-f7
接口tapaa048a76-f7挂接到了Linux bridge: qbraa048a76-f7上，而且bridge: qbraa048a76-f7还挂接了qvbaa048a76-f7。neutron中，Linux bridge设备以qbr命名，qvb和qvo表示分示veth pair设备的两端。qvb表示Linux bridge端，qvo表示OVS bridge端。
查看veth设备:
1 2 3 4 5 [root@localhost ~(keystone_demo)] 19: qvoaa048a76-f7@qvbaa048a76-f7: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master ovs-system state UP qlen 1000 20: qvbaa048a76-f7@qvoaa048a76-f7: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master qbraa048a76-f7 state UP qlen 1000 23: qvo55e885f5-93@qvb55e885f5-93: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master ovs-system state UP qlen 1000 24: qvb55e885f5-93@qvo55e885f5-93: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master qbr55e885f5-93 state UP qlen 1000
可以看到veth pair的对应关系，再来查看OVS bridge：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 [root@localhost ~(keystone_demo)] 723d8ad6-f2dd-4955-a7b3-3b8ec602cb05 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge br-ex Port "qg-9395e4c9-54" Interface "qg-9395e4c9-54" type : internal Port br-ex Interface br-ex type : internal Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "qvo55e885f5-93" tag: 1 Interface "qvo55e885f5-93" Port patch-tun Interface patch-tun type : patch options: {peer=patch-int} Port "qr-af6b8f51-46" tag: 1 Interface "qr-af6b8f51-46" type : internal Port br-int Interface br-int type : internal Port "tapbe7511c5-68" tag: 1 Interface "tapbe7511c5-68" type : internal Port "qvoaa048a76-f7" tag: 1 Interface "qvoaa048a76-f7" Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port patch-int Interface patch-int type : patch options: {peer=patch-tun} Port br-tun Interface br-tun type : internal ovs_version: "2.5.0"
可以发现qvo开头的设备挂载到OVS bridge: br-int上。
之所以在TAP设备和OVS bridge之间添加一层Linux bridge，是因为OpenStack基于IPTABLES实现安全组，而OVS目前不支持在OVS bridge上直接挂接的TAP设备上配置IPTABLES规则。
从上述OVS bridge信息可以看到，挂接到br-int上的qvo设备配置了VLAN ID:1。neutron在br-int上基于VLAN来实现不同私有网络隔离。br-int上挂接的设备除了veth设备，还有设备tapbe7511c5-68和qr-af6b8f51-46。
其中TAP设备: tapbe7511c5-68为DHCP服务器使用的网络接口。neutron基于network namespace和dnsmasq来实现DHCP。
查看network namespace:
1 2 3 [root@localhost ~(keystone_demo)] qrouter-a8f68832-ba0f-4176-9536-86fa6fd6ef1e qdhcp-1ced55ea-d30f-4c1d-ba97-6a7bbbd9ecca
DHCP的namespace为qdhcp-1ced55ea-d30f-4c1d-ba97-6a7bbbd9ecca。
查看namespace中的设备:
1 2 3 4 5 6 7 8 9 10 11 12 13 [root@localhost ~(keystone_demo)] 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link /loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 11: tapbe7511c5-68: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN link /ether fa:16:3e:e3:eb:66 brd ff:ff:ff:ff:ff:ff inet 10.0.0.2/24 brd 10.0.0.255 scope global tapbe7511c5-68 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fee3:eb66/64 scope link valid_lft forever preferred_lft forever
再来查看dnsmasq进程:
1 2 3 [root@localhost ~(keystone_demo)] nobody 13598 0.0 0.0 15556 940 ? S Jan02 0:00 dnsmasq --no-hosts --no-resolv --strict-order --except-interface=lo --pid-file=/var/lib/neutron/dhcp/1ced55ea-d30f-4c1d-ba97-6a7bbbd9ecca/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/1ced55ea-d30f-4c1d-ba97-6a7bbbd9ecca/host --addn-hosts=/var/lib/neutron/dhcp/1ced55ea-d30f-4c1d-ba97-6a7bbbd9ecca/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/1ced55ea-d30f-4c1d-ba97-6a7bbbd9ecca/opts --dhcp-leasefile=/var/lib/neutron/dhcp/1ced55ea-d30f-4c1d-ba97-6a7bbbd9ecca/leases --dhcp-match=set :ipxe,175 --bind-interfaces --interface=tapbe7511c5-68 --dhcp-range=set :tag0,10.0.0.0,static,86400s --dhcp-option-force=option:mtu,1450 --dhcp-lease-max=256 --conf-file= --domain=openstacklocal root 32126 0.0 0.0 112652 976 pts/0 S+ 09:11 0:00 grep --color=auto dnsmasq
可以确认dnsmasq进程基于接口tapbe7511c5-68发送和接收DHCP数据包。
br-int上还挂接着设备:qr-af6b8f51-46，qr设备属于qrouter的namespace: qrouter-a8f68832-ba0f-4176-9536-86fa6fd6ef1e
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 [root@localhost ~(keystone_demo)] 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link /loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 12: qr-af6b8f51-46: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN link /ether fa:16:3e:f7:d8:ef brd ff:ff:ff:ff:ff:ff inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-af6b8f51-46 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fef7:d8ef/64 scope link valid_lft forever preferred_lft forever 13: qg-9395e4c9-54: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN link /ether fa:16:3e:52:dd :06 brd ff:ff:ff:ff:ff:ff inet 172.24.4.229/28 brd 172.24.4.239 scope global qg-9395e4c9-54 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe52:dd06/64 scope link valid_lft forever preferred_lft forever
虚拟机的默认网关为10.0.0.1, 配置在设备qr-af6b8f51-46上。虚拟机访问外网时，数据包会由该接口进入qrouter的namespace。
查看namespace内的路由信息:
1 2 3 4 [root@localhost ~(keystone_demo)] default via 172.24.4.225 dev qg-9395e4c9-54 10.0.0.0/24 dev qr-af6b8f51-46 proto kernel scope link src 10.0.0.1 172.24.4.224/28 dev qg-9395e4c9-54 proto kernel scope link src 172.24.4.229
可以看到qrouter的默认路由为从接口qg-9395e4c9-54发送至172.24.4.225。
从上边的OVS bridge信息看到: 接口qg-9395e4c9-54挂接在OVS bridge: br-ex上。OVS bridge默认存在一个同名的网络接口，来查看br-ex接口信息:
1 2 3 4 5 6 7 [root@localhost ~(keystone_demo)] 9: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link /ether 0a:c0:1c:ac:17:4f brd ff:ff:ff:ff:ff:ff inet 172.24.4.225/28 scope global br-ex valid_lft forever preferred_lft forever inet6 fe80::8c0:1cff:feac:174f/64 scope link valid_lft forever preferred_lft forever
br-ex上配置了IP：172.24.4.225。可以看到，虚拟机访问外网的数据包会由qrouter发送到br-ex接口。
再来查看IPTABLES规则:
1 2 3 4 5 6 7 [root@localhost ~(keystone_demo)] -P POSTROUTING ACCEPT -A POSTROUTING -j neutron-openvswi-POSTROUTING -A POSTROUTING -j neutron-postrouting-bottom -A POSTROUTING -j nova-api-POSTROUTING -A POSTROUTING -s 172.24.4.224/28 -o eth0 -m comment --comment "000 nat" -j MASQUERADE -A POSTROUTING -j nova-postrouting-bottom
可以发现IPTABLES会将源地址为网段172.24.4.224/28的数据包进行NAT由接口eth0发出。
经过上述分析，all-in-one模式的整体网络架构如图:
私有网络内的两个VM之间的数据通信，直接由br-int转发。虚拟机访问外网时，数据包则通过qrouter发送到br-ex接口，再由IPTABLES完成NAT从物理网卡发出。
