Kubernetes flannel网络分析

flannel是coreos开源的Kubernetes CNI实现。它使用etcd或者Kubernetes API存储整个集群的网络配置。每个kubernetes节点上运行flanneld组件，它从etcd或者Kubernetes API获取集群的网络地址空间，并在空间内获取一个subnet,该节点上的容器IP都从这个subnet中分配，从而保证不同节点上的IP不会冲突。flannel通过不同的backend来实现跨主机的容器网络通信，目前支持udp,vxlan,host-gw等一系列backend实现。本文介绍vxlan backend下的容器通信过程。

flannel在v0.9.0版本上对vxlan的实现作了改动。源码中有一段非常详细的注释介绍了不同版本的设计与实现:

// Some design notes and history:
// VXLAN encapsulates L2 packets (though flannel is L3 only so don't expect to be able to send L2 packets across hosts)
// The first versions of vxlan for flannel registered the flannel daemon as a handler for both "L2" and "L3" misses
// - When a container sends a packet to a new IP address on the flannel network (but on a different host) this generates
//   an L2 miss (i.e. an ARP lookup)
// - The flannel daemon knows which flannel host the packet is destined for so it can supply the VTEP MAC to use.
//   This is stored in the ARP table (with a timeout) to avoid constantly looking it up.
// - The packet can then be encapsulated but the host needs to know where to send it. This creates another callout from
//   the kernal vxlan code to the flannel daemon to get the public IP that should be used for that VTEP (this gets called
//   an L3 miss). The L2/L3 miss hooks are registered when the vxlan device is created. At the same time a device route
//   is created to the whole flannel network so that non-local traffic is sent over the vxlan device.
//
// In this scheme the scaling of table entries (per host) is:
//  - 1 route (for the configured network out the vxlan device)
//  - One arp entry for each remote container that this host has recently contacted
//  - One FDB entry for each remote host
//
// The second version of flannel vxlan removed the need for the L3MISS callout. When a new remote host is found (either
// during startup or when it's created), flannel simply adds the required entries so that no further lookup/callout is required.
//
//
// The latest version of the vxlan backend  removes the need for the L2MISS too, which means that the flannel deamon is not
// listening for any netlink messages anymore. This improves reliability (no problems with timeouts if
// flannel crashes or restarts) and simplifies upgrades.
//
// How it works:
// Create the vxlan device but don't register for any L2MISS or L3MISS messages
// Then, as each remote host is discovered (either on startup or when they are added), do the following
// 1) create routing table entry for the remote subnet. It goes via the vxlan device but also specifies a next hop (of the remote flannel host).
// 2) Create a static ARP entry for the remote flannel host IP address (and the VTEP MAC)
// 3) Create an FDB entry with the VTEP MAC and the public IP of the remote flannel daemon.
//
// In this scheme the scaling of table entries is linear to the number of remote hosts - 1 route, 1 arp entry and 1 FDB entry per host
//
// In this newest scheme, there is also the option of skipping the use of vxlan for hosts that are on the same subnet,
// this is called "directRouting"

v0.9.0之前版本的实现主要依赖vxlan内核模块的L2MISS和L3MISS消息机制。L2MISS是指vxlan设备在ARP表中找不到内层IP对应的MAC地址时会给用户态程序发送netlink消息。L3MISS是指vxlan设备在FDB表中找不到VXLAN协议内层MAC地址所属的VTEP的IP地址时会给用户态程序发送netlink消息。之前的文章<<动态维护FDB表项实现VXLAN通信>>介绍过相关概念和操作。本文主要分析v0.9.0版本上的实现方式。

之前的方式实现是，flanneld作为L2MISS和L3MISS消息的处理器,当收到相应消息时从etcd或者kubernetes API获取到相应的ARP或者FDB信息来填充相应条目。如果flanneld异常退出，那么整个容器网络集群的网络就中断了。这是一个很大的隐患。v0.9.0实现不再需要处理L2MISS和L3MISS消息，而是由flanneld通过watch etcd或者kubernetes API的相关节点信息来动态地维护各节点通信所需的ARP、FDB以及路由条目。即使flanneld崩溃，整个集群网络数据转发依然可以运行。这个实现很优雅，每个节点只需要一条路由，一个ARP缓存条目和一个FDB条目。

下面在实验环境中分析flannel vxlan的网络通信过程。整个网络架构如图:

CNI配置文件/etc/cni/net.d/09-flannel.conf内容如下:

{
    "name": "cbr0",
    "cniVersion": "0.3.1",
    "type": "flannel",

    "delegate": {
        "isDefaultGateway": true
    }
}

节点上每个pod会有一对veth pair设备，其中一端放在pod的network namespace中，另一端在宿主机上接在cni0网桥上。flanneld启动时创建了vxlan设备:flannel.1。

node1上的flannel网络信息如下,分配的subnet为10.230.41.1/24:

[root@node1 ~]# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.230.0.0/16
FLANNEL_SUBNET=10.230.41.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

node2上的flannel网络信息如下, 分配的subnet为10.230.93.1/24:

[root@node2 ~]# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.230.0.0/16
FLANNEL_SUBNET=10.230.93.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

我们来看10.230.41.17向10.230.93.2发送数据包的过程。

10.230.93.2与10.230.41.17不在同一二层网络，因而需要查找路由来决定由哪个设备发送到哪里。10.230.41.17的路由如下:

[root@master1 ~]# kubectl exec -it busybox2-6f8fdb784d-r6ln2 -- ip route
default via 10.230.41.1 dev eth0
10.230.0.0/16 via 10.230.41.1 dev eth0
10.230.41.0/24 dev eth0 scope link  src 10.230.41.17

匹配到默认路由，因而需要发送到网关10.230.41.1。10.230.41.1配置在网桥cni0上。内核通过ARP请求获得10.230.41.1的MAC地址, 将数据包转发到cni0上。

[root@node1 ~]# ip addr show dev cni0
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 86:99:b6:37:95:b2 brd ff:ff:ff:ff:ff:ff
    inet 10.230.41.1/24 brd 10.230.41.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::8499:b6ff:fe37:95b2/64 scope link
       valid_lft forever preferred_lft forever

flanneld在加入集群时会为每个其他节点生成一条on-link路由，on-link路由表示是直连路由，匹配该条路由的数据包将触发ARP请求获取目的IP的MAC地址。在node1上查看路由信息:

[root@node1 ~]# ip route show dev flannel.1
10.230.93.0/24 via 10.230.93.0 onlink

cni0设备根据这条路由将数据包转给vxlan设备flannel.1，并且接收端的IP地址为10.230.93.0, 需要通过ARP获取MAC地址。

flannel.1的信息如下, 可以看到没有开启l2miss和l3miss:

[root@node1 ~]# ip -d link show flannel.1
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/ether a6:f7:8b:a4:60:b0 brd ff:ff:ff:ff:ff:ff promiscuity 0
    vxlan id 1 local 10.240.0.101 dev eth1 srcport 0 0 dstport 8472 nolearning ageing 300 noudpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

vxlan设备需要对接收到的数据包进行VXLAN协议封装。它需要知道对端10.230.93.0的MAC地址。而flanneld在启动时已经根据从etcd或kubernetes API获取到的信息写入到ARP表中:

[root@node1 ~]# ip neigh show dev flannel.1
10.230.93.0 lladdr 2a:02:24:58:e9:07 PERMANENT

这样获取到10.230.93.0的MAC地址后，就可以完成内层数据的封装。数据包封装完成后，它需要获得对应这个MAC地址的VTEP的IP地址。flanneld已经在启动时写入FDB条目:

[root@node1 ~]# bridge fdb show dev flannel.1
2a:02:24:58:e9:07 dst 10.240.0.102 self permanent

可以看到2a:02:24:58:e9:07对应的VTEP IP为10.240.0.102。这时flannel.1这个vxlan设备知道数据包要发送的目的IP，根据主机的路由策略从eth1设备发出。主机路由信息如下:

[root@node1 ~]# ip route
default via 10.0.2.2 dev eth0
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
10.230.41.0/24 dev cni0 proto kernel scope link src 10.230.41.1
10.230.93.0/24 via 10.230.93.0 dev flannel.1 onlink
10.240.0.0/24 dev eth1 proto kernel scope link src 10.240.0.101
169.254.0.0/16 dev eth0 scope link metric 1002
169.254.0.0/16 dev eth1 scope link metric 1003

数据包到达node2的eth1后，eth1将收到VXLAN数据包, 数据包中的MAC地址为:2a:02:24:58:e9:07, 正是node2节点上flannel.1的地址, 将它转给flannel.1设备:

[root@node2 ~]# ip addr show flannel.1
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether 2a:02:24:58:e9:07 brd ff:ff:ff:ff:ff:ff
    inet 10.230.93.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::2802:24ff:fe58:e907/64 scope link
       valid_lft forever preferred_lft forever

flannel.1解包之后，根据内层目的地址:10.240.93.2查找路由转发到cni0:

[root@node2 ~]# ip route
default via 10.0.2.2 dev eth0 proto dhcp metric 100
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100
10.230.41.0/24 via 10.230.41.0 dev flannel.1 onlink
10.230.93.0/24 dev cni0 proto kernel scope link src 10.230.93.1
10.240.0.0/24 dev eth1 proto kernel scope link src 10.240.0.102 metric 101

cni0再通过ARP请求获得10.230.93.2的MAC地址，从而将数据包转发到相应的POD中的veth pair设备，从而到达容器中。

回包的路径是一样的，不再详述。

下面简要分析一下flanneld的源码实现。

main函数中首先调用newSubnetManager创建SubnetManager。

sm, err := newSubnetManager()
if err != nil {
    log.Error("Failed to create SubnetManager: ", err)
    os.Exit(1)
}
log.Infof("Created subnet manager: %s", sm.Name())

SubnetManager用于向网络配置存储租用或续组subnet。每个节点都会有自己的一个subnet,保证了节点之间的IP不会冲突。

func newSubnetManager() (subnet.Manager, error) {
    if opts.kubeSubnetMgr {
        return kube.NewSubnetManager(opts.kubeApiUrl, opts.kubeConfigFile)
    }

    cfg := &etcdv2.EtcdConfig{
        Endpoints: strings.Split(opts.etcdEndpoints, ","),
        Keyfile:   opts.etcdKeyfile,
        Certfile:  opts.etcdCertfile,
        CAFile:    opts.etcdCAFile,
        Prefix:    opts.etcdPrefix,
        Username:  opts.etcdUsername,
        Password:  opts.etcdPassword,
    }

    // Attempt to renew the lease for the subnet specified in the subnetFile
    prevSubnet := ReadSubnetFromSubnetFile(opts.subnetFile)

    return etcdv2.NewLocalManager(cfg, prevSubnet)
}

如果命令行参数中指定了kube-subnet-mgr, 则使用kubernetes API作为全局网络配置存储，否则使用etcd。

接着调用getConfig从全局配置存储获取网络配置, 包括容器集群的网络信息，backend的配置等等:

// Fetch the network config (i.e. what backend to use etc..).
config, err := getConfig(ctx, sm)
if err == errCanceled {
    wg.Wait()
    os.Exit(0)
}

比如，我的实验环境写到etcd的配置内容为:

{"Network":"10.230.0.0/16","SubnetLen":24, "Backend":{"Type": "vxlan"}}

接下来，main函数会调用backend.NewManager。

// Create a backend manager then use it to create the backend and register the network with it.
bm := backend.NewManager(ctx, sm, extIface)
be, err := bm.GetBackend(config.BackendType)
if err != nil {
    log.Errorf("Error fetching backend: %s", err)
    cancel()
    wg.Wait()
    os.Exit(1)
}

bn, err := be.RegisterNetwork(ctx, config)
if err != nil {
    log.Errorf("Error registering network: %s", err)
    cancel()
    wg.Wait()
    os.Exit(1)
}

开头时也介绍过，flannel通过backend机制来支持各种不同的跨主机通信方式。不同的实现方式会在init函数中向backend注册自己的构造函数。比如，package vxlan的init函数:

func init() {
    backend.Register("vxlan", New)
}

const (
    defaultVNI = 1
)

type VXLANBackend struct {
    subnetMgr subnet.Manager
    extIface  *backend.ExternalInterface
}

func New(sm subnet.Manager, extIface *backend.ExternalInterface) (backend.Backend, error) {
    backend := &VXLANBackend{
        subnetMgr: sm,
        extIface:  extIface,
    }

    return backend, nil
}

be.RegisterNetwork会调用到package vxlan的RegisterNetwork:

func (be *VXLANBackend) RegisterNetwork(ctx context.Context, config *subnet.Config) (backend.Network, error) {
    // Parse our configuration
    cfg := struct {
        VNI           int
        Port          int
        GBP           bool
        DirectRouting bool
    }{
        VNI: defaultVNI,
    }

    if len(config.Backend) > 0 {
        if err := json.Unmarshal(config.Backend, &cfg); err != nil {
            return nil, fmt.Errorf("error decoding VXLAN backend config: %v", err)
        }
    }
    log.Infof("VXLAN config: VNI=%d Port=%d GBP=%v DirectRouting=%v", cfg.VNI, cfg.Port, cfg.GBP, cfg.DirectRouting)

    devAttrs := vxlanDeviceAttrs{
        vni:       uint32(cfg.VNI),
        name:      fmt.Sprintf("flannel.%v", cfg.VNI),
        vtepIndex: be.extIface.Iface.Index,
        vtepAddr:  be.extIface.IfaceAddr,
        vtepPort:  cfg.Port,
        gbp:       cfg.GBP,
    }

    dev, err := newVXLANDevice(&devAttrs)
    if err != nil {
        return nil, err
    }
    dev.directRouting = cfg.DirectRouting

    subnetAttrs, err := newSubnetAttrs(be.extIface.ExtAddr, dev.MACAddr())
    if err != nil {
        return nil, err
    }

    lease, err := be.subnetMgr.AcquireLease(ctx, subnetAttrs)
    switch err {
    case nil:
    case context.Canceled, context.DeadlineExceeded:
        return nil, err
    default:
        return nil, fmt.Errorf("failed to acquire lease: %v", err)
    }

    // Ensure that the device has a /32 address so that no broadcast routes are created.
    // This IP is just used as a source address for host to workload traffic (so
    // the return path for the traffic has an address on the flannel network to use as the destination)
    if err := dev.Configure(ip.IP4Net{IP: lease.Subnet.IP, PrefixLen: 32}); err != nil {
        return nil, fmt.Errorf("failed to configure interface %s: %s", dev.link.Attrs().Name, err)
    }

    return newNetwork(be.subnetMgr, be.extIface, dev, ip.IP4Net{}, lease)
}

RegisterNetwork函数会调用newVXLANDevice创建一个vxlan设备，就对应我们实验环境中的flannel.1。从代码也可以看到flannel.1设备名中的1指的是VNI, 我们可以通过在全局配置存储中设置为其他值。然后获取本地VTEP的IP地址以及vxlan设备的MAC地址填充到subnetAttrs结构调用be.subnetMgr.AcquireLease。这最终会调用到package etcdv2的tryAcquireLease。tryAcquireLease则会调用m.registry.createSubnet或者m.registry.updateSubnet去向etcd中写入相应的Subnet信息，完成相应Subnet的租用。这时，如果已经有其他节点的flanneld在watch etcd上的subnets的key，则会触发添加路由、ARP及FDB条目的逻辑。这个下面我们再详细描述具体实现。之后，调用dev.Configure给vxlan设备配置一个掩码为32的地址防止广播路由创建。

RegisterNetwork返回后，main函数会调用WriteSubnetFile将获取到的网络信息写入subnetFile中，默认是/run/flannel/subnet.env，后续flanneld再启动时就会优先尝试使用这个文件中记录的信息去续组subnet:

if err := WriteSubnetFile(opts.subnetFile, config.Network, opts.ipMasq, bn); err != nil {
    // Continue, even though it failed.
    log.Warningf("Failed to write subnet file: %s", err)
} else {
    log.Infof("Wrote subnet file to %s", opts.subnetFile)
}

接着，main函数中启动一个goroutine去运行bn.Run:

// Start "Running" the backend network. This will block until the context is done so run in another goroutine.
log.Info("Running backend.")
wg.Add(1)
go func() {
    bn.Run(ctx)
    wg.Done()
}()

这会调用到package vxlan的Run实现，它会调用subnet.WatchLeases去获取全局范围的subnet情况:

func (nw *network) Run(ctx context.Context) {
    wg := sync.WaitGroup{}

    log.V(0).Info("watching for new subnet leases")
    events := make(chan []subnet.Event)
    wg.Add(1)
    go func() {
        subnet.WatchLeases(ctx, nw.subnetMgr, nw.SubnetLease, events)
        log.V(1).Info("WatchLeases exited")
        wg.Done()
    }()

    defer wg.Wait()

    for {
        select {
        case evtBatch := <-events:
            nw.handleSubnetEvents(evtBatch)

        case <-ctx.Done():
            return
        }
    }
}

package subnet的WatchLeases函数中会一直循环调用sm.WatchLeases。sm.WatchLeases首次运行时会获取到当前etcd中已有的subnet信息，之后则开始watch etcd中subnets key获得变更的subnet信息。这些subnet信息传送给channel:receiver:

func WatchLeases(ctx context.Context, sm Manager, ownLease *Lease, receiver chan []Event) {
    lw := &leaseWatcher{
        ownLease: ownLease,
    }
    var cursor interface{}

    for {
        res, err := sm.WatchLeases(ctx, cursor)
        if err != nil {
            if err == context.Canceled || err == context.DeadlineExceeded {
                return
            }

            log.Errorf("Watch subnets: %v", err)
            time.Sleep(time.Second)
            continue
        }

        cursor = res.Cursor

        var batch []Event

        if len(res.Events) > 0 {
            batch = lw.update(res.Events)
        } else {
            batch = lw.reset(res.Snapshot)
        }

        if len(batch) > 0 {
            receiver <- batch
        }
    }
}

receiver的接收端协程则调用nw.handleSubnetEvents(evtBatch)来处理这些消息:

func (nw *network) handleSubnetEvents(batch []subnet.Event) {
    for _, event := range batch {
        sn := event.Lease.Subnet
        attrs := event.Lease.Attrs
        if attrs.BackendType != "vxlan" {
            log.Warningf("ignoring non-vxlan subnet(%s): type=%v", sn, attrs.BackendType)
            continue
        }

        var vxlanAttrs vxlanLeaseAttrs
        if err := json.Unmarshal(attrs.BackendData, &vxlanAttrs); err != nil {
            log.Error("error decoding subnet lease JSON: ", err)
            continue
        }

        // This route is used when traffic should be vxlan encapsulated
        vxlanRoute := netlink.Route{
            LinkIndex: nw.dev.link.Attrs().Index,
            Scope:     netlink.SCOPE_UNIVERSE,
            Dst:       sn.ToIPNet(),
            Gw:        sn.IP.ToIP(),
        }
        vxlanRoute.SetFlag(syscall.RTNH_F_ONLINK)

        // directRouting is where the remote host is on the same subnet so vxlan isn't required.
        directRoute := netlink.Route{
            Dst: sn.ToIPNet(),
            Gw:  attrs.PublicIP.ToIP(),
        }
        var directRoutingOK = false
        if nw.dev.directRouting {
            routes, err := netlink.RouteGet(attrs.PublicIP.ToIP())
            if err != nil {
                log.Errorf("Couldn't lookup route to %v: %v", attrs.PublicIP, err)
                continue
            }
            if len(routes) == 1 && routes[0].Gw == nil {
                // There is only a single route and there's no gateway (i.e. it's directly connected)
                directRoutingOK = true
            }
        }

        switch event.Type {
        case subnet.EventAdded:
            if directRoutingOK {
                log.V(2).Infof("Adding direct route to subnet: %s PublicIP: %s", sn, attrs.PublicIP)

                if err := netlink.RouteReplace(&directRoute); err != nil {
                    log.Errorf("Error adding route to %v via %v: %v", sn, attrs.PublicIP, err)
                    continue
                }
            } else {
                log.V(2).Infof("adding subnet: %s PublicIP: %s VtepMAC: %s", sn, attrs.PublicIP, net.HardwareAddr(vxlanAttrs.VtepMAC))
                if err := nw.dev.AddARP(neighbor{IP: sn.IP, MAC: net.HardwareAddr(vxlanAttrs.VtepMAC)}); err != nil {
                    log.Error("AddARP failed: ", err)
                    continue
                }

                if err := nw.dev.AddFDB(neighbor{IP: attrs.PublicIP, MAC: net.HardwareAddr(vxlanAttrs.VtepMAC)}); err != nil {
                    log.Error("AddFDB failed: ", err)

                    // Try to clean up the ARP entry then continue
                    if err := nw.dev.DelARP(neighbor{IP: event.Lease.Subnet.IP, MAC: net.HardwareAddr(vxlanAttrs.VtepMAC)}); err != nil {
                        log.Error("DelARP failed: ", err)
                    }

                    continue
                }

                // Set the route - the kernel would ARP for the Gw IP address if it hadn't already been set above so make sure
                // this is done last.
                if err := netlink.RouteReplace(&vxlanRoute); err != nil {
                    log.Errorf("failed to add vxlanRoute (%s -> %s): %v", vxlanRoute.Dst, vxlanRoute.Gw, err)

                    // Try to clean up both the ARP and FDB entries then continue
                    if err := nw.dev.DelARP(neighbor{IP: event.Lease.Subnet.IP, MAC: net.HardwareAddr(vxlanAttrs.VtepMAC)}); err != nil {
                        log.Error("DelARP failed: ", err)
                    }

                    if err := nw.dev.DelFDB(neighbor{IP: event.Lease.Attrs.PublicIP, MAC: net.HardwareAddr(vxlanAttrs.VtepMAC)}); err != nil {
                        log.Error("DelFDB failed: ", err)
                    }

                    continue
                }
            }
        case subnet.EventRemoved:
            if directRoutingOK {
                log.V(2).Infof("Removing direct route to subnet: %s PublicIP: %s", sn, attrs.PublicIP)
                if err := netlink.RouteDel(&directRoute); err != nil {
                    log.Errorf("Error deleting route to %v via %v: %v", sn, attrs.PublicIP, err)
                }
            } else {
                log.V(2).Infof("removing subnet: %s PublicIP: %s VtepMAC: %s", sn, attrs.PublicIP, net.HardwareAddr(vxlanAttrs.VtepMAC))

                // Try to remove all entries - don't bail out if one of them fails.
                if err := nw.dev.DelARP(neighbor{IP: sn.IP, MAC: net.HardwareAddr(vxlanAttrs.VtepMAC)}); err != nil {
                    log.Error("DelARP failed: ", err)
                }

                if err := nw.dev.DelFDB(neighbor{IP: attrs.PublicIP, MAC: net.HardwareAddr(vxlanAttrs.VtepMAC)}); err != nil {
                    log.Error("DelFDB failed: ", err)
                }

                if err := netlink.RouteDel(&vxlanRoute); err != nil {
                    log.Errorf("failed to delete vxlanRoute (%s -> %s): %v", vxlanRoute.Dst, vxlanRoute.Gw, err)
                }
            }
        default:
            log.Error("internal error: unknown event type: ", int(event.Type))
        }
    }
}

这里我们忽略directRouting相关内容。EventAdded表示有新的节点上线，首先调用nw.dev.AddARP给vxlan设备添加ARP条目，MAC和IP分别为新上线节点上vxlan设备的MAC地址以及上面所配置的32位掩码的IP地址。接着调用nw.dev.AddFDB在vxlan设备上添加FDB条目，MAC和IP分别为新上线节点上的vxlan设备的MAC地址以及新节点上的VTEP的IP地址。最后，再调用netlink.RouteReplace(&vxlanRoute)去添加经由32位掩码地址到达新上线subnet的路由。代码注释里也说明了，最后再添加路由是为了防止在ARP缓存没有填加的情况下发起ARP请求。

EventRemoved表示有节点下线，这里分别调用nw.dev.DelARP,nw.dev.DelFDB,netlink.RouteDel删除相应的ARP,FDB和路由条目。

在main函数的逻辑里接下来还会调用MonitorLease去定期续租subnet，这里不再详述。

参考:

• https://www.cnblogs.com/robinunix/articles/13275530.html
• https://programmer.ink/think/5da939768e5cb.html