mjhjmtu

I'm working on NixOS, creating some functions in C to programmatically manipulate veth pairs and assign IPv6 addresses in order to create a program for setting up a network for containers on a single host.

To do so, I'm using libnftnl and libmnl to handle the fine details of constructing rtnetlink packets and setns() to set the network namespace of the calling thread to the correct network namespace before performing actions such as bringing an interface up.

The general flow of the program goes like:

- create veth pairs in the host network namespace
- move one end of each veth pair to the corresponding network namespace
- set all the veth endpoints in the host network namespace to UP
- call `setns()` into each other network namespace to set
 the loopback interface and vethpair end to UP
- add an IPv6 address to each of the veth ends in the host network namespace
- call `setns()` into each other network namespace to set the IPv6 address
 of the other veth endpoints

The Problem

When I run the program, IPv6 addresses in the host network namespace aren't always assigned to the interfaces. All the previous steps occur and exist and even the IPv6 addresses are assigned to all interfaces in other network namespaces but not always in the host network namespace.

This happens when I run the program, delete all the interfaces and rerun the program within about a minute of each other.

Some solutions that I've come up with are that if I make the program sleep for, say 5 seconds, before assigning addresses to interfaces in the host namespace, I can run the program, delete the interfaces and rerun the program nonstop and the addresses are all assigned. I can also wait a while after deleting the interfaces before rerunning the program and this will also cause all IPv6 addresses to be assigned correctly.

Other weird stuff

When running the program, there are times when some IPv6 addresses in the host network namespace are added and some aren't. The netlink ACK message returned for the packets also indicate that no error occurred so I can't tell that an address has failed to be assigned until I run ip address after the program exits.

Since this program is eventually going to be part of a larger container orchestration system, I'm looking to see if there's a way to either detect or prevent this kind of error.

I've thought about it potentially being duplicate address detection or the IP address being still stored somewhere in the kernel for a while after being delete but I receive no error from the netlink module.

EDIT 1

Here's a bit of code showing the C code which deals with creating a rtnetlink message and the sending and receiving of netlink messages.

 1 #include <stdlib.h> 
 2 #include <time.h> 
 3 #include <poll.h> 
 4 
 5 /* Linux specific headers */ 
 6 #include <linux/if_link.h> 
 7 #include <linux/rtnetlink.h> 
 8 
 9 /* Libmnl dependency */ 
 10 #include <libmnl/libmnl.h> 
 11
 13 int send_recv(struct mnl_socket *sock, 
 14 struct mnl_nlmsg_batch *batch, 
 15 void *receive_buffer, uint32_t receive_size, 
 16 const int portid) 
 17 
 18 int fd = mnl_socket_get_fd(sock); 
 19 int timeout = 0; 
 20 int status; 
 21 nfds_t nfds = 1; 
 22 ssize_t receive_size; 
 23 struct pollfd fds[nfds]; 
 24 
 25 /* Send the buffered data via a pre-created netlink socket. */ 
 26 status = mnl_socket_sendto(sock, mnl_nlmsg_batch_head(batch), 
 27 mnl_nlmsg_batch_size(batch)); 
 28 if (status < 0) return -1; 
 29 
 30 fds[0].fd = fd; 
 31 fds[0].events = POLLIN; 
 32 
 33 while (poll(fds, nfds, timeout) > 0) 
 34 /* Receive the response from the kernel. */ 
 35 receive_size = mnl_socket_recvfrom(sock, receive_buf, receive_size); 
 36 if (receive_size == -1) 
 37 return -1; 
 38 
 39 /* Run a callback function on the response. 
 40 * cb_ctl_array is an array of function pointers based on the status of 
 41 * the response. 0 should be the sequence number, but the function also 
 42 * behaves correctly if the actual sequence number wasn't 0 and we feed it
 43 * 0 still. 
 44 */ 
 45 status = mnl_cb_run2(receive_buf, receive_size, 0, portid, NULL, 
 46 NULL, cb_ctl_array, MNL_ARRAY_SIZE(cb_ctl_array)); 
 47 
 48 if (status != MNL_CB_OK) 
 49 return -1; 
 50 
 51 
 52 return MNL_CB_OK; 
 53 
 54 
 55 
 56 int set_interface_address_message(void *message_buffer, int seq, 
 57 uint8_t prefix, 
 58 const char if_name[IFNAMSIZ], 
 59 const struct in6_addr *in6_addr) 
 60 
 76 NLM_F_ACK 

The sequence of ip commands that this program is meant to mirror is

sudo ip netns add A
sudo ip link add veth1 type veth peername veth2
sudo ip link set dev veth2 netns A
sudo ip link set dev veth1 up
sudo ip -n A link set dev veth2 up
sudo ip -n A link set dev lo up
sudo ip address add fc00::1/64 dev veth1
sudo ip -n A address add fc00::2/64 dev veth2

Running the above sequence of commands in a bash script has never failed to add an ipv6 address to an interface.