mjhjmtu

I have a linux box where we see a lot of (30%) TCP retransmission in tcpdump when receiving (downloading) files from outside. Using the dropwatch utility we see many packet drops in kernel function net_receive_skb(). That means data have been received on the NIC, but later some of them are dropped in the kernel when processing the packets. The many dropped packets can explain the necessity of retransmissions.

dropwatch output is like the following:

dropwatch -l kas 
Initalizing kallsyms db
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
1 drops at tcp_rcv_established+906 (0xffffffff814d0a66)
6 drops at unix_dgram_connect+4ac (0xffffffff8151890c)
6 drops at unix_dgram_connect+4ac (0xffffffff8151890c)
19 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
5 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
9 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
7 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
6 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
14 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
15 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
2 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
2 drops at inet_csk_reset_xmit_timer.clone.1+265 (0xffffffff814d9cb5) ^CGot a stop message
dropwatch> exit
Shutting down ...

The system is a CentOS 6.2 with a 2.6.32 kernel (centOS package name 2.6.32-696.el6.x86_64). So I looked into the version of netif_receive_skb in kernel source code, trying to find the reason for the packet drops. And I see there is only one place calling kfree_skb (near the end of the function) that will leave a trace on dropped packets. The code is:

int netif_receive_skb(struct sk_buff *skb)

 struct packet_type *ptype, *pt_prev;
 struct net_device *orig_dev;
 struct net_device *null_or_orig;
 int ret = NET_RX_DROP;
 __be16 type;

 if (!skb->tstamp.tv64)
 net_timestamp(skb);

 if (skb->vlan_tci && vlan_hwaccel_do_receive(skb))
 return NET_RX_SUCCESS;

 /* if we've gotten here through NAPI, check netpoll */
 if (netpoll_receive_skb(skb))
 return NET_RX_DROP;

 if (!skb->iif)
 skb->iif = skb->dev->ifindex;

 null_or_orig = NULL;
 orig_dev = skb->dev;
 if (orig_dev->master) 
 if (skb_bond_should_drop(skb))
 null_or_orig = orig_dev; /* deliver only exact match */
 else
 skb->dev = orig_dev->master;
 

 __get_cpu_var(netdev_rx_stat).total++;

 skb_reset_network_header(skb);
 skb_reset_transport_header(skb);
 skb->mac_len = skb->network_header - skb->mac_header;

 pt_prev = NULL;

 rcu_read_lock();

#ifdef CONFIG_NET_CLS_ACT
 if (skb->tc_verd & TC_NCLS) 
 skb->tc_verd = CLR_TC_NCLS(skb->tc_verd);
 goto ncls;
 
#endif

 list_for_each_entry_rcu(ptype, &ptype_all, list) 

#ifdef CONFIG_NET_CLS_ACT
 skb = handle_ing(skb, &pt_prev, &ret, orig_dev);
 if (!skb)
 goto out;
ncls:
#endif

 skb = handle_bridge(skb, &pt_prev, &ret, orig_dev);
 if (!skb)
 goto out;
 skb = handle_macvlan(skb, &pt_prev, &ret, orig_dev);
 if (!skb)
 goto out;

 type = skb->protocol;
 list_for_each_entry_rcu(ptype,
 &ptype_base[ntohs(type) & PTYPE_HASH_MASK], list) 

 if (pt_prev) 
 ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
 else 
 kfree_skb(skb);
 /* Jamal, now you will not able to escape explaining
 * me how you were going to use this. :-)
 */
 ret = NET_RX_DROP;
 

out:
 rcu_read_unlock();
 return ret;

It looks like the call to kfree_skb will only happen when the skb->dev is not registered in any of the protocol's ptype list, so that the pt_prev remains NULL after the 2 loops against the ptype lists. This does not make sense as the system is dropping only a small part of all packages - meaning the device is "most of time registered in protocol ptype list but sometime not there".

So, question is - what mistakes did I make in understanding the dropwatch result and the netif_receive_skb code? And what is a more reasonable explanation of the packet drops being reported at this function?