Udp packet drops and packet receive error difference


#1

If you write a program which receives very high amount of udp packets, it is good to know whether all the packets processed by your program or not.

To get the information about statictics of udp stack, you can use netstat with -anus parameter like that:

$ netstat -anus
...
Udp:
    531412134 packets received
    125 packets to unknown port received.
    38491 packet receive errors
    531247364 packets sent
...

In this example there are 125 unknown port errors. It can be normal because you can’t control the incoming traffic if no firewall exists and any host can send udp packets to any port, which your server doesn’t listen. In this scenario, you don’t much worry about it. But if this number is so high on average for a day, you may want to capture udp traffics and make further analysis on that.

Another reason for unknown port case can be very important issue. If your program crashes randomly, your system may be receiving packets but because of the crash there are no software to accept that packets, so unknown port error counter increments. You have to fix this at the software level.

There are 38491 packet receive errors and we have to take it seriously. Packet receive errors doesn’t include the problems occured on network card level, it shows only received packets on udp protocol stack. Main reasons for packet receive errors:

  • udp packet header corruption or checksum problems
  • packet receive buffer problems in application or kernel side

Unlike TCP, UDP protocol does not have built-in flow-control capabilities, so if you can’t process all of the received packets fast enough, kernel will start to drop new incoming packets because of the socket receive buffer is full. When you don’t make any tuning on udp stack, default udp receive buffer size is between 32-128 kilobytes per socket. You can set it to much higher value with setsockopt like below:

int size = 2 * 1024 * 1024;
setsockopt(socket, SOL_SOCKET, SO_RCVBUF, &size, (socklen_t)sizeof(int));

2 MB’s of receive buffer handle up to 1 Gbps of data if you can process fast, but you can also increase it to 16 or 32 MB if you need.

Please note that, you can’t set socket receive buffer to maximum value defined in kernel which you can see on /proc/sys/net/core/rmem_max. You have to change this value to use big socket receive buffer size in application:

$ sudo sysctl -w net.core.rmem_max=33554432

There is one more parameter: netdev_max_backlog. It controls the number of packets allowed to queue for network cards in kernel side. If you are receiving very high level of traffic, you may want to inrease packet backlog queue in kernel to 2000 (Default value is 1000):

$ sudo sysctl -w net.core.netdev_max_backlog=2000

After these settings, you must check udp packet statistics again. There is no way to get same statistics grouped by process, so they are always showing the whole udp protocol stack. It will be reset with system reboot.

See also: man setsockopt