I use a netfilter-based daemon, ibd-judge (part of ipqbdb). It runs undisturbed from system boot to the next. Its job is to block certain network packets, based on values in a database and to iptables’ -j NFQUEUE targets. To achieve this, it loops around a call, mnl_socket_recvfrom(), which retrieves the headers of network packets from the kernel. This call can fail, returning, for example, ENOBUFS, “No buffer space available”, indicating that the daemon wasn’t fast enough to retrieve the packets from the kernel. This usually never happens because the daemon fetches only a tiny fraction of the packets (40 bytes) and because the Berkeley DB lookups are extremely fast. However, the daemon logs a single packet loss, almost every day, during the execution of the cron.daily scripts.
No script makes any attempt to contact ibd-judge. I believe run-parts, which manages the cron.daily scripts, executes them one at a time (this isn’t explicitly stated, but there are options to change the execution order which wouldn’t make sense if they were running concurrently; tested here). The packet loss occurs in the middle of the longest cron.daily process, which sends several reports via email. It pauses between messages, so yesterday’s 221 messages were sent in 16 minutes (one message every 4.3 seconds). The sleep time in the script was 1; I’ve now increased it to 2 to see if that changes anything. However, I doubt that excessive packet flow is actually the cause. There are times during the day when network peaks are higher than cron.daily. And internally initiated connections are much less susceptible to iptables filtering.
Maybe there’s something else that kicks in around 4am and affects the kernel buffers?
