How To Fix Error nf_conntrack table full dropping packet in Linux Servers

In this article we will help you resolve an issue which can come up on Linux CentOS servers that creates network flapping and packet loss.

Issue

During network monitoring of one of our customers’ servers we detected a network flapping. Accessing the server via ssh was not possible (due to the heavy network usage) so we connected to it via iDRAC.

The following messages were seen in /var/log/messages (they may also appear in /var/log/kern):

$ tail -f /var/log/messages
Jan  14 02:11:05 centos kernel: : nf_conntrack: table full, dropping packet
Jan  14 02:11:09 centos kernel: : nf_conntrack: table full, dropping packet

A server can enter this panic state in cases that it’s under a DDoS attack or very likely to happen when there are a lot of network connections. Our client’s server was used for virtualization so one or more Virtual machines happened to generate a lot of network packets.

CentOS servers, by default, allow 65,536 connections. This limit is enough for lightly loaded servers, but it can be exhausted on high traffic servers.

How To Fix

During a disaster it is important to remain calm. Below you can read how to mitigate this issue and resolve the network flapping. We provide the solution for both CentOS 6 and CentOS 7 environments.

Solution for CentOS 6

Step 1: Check value of nf_conntrack_max

Run the following command to check the value of nf_conntrack_max:

$ sysctl net.netfilter.nf_conntrack_max

Step 2: Check number of active connections

$ cat /proc/sys/net/netfilter/nf_conntrack/count

If the value of nf_conntrack_max is reached, then the server is dropping packets already.

Step 3: Increase value for nf_conntrack_max *

$ sysctl -w net.netfilter.nf_conntrack_max=2097152

To make the above change persistent after reboot, do this:

$ echo "net.netfilter.nf_conntrack_max=2097152" >> /etc/sysctl.conf

* To properly calculate the ideal nf_conntrack_max value, use the following formula:

CONNTRACK_MAX = RAM_MEMORY_SIZE (in bytes) / 16384 / 2

Step 4: Restart Iptables

$ service iptables restart && service iptables save

Solution for CentOS 7

Step 1: Increase size of has table for conntrack connections *

$ echo "options nf_conntrack expect_hashsize=131072 hashsize=131072" >> /etc/modprobe.d/firewalld-sysctls.conf

* To calculate the new hash value use this formula:

HASHSIZE = CONNTRACK_MAX / 4

Step 2: Restart Firewalld

$ systemctl restart firewalld

Step 3: Verify new nf_conntrack_max value

$ sysctl -a | grep nf_conntrack_max