When investigating a connectivity issue, you may experience errors like the below in the system logs:
nf_conntrack: table full, dropping packets
This error indicates the connection tracking table size has been exhausted. This can manifest with different symptoms, such as intermittent or consistent network timeouts.
The conntrack table keeps state on open connections that the kernel is translating. This occurs often in a Kubernetes cluster when pods access an external endpoint, or another service within the cluster. These scenarios use NAT and stateful firewall rules which are maintained as entries in the conntrack table.
By default, the table size is calculated based on the memory allocated to the node. This does not fit all workloads demands, for example in a microservice environment typically a higher number of inter-service connections could be expected without consuming a high amount of memory.
To output the current max table size:
To get a point in time count of the current entries in the table:
Note: With the
conntrack package installed, you can also use
nf_conntrack_max are close, it is indicating that the current workload requires a larger table size.
If the current number of entries are not approaching the table size, this could indicate that a burst of workload was experienced historically, in a containerized environment this can be common. For example, if the high-traffic Pods may now running on different nodes.
Increasing the conntrack table size is achieved with
Calculate a higher value, this can be applied to the node immediately with:
sysctl -w net.netfilter.nf_conntrack_max=<value>
To persist through reboot, add the tunable to either
/etc/sysctl.conf, or a specific config file in
For example, if your Linux distribution follows the /etc/sysctl.d/ directory structure:
echo "net.netfilter.nf_conntrack_max=<value>" > /etc/sysctl.d/10-conntrack-max.conf sysctl -p /etc/sysctl.d/10-conntrack-max.conf
This creates a new config file to set the table size at each boot.
Additionally, if you configure nodes with configuration management, UserData, or build custom images etc., you may wish to add this to your usual approach to configure this for future nodes.