High packet drops occur during traffic bursts on a FortiGate VM hosted in Google Cloud Platform

admin

Description

This article describes the case when a FortiGate VM hosted on Google Cloud Platform (GCP) experiences significant packet loss and latency during high bursts of traffic. While interface-level counters do not show RX drops, the internal RX queue statistics indicate drops.

Scope

FortiGate VM, GCP.

Solution

The issue is caused by RX queue buffer exhaustion. When the GCP instance presents only one RX queue, the ring buffer descriptors are exhausted during traffic bursts, leading to packet drops. This occurs even if the CPU utilization is not high, as the bottleneck is at the NIC descriptor level rather than the CPU processing level.

The following command will list the interface counters. Notice that all the RX drops are zero.

diagnose hardware deviceinfo nic port1
Name: port1
Driver: gve
Version: 1.3.0
Bus: 0000:00:04.0
HWaddr: 42:01:0a:5c:1a:02
State: up
Link: up
Mtu: 1460
Supported:
Advertised:
Speed: 32000half
Auto: disabled
Rx packets: 14891007
Rx bytes: 1101897774
Rx compressed: 0
Rx dropped: 0 
Rx errors: 0   
Rx Length err: 0

Use the command below to get more detailed interface stats; it lists the drops on the RX queue.

diagnose hardware deviceinfo nic-stats port1
port1 statistics:
rx_packets: 1833537831
tx_packets: 1329988458
rx_bytes: 1997339306123
tx_bytes: 1541708454631
-- output trimmed---
rx_queue_drop_cnt[0]: 1716637 
rx_no_buffers_posted[0]: 1716637 
rx_drops_packet_over_mru[0]: 0
rx_drops_invalid_checksum[0]: 0

A non-zero and incrementing values of the counters (rx_queue_drop_cnt and rx_no_buffers_posted) indicate that packets are being dropped at the queue level as the NIC has run out of available descriptors.

To resolve the buffer exhaustion, increase the number of RX queues. This provides more descriptors and allows multiple CPUs to process packets in parallel. In Google Cloud Platform, this is achieved by changing the VM instance type to a supported machine series that provides multiple queues.

It is possible to notice more queue indexes ([0], [1]) when RX queues are increased.

diagnose hardware deviceinfo nic-stats port1
---output clipped---
Rx dropped: 0
Tx dropped: 0
rx_dropped: 0
tx_dropped: 0
rx_desc_err_dropped_pkt: 0
rx_dropped_pkt[0]: 0  <--
rx_queue_drop_cnt[0]: 0
rx_drops_packet_over_mru[0]: 0
rx_drops_invalid_checksum[0]: 0
rx_dropped_pkt[1]: 0  <--
rx_queue_drop_cnt[1]: 0
rx_drops_packet_over_mru[1]: 0
rx_drops_invalid_checksum[1]: 0

Read more here: Source link