Linux ethernet driver ring buffer

While performing some tests with a congested 10Mbps link, a strange thing happened: The link was congested only on one direction and both endpoint queues were RED queues. Based on the parameters and the queue size, the delay between those two links should be something near 170ms. However, the delay was much larger: >300ms (!).

The “problem” was the ring-buffer of the underlying driver (e1000). This one used a buffer of 128 packets which when added to the average 150 packets in the queue, resulted in >300ms delay.

You can see this buffer by running:

#ethtool -g eth0

And you can modify this buffer by running:

#ethtool -G eth0 tx 80

This is the transmit buffer which (when filled) adds to the delay of the local queue.

Of course, in normal use, this buffer is a good thing as it will allow to get higher transfer rates easier (from the POV of the operating system). But when making experiments, this little thing gets in the way.

Another thing is that there seems to be a minimum value for this number. For example, on this card:

Marvell Technology Group Ltd. 88E8001 Gigabit Ethernet Controller

the minimum value is 80 (using e1000 driver from kernel 2.6.32)

So beware and don’t loose a week looking for this thing like I did.

NOTE: This is not related to the transmit queue length that is used on the interface, as shown by:

# ifconfig eth0
# ip link show eth0

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.