While performing some tests with a congested 10Mbps link, a strange thing happened: The link was congested only on one direction and both endpoint queues were RED queues. Based on the parameters and the queue size, the delay between those two links should be something near 170ms. However, the delay was much larger: >300ms (!).
The “problem” was the ring-buffer of the underlying driver (e1000). This one used a buffer of 128 packets which when added to the average 150 packets in the queue, resulted in >300ms delay.
You can see this buffer by running:
#ethtool -g eth0
And you can modify this buffer by running:
#ethtool -G eth0 tx 80
This is the transmit buffer which (when filled) adds to the delay of the local queue.
Of course, in normal use, this buffer is a good thing as it will allow to get higher transfer rates easier (from the POV of the operating system). But when making experiments, this little thing gets in the way.
Another thing is that there seems to be a minimum value for this number. For example, on this card:
Marvell Technology Group Ltd. 88E8001 Gigabit Ethernet Controller
the minimum value is 80 (using e1000 driver from kernel 2.6.32)
So beware and don’t loose a week looking for this thing like I did.
NOTE: This is not related to the transmit queue length that is used on the interface, as shown by:
# ifconfig eth0
# ip link show eth0
V13 