I'm trying to limit bandwidth with tc and check the results with iperf. I started like this:

# iperf -c 192.168.2.1
------------------------------------------------------------
Client connecting to 192.168.2.1, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.7 port 35213 connected with 192.168.2.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 830 MBytes 696 Mbits/sec

The two instances are directly connected with through Ethernet.

I then set up a htb qdisc with one default class to limit bandwidth to 1mbit/sec:

# tc qdisc add dev bond0 root handle 1: htb default 12
# tc class add dev bond0 parent 1: classid 1:12 htb rate 1mbit

But I don't get what I expect:

# iperf -c 192.168.2.1
------------------------------------------------------------
Client connecting to 192.168.2.1, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.7 port 35217 connected with 192.168.2.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-12.8 sec 768 KBytes 491 Kbits/sec

If I double the rate, the measured bandwidth does not change. What am I missing? Why doesn't the the measured bandwidth correspond to the 1mbit from the rate parameter? What parameters do I need to set to limit the bandwidth to an exact given rate?

However, the man page says that tbf should be the qdisc of choice for this task:

The Token Bucket Filter is suited for slowing traffic down to a precisely configured rate. Scales well to large bandwidths.

tbf requires parameters rate, burst and (limit| latency). So I tried the following without understanding how burst and (limit| latency) affect the available bandwidth:

# tc qdisc add dev bond0 root tbf rate 1mbit limit 10k burst 10k

This got me a measured bandwidth of 113 Kbits/sec. Playing around with those parameters didn't change that much until I noticed that adding a value for mtu changes things drastically:

# tc qdisc add dev bond0 root tbf rate 1mbit limit 10k burst 10k mtu 5000

resulted in a measured bandwidth of 1.00 Mbits/sec.

What parameters would I need to set to limit the bandwidth to an exact given rate?

Should I use the htb or tbf queueing discipline for this?

EDIT:

Based on these resources, I have made some tests:

• https://help.ubuntu.com/community/UbuntuBonding


• https://help.ubuntu.com/community/LinkAggregation


• /usr/share/doc/ifenslave-2.6/README.Debian.gz http://lartc.org/
  

I have tried the following setups.

On a Physical Machine

/etc/network/interfaces:

auto lo
iface lo inet loopback

auto br0
iface br0 inet dhcp
bridge_ports eth0

Measurement with iperf:

# tc qdisc add dev eth0 root handle 1: htb default 12
# tc class add dev eth0 parent 1: classid 1:12 htb rate 1mbit
# iperf -c 192.168.2.1
------------------------------------------------------------
Client connecting to 192.168.2.1, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.4 port 51804 connected with 192.168.2.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-11.9 sec 1.62 MBytes 1.14 Mbits/sec

Whereas the iperf server calculated a different bandwidth:

[ 4] local 192.168.2.1 port 5001 connected with 192.168.2.4 port 51804
[ 4] 0.0-13.7 sec 1.62 MBytes 993 Kbits/sec

On a Virtual Machine without Bonding

/etc/network/interfaces:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

Measurement with iperf:

# tc qdisc add dev eth0 root handle 1: htb default 12
# tc class add dev eth0 parent 1: classid 1:12 htb rate 1mbit
# iperf -c 192.168.2.1
------------------------------------------------------------
Client connecting to 192.168.2.1, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.7 port 34347 connected with 192.168.2.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-11.3 sec 1.62 MBytes 1.21 Mbits/sec

Whereas the iperf server calculated a different bandwidth:

[ 4] local 192.168.2.1 port 5001 connected with 192.168.2.7 port 34347
[ 4] 0.0-14.0 sec 1.62 MBytes 972 Kbits/sec

On a Virtual Machine with Bonding (tc configured on eth0)

/etc/network/interfaces:

auto lo
iface lo inet loopback

auto eth0
allow-bond0 eth0
iface eth0 inet manual
 bond-master bond0
 bond-primary eth0 eth1

auto eth1
allow-bond0 eth1
iface eth1 inet manual
 bond-master bond0
 bond-primary eth0 eth1

auto bond0
iface bond0 inet dhcp
 bond-slaves none
 bond-mode 1
# bond-arp-interval 250
# bond-arp-ip-target 192.168.2.1
# bond-arp-validate 3

Measurement with iperf:

# tc qdisc add dev eth0 root handle 1: htb default 12
# tc class add dev eth0 parent 1: classid 1:12 htb rate 1mbit
# iperf -c 192.168.2.1
------------------------------------------------------------
Client connecting to 192.168.2.1, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.9 port 49054 connected with 192.168.2.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-11.9 sec 1.62 MBytes 1.14 Mbits/sec

Whereas the iperf server calculated a different bandwidth:

[ 4] local 192.168.2.1 port 5001 connected with 192.168.2.9 port 49054
[ 4] 0.0-14.0 sec 1.62 MBytes 972 Kbits/sec

On a Virtual Machine with Bonding (tc configured on bond0)

/etc/network/interfaces:

auto lo
iface lo inet loopback

auto eth0
allow-bond0 eth0
iface eth0 inet manual
 bond-master bond0
 bond-primary eth0 eth1

auto eth1
allow-bond0 eth1
iface eth1 inet manual
 bond-master bond0
 bond-primary eth0 eth1

auto bond0
iface bond0 inet dhcp
 bond-slaves none
 bond-mode 1
# bond-arp-interval 250
# bond-arp-ip-target 192.168.2.1
# bond-arp-validate 3

Measurement with iperf:

# tc qdisc add dev bond0 root handle 1: htb default 12
# tc class add dev bond0 parent 1: classid 1:12 htb rate 1mbit
# iperf -c 192.168.2.1
------------------------------------------------------------
Client connecting to 192.168.2.1, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.9 port 49055 connected with 192.168.2.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-13.3 sec 768 KBytes 475 Kbits/sec

Whereas the iperf server calculated a different bandwidth:

[ 4] local 192.168.2.1 port 5001 connected with 192.168.2.9 port 49055
[ 4] 0.0-14.1 sec 768 KBytes 446 Kbits/sec

The result does not change if I remove eth1 (the passive interface) from the bond.

Conclusion

Traffic Control on a bond interface does not work, or at least not as expected. I will have to investigate further.

As a workaround one could add the queueing disciplines directly to the interfaces belonging to the bond.

When you are unsure about how tc works you can still monitor tc and look how the packets flow? You can use my script to monitor tc and need to run it in a terminal with lifted privilege. You can change wlan0 to another interface and you also need grep and awk:

 #!/bin/sh
 INTERVAL=15
 while sleep $INTERVAL
 do
 /usr/sbin/tc -s -d class show dev wlan0

 uptime
 more /proc/meminfo | grep MemFree | grep -v grep
 echo cache-name num-active-objs total-objs obj-size
 SKBUFF=`more /proc/slabinfo | grep skbuff | grep -v grep | awk 
 'print $2 print $3 print $4'`

 echo skbuff_head_cache: $SKBUFF
 done

Try increasing the burst/limit values. The token bucket algorithms scale well, but have a limited accuracy/speed ratio.

Accuracy is achieved by using a small bucket, speed by increasing the size of the tokens. Large tokens mean the rate at which they are replenished is decreased (tokens per second = bytes per second / bytes per token).

The rate parameter gives the average rate that is not to be exceeded, the burst or limit parameters give the size of the averaging window. As sending out a packet at line speed exceeds the set rate for the time where the packet is transferred, the averaging window needs to be at least large enough that sending a single packet does not push the entire window over the limit; if more packets fit in the window, the algorithm will have a better chance of hitting the target exactly.

run this before add queue discipline on bonding interface (bond0 in this case)

ipconfig bond0 txqueuelen 1000

it not work because software virtual device like bonding interface has no default queue.

Since bond devices doesn't have defined queue, setting the qdisc size explicitly fixes the issue for me.

Here is an example for a leaf qdisc to be used under HTB structure :
tc qdisc add dev $dev parent $parent handle $handle pfifo limit 1000