Hierarchical Token Bucket

In tc(8) command syntax: “kbps means kilobytes and kbit means kilobits”

Warning: iptables MARK is a “non-terminating” target. It means that You should have the most generic rules first. Packet will be marked with the last matching rule, unless you use -j RETURN target.

HTB allows to control only outgoing traffic on interface. Use IMQ if you want to control all traffic from Internet (to server and to LAN). It gives you much more flexibility.

Interactive SSH session requires at least 10 KB/s of guaranteed bandwith (rate). 15-20 KB/s would be better.

To classify peer-to-peer (bulk) traffic ipp2pis a great solution.

Examples

2mbit / 256kbit connection (no IMQ):

outgoing traffic: 256kbit = 32 kbps
1  tcp ssh, telnet, ftp21 from server             2 kbps prio 1
2  src udp/53 or dst udp/53                       2 kbps prio 2
3  http, https traffic from server               13 kbps prio 2
4  tcp imap,imaps from server                    11 kbps prio 3
5  other traffic (from server or LAN)             4 kbps prio 5

incoming traffic: 2mbit = 2048kbit = 256 kbps
6  traffic from server to LAN                   100 mbit
   traffic from Internet to LAN                1600 kbit
7  from Inet tcp/21,22,23 to LAN traffic         32 kbit prio 1
8  from Inet tcp/80,443 to LAN traffic          634 kbit prio 2
9  from Inet tcp/110,995,143,993 to LAN traffic 634 kbit prio 3
10 other traffic to LAN                         300 kbit prio 5
      

Convert it to rules:

iptables -t mangle -A POSTROUTING -o eth2 \
  -s AA.BB.CC.DD -j MARK --set-mark 5
iptables -t mangle -A POSTROUTING -o eth2 \ 
  -s 192.168.0.0/16 -j MARK --set-mark 5
iptables -t mangle -A POSTROUTING -o eth2 \
  -p udp -s AA.BB.CC.DD --sport 53 -j MARK --set-mark 2
iptables -t mangle -A POSTROUTING -o eth2 \
  -p udp -s AA.BB.CC.DD --dport 53 -j MARK --set-mark 2
...

for example:

• iptables -A POSTROUTING -o eth1 -p tcp -m multiport –sports 21,22,23 -j MARK –set-mark 0×1
• iptables -A POSTROUTING -o eth1 -p tcp -m multiport –sports 21,22,23 -j RETURN
• iptables -A POSTROUTING -o eth1 -p udp -m udp –sport 53 -j MARK –set-mark 0×2
• iptables -A POSTROUTING -o eth1 -p udp -m udp –dport 53 -j MARK –set-mark 0×2
• iptables -A POSTROUTING -o eth1 -p udp -m udp –sport 53 -j RETURN
• iptables -A POSTROUTING -o eth1 -p udp -m udp –dport 53 -j RETURN
• iptables -A POSTROUTING -s AA.BB.CC.DD -o eth1 -p tcp -m multiport –sports 80,443 -j MARK –set-mark 0×3
• iptables -A POSTROUTING -s AA.BB.CC.DD -o eth1 -p tcp -m multiport –sports 80,443 -j RETURN
• iptables -A POSTROUTING -o eth1 -p tcp -m multiport –dports 80,443 -j MARK –set-mark 0×4
• iptables -A POSTROUTING -o eth1 -p tcp -m multiport –dports 80,443 -j RETURN
• iptables -A POSTROUTING -s AA.BB.CC.DD -o eth1 -p tcp -m multiport –sports 110,995 -j MARK –set-mark 0×5
• iptables -A POSTROUTING -s AA.BB.CC.DD -o eth1 -p tcp -m multiport –sports 110,995 -j RETURN
• iptables -A POSTROUTING -o eth1 -j MARK –set-mark 0×6

To reset configuration (delete HTB disciplines):

tc qdisc del root dev eth0
tc qdisc del root dev eth1
tc qdisc del root dev eth2

HTB root:

tc qdisc add dev eth2 root handle 1:0 htb
tc class add dev eth2 parent 1:0 classid 1:1 htb rate 32kbps ceil 32kbps

classes:

tc class add dev eth2 parent 1:1 classid 1:2 htb rate  2kbps ceil 32kbps prio 1
tc class add dev eth2 parent 1:1 classid 1:3 htb rate  2kbps ceil 32kbps prio 2
tc class add dev eth2 parent 1:1 classid 1:4 htb rate 13kbps ceil 32kbps prio 3
tc class add dev eth2 parent 1:1 classid 1:5 htb rate  9kbps ceil 32kbps prio 4
tc class add dev eth2 parent 1:1 classid 1:6 htb rate  6kbps ceil 32kbps prio 5

filters:

tc filter add dev eth2 protocol ip parent 1:0 handle 1 fw flowid 1:2
tc filter add dev eth2 protocol ip parent 1:0 handle 2 fw flowid 1:3
tc filter add dev eth2 protocol ip parent 1:0 handle 3 fw flowid 1:4
tc filter add dev eth2 protocol ip parent 1:0 handle 4 fw flowid 1:5
tc filter add dev eth2 protocol ip parent 1:0 handle 5 fw flowid 1:6

sfq on tree nodes:

tc qdisc add dev eth2 parent 1:2 handle 2:0 sfq perturb 10
tc qdisc add dev eth2 parent 1:3 handle 3:0 sfq perturb 10
tc qdisc add dev eth2 parent 1:4 handle 4:0 sfq perturb 10
tc qdisc add dev eth2 parent 1:5 handle 5:0 sfq perturb 10
tc qdisc add dev eth2 parent 1:6 handle 6:0 sfq perturb 10

3600kbit / 256kbit connection:

 Outgoing: 32kbps
1  tcp from server, ports 21,22,23       2kbps prio 1
2  udp from server, from or to 53        2kbps prio 2
3  tcp from server, ports 80,443        13kbps prio 3
4  tcp from server, port 110             9kbps prio 4
5  other traffic                         6kbps prio 5

750kbit = 93 kbps outgoing interface

1  udp ports 27960, 44830, 44831          prio 1 20kbps
2  tcp ssh, telnet, ftp21                 prio 2 20kbps
3  udp traffic, port 53                   prio 3 2kbps
4  tcp traffic, src ports 80, 443         prio 4
5  tcp traffic, dst ports 80, 443         prio 5
6  tcp traffic, dst ports 25, 465         prio 6
7  other traffic                          prio 8
8  outgoing p2p traffic                   prio 9 1kbps, ceil 1kbps

8 mb/s / 640 kb/s – bandwidth limits per IP in LAN

tc qdisc add dev eth0 root handle 1:0 htb
tc class add dev eth0 parent 1:0 classid 1:1 htb rate 79kbps ceil 79kbps
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 18kbps ceil 73kbps
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 9kbps ceil 25kbps
tc class add dev eth0 parent 1:1 classid 1:4 htb rate 9kbps ceil 25kbps

tc filter add dev eth0 protocol ip parent 1:0 handle 2 fw flowid 1:2
tc filter add dev eth0 protocol ip parent 1:0 handle 3 fw flowid 1:3
tc filter add dev eth0 protocol ip parent 1:0 handle 4 fw flowid 1:4

tc qdisc add dev eth0 parent 1:2 handle 2:0 sfq perturb 10
tc qdisc add dev eth0 parent 1:3 handle 3:0 sfq perturb 10
tc qdisc add dev eth0 parent 1:4 handle 4:0 sfq perturb 10

# incomming traffic:
tc qdisc add dev br0 root handle 1:0 htb
tc class add dev br0 parent 1:0 classid 1:1 htb rate 13800kbps ceil 13800kbps

tc class add dev br0 parent 1:1 classid 1:2 htb rate 12800kbps # 100 mbit
tc class add dev br0 parent 1:1 classid 1:3 htb rate 1000kbps ceil 1000kbps

tc class add dev br0 parent 1:3 classid 1:4 htb rate  124kbps ceil 900kbps
tc class add dev br0 parent 1:3 classid 1:5 htb rate  60kbps ceil 70kbps
tc class add dev br0 parent 1:3 classid 1:6 htb rate  60kbps ceil 70kbps

tc filter add dev br0 protocol ip parent 1:0 handle 101 fw flowid 1:2

tc filter add dev br0 protocol ip parent 1:0 handle 102 fw flowid 1:4
tc filter add dev br0 protocol ip parent 1:0 handle 103 fw flowid 1:5
tc filter add dev br0 protocol ip parent 1:0 handle 104 fw flowid 1:6

tc qdisc add dev br0 parent 1:2 handle 2:0 sfq perturb 10
tc qdisc add dev br0 parent 1:4 handle 3:0 sfq perturb 10
tc qdisc add dev br0 parent 1:5 handle 4:0 sfq perturb 10
tc qdisc add dev br0 parent 1:6 handle 5:0 sfq perturb 10

• download patch-o-matic-ng snapshot
• unpack
• run
  KERNEL_DIR=/usr/src/linux IPTABLES_DIR=/usr/src/iptables-1.3.6.0debian1/iptables ./runme --download
• select patches:
  %KERNEL_DIR=/usr/src/linux IPTABLES_DIR=/usr/src/iptables-1.3.6.0debian1/iptables ./runme extra * albo:\\ %KERNEL_DIR=/usr/src/linux IPTABLES_DIR=/usr/src/iptables-1.3.6.0debian1/iptables ./runme ip2p [itp]
• cd /usr/src/linux
• make oldconfig [enable patches]
• recompile kernel and modules
• compile and install new iptables

/bin/echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
/bin/echo 0 > /proc/sys/net/ipv4/conf/all/log_martians
/bin/echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
/bin/echo 0 > /proc/sys/net/ipv4/conf/all/accept_source_route 
/bin/echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
/bin/echo 1 > /proc/sys/net/ipv4/tcp_syncookies
#SMURF protection
/bin/echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
/bin/echo 1 > /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses
/bin/echo 1 > /proc/sys/net/ipv4/tcp_timestamps
/bin/echo 1 > /proc/sys/net/ipv4/ip_forward
# TCP sessions limits:
# /bin/echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
# /bin/echo 2400 > /proc/sys/net/ipv4/tcp_keepalive_time
# /bin/echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
# /bin/echo 0 > /proc/sys/net/ipv4/tcp_sack
# /bin/echo 65000 > /proc/sys/net/ipv4/ip_conntrack_max
# /bin/echo 20 > /proc/sys/net/ipv4/ipfrag_time
# /bin/echo 1280 > /proc/sys/net/ipv4/tcp_max_syn_backlog
# /bin/echo 262140 > /proc/sys/net/ipv4/route/max_size

QoS Linux with HFSC

QoS Linux HTB

HowTo / EnableTrafficShaping

Automatthias HFSC Script

www.mastershaper.org

ip link set imq0 up
ip link set imq1 up

Qeueing disciplines should be assigned to imq0, imq1 etc devices (instead of physical devices)

Routing packets into IMQ devices:

For incoming packets:

iptables -t mangle -A PREROUTING [conditions] -j IMQ --todev 0
iptables -t mangle -A PREROUTING [conditions] -j IMQ --todev 1
iptables -t mangle -A PREROUTING [conditions] -j IMQ --todev 2

For outgoing packets:

iptables -t mangle -A POSTROUTING [conditions] -j IMQ --todev 0
iptables -t mangle -A POSTROUTING [conditions] -j IMQ --todev 1
iptables -t mangle -A POSTROUTING [conditions] -j IMQ --todev 2

        +------------+                +---------+               +-------------+
Packet -| PREROUTING |--- routing-----| FORWARD |-------+-------| POSTROUTING |- Packets
input   +------------+    decision    +-­-------+       |       +-------------+    out
                             |                          |
                        +-------+                    +--------+   
                        | INPUT |---- Local process -| OUTPUT |
                        +-------+                    +--------+

• iptables packet flow diagram 1
• iptables packet flow diagram 2

1. iptables PREROUTING
   • conntrack
   • mangle
   • imq
   • nat
2. ingress qos
3. routing decision: local machine or forwarding
   1. case1: local machine
      • iptables INPUT
        • mangle
        • filter
      • some local process
      • iptables OUTPUT
        • conntrack
        • mangle
        • nat
        • filter
   2. case2: forwarding to other host
      • iptables FORWARD
        • mangle
        • filter
4. iptables POSTROUTING
   • mangle
   • nat
   • imq
5. QoS egress