mjhjmtu

Network Interface Configuration

192.168.0.0 Out 1(enp10s0f0) - Local 01(enp10s0f1) 192.168.10.0
 - Local 02(enp10s0f2)
 - Local 03(enp10s0f3)

59.27.1.128 Out 2(enp11s0f0) - Local 04(enp11s0f1) 192.168.11.0
 - Local 05(enp11s0f2)
 - Local 06(enp11s0f3)

There are 2 lines for out-bound. Each line share its outbound connection to Locals. Out 1 => Local 01~02, Out 2 => Local 03~04

So I made a script to configure iptables as following

iptables -t nat -A POSTROUTING -o enp10s0f0 -j MASQUERADE -s 192.168.10.0/24
iptables -t nat -A POSTROUTING -o enp11s0f0 -j MASQUERADE -s 192.168.11.0/24

iptables -A FORWARD -i enp10s0f0 -o enp10s0f1 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i enp10s0f0 -o enp10s0f2 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i enp10s0f0 -o enp10s0f3 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i enp10s0f1 -o enp10s0f0 -j ACCEPT
iptables -A FORWARD -i enp10s0f2 -o enp10s0f0 -j ACCEPT
iptables -A FORWARD -i enp10s0f3 -o enp10s0f0 -j ACCEPT

iptables -t mangle -A PREROUTING -i enp10s0f1 -p tcp -m tcp --dport 80 -j TPROXY --on-ip 0.0.0.0 --on-port 8080 --tproxy-mark 0x1/0x1
iptables -t mangle -A PREROUTING -i enp10s0f2 -p tcp -m tcp --dport 80 -j TPROXY --on-ip 0.0.0.0 --on-port 8080 --tproxy-mark 0x1/0x1
iptables -t mangle -A PREROUTING -i enp10s0f3 -p tcp -m tcp --dport 80 -j TPROXY --on-ip 0.0.0.0 --on-port 8080 --tproxy-mark 0x1/0x1

iptables -t mangle -A PREROUTING -i enp10s0f0 --source 192.168.10.0/24 -j ACCEPT
iptables -t mangle -A PREROUTING -i enp10s0f0 --destination 192.168.10.0/24 -j ACCEPT

iptables -t mangle -A PREROUTING -i enp10s0f0 -p tcp -m tcp --sport 80 -j MARK --set-mark 0x1/0x1

ip rule add fwmark 0x1/0x1 table 2
ip route add local 0.0.0.0/0 dev lo table 2 
//Do this for enp11, too

But One of out-line, which is plugged later, doesn't work as expected.
(OS reconfigure the routing table after the line is plugged. So out-line first plugged only works. Second one not.)
Local-lines connected on another out-line works.

I guess it is caused by routing table... but I couldn't figure out the problem.

This is 'ip route' shows.

default via 59.27.1.129 dev enp11s0f0 proto static metric 2 
default via 192.168.0.1 dev enp10s0f0 proto static metric 100 
59.27.1.128/26 dev enp11s0f0 proto kernel scope link src 59.27.1.176 metric 2 
169.254.0.0/16 dev enp11s0f0 scope link metric 2 
192.168.0.0/24 dev enp10s0f0 proto kernel scope link src 192.168.0.22 metric 100 
192.168.10.0/24 dev enp10s0f1 proto kernel scope link src 192.168.10.251 metric 100 
192.168.11.0/24 dev enp11s0f1 proto kernel scope link src 192.168.11.251 metric 2 
222.118.178.36 via 59.27.1.129 dev enp11s0f0 proto dhcp metric 2

Thank you.

It was a problem of routing table.

Exactly the same one.
Ref.
NAT box with multiple internal and external interfaces

have to divide table and add rules for lookup based on IP.