mjhjmtu

I'm using a Linux based router appliance (named Zeroshell), but it should be a question related to generic Linux routing.

The router computer has 4 NICs, named ETH0 to ETH3.

• ETH0 is on the actual LAN (subnet 192.168.241.0/24) IP 192.168.241.254


• ETH1 is on a WAN connection router (subnet 192.168.1.0/24) IP 192.168.1.1, GW 192.168.1.254


• ETH2 is on a another WAN connection router (subnet 192.168.2.0/24) IP 192.168.2.1, GW 192.168.2.254


• ETH3 is on another LAN dedicated for guests (subnet 192.168.230.0/24) IP 192.168.230.254

The default gateway on the router is set to 192.168.2.254 so all outgoing traffic uses the second WAN connection (optic fiber), and NAT is enabled on both ETH1 and ETH2.

On first WAN router, 192.168.1.1 is set as the DMZ.
On second WAN router, 192.168.2.1 is set as the DMZ.

I've set some port forwarding on port 80 on both ETH1 and ETH2 to a computer lying in the ETH0 subnet.

When connecting to the second WAN's public IP with a browser, I get the website hosted on the internal computer.

When connecting to the first WAN's public IP with a browser, connection stays stuck.

I'm pretty sure this has to deal with the default gateway set to the second WAN's router making all traffic go to him, even if it originated from the first WAN router.

So my question is : HOW should I configure the routing tables on my router so that it can handle incoming connections from both WANs, forward them to the relevant LAN computer and route answers to the proper WAN ?

EDIT:

Adding routing tables from the Webserver :

root@webserver:/# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
 valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
 link/ether 00:15:5d:f1:03:10 brd ff:ff:ff:ff:ff:ff
 inet 192.168.241.23/24 brd 192.168.241.255 scope global eth0
 valid_lft forever preferred_lft forever
root@webserver:/# ip route show
default via 192.168.241.254 dev eth0
10.8.0.0/24 via 192.168.241.21 dev eth0
192.168.240.0/24 via 192.168.241.21 dev eth0
192.168.241.0/24 dev eth0 proto kernel scope link src 192.168.241.23

Adding routing tables from the Router :

root@rtr ~> ip addr show
1: lo: <LOOPBACK,UP,10000> mtu 65536 qdisc noqueue
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
 valid_lft forever preferred_lft forever
 inet6 ::1/128 scope host
 valid_lft forever preferred_lft forever
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop
 link/sit 0.0.0.0 brd 0.0.0.0
3: ETH00: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc mq qlen 1000
 link/ether 00:15:5d:f1:05:08 brd ff:ff:ff:ff:ff:ff
 inet 192.168.230.254/24 brd 192.168.230.255 scope global ETH00:00
 valid_lft forever preferred_lft forever
 inet6 fe80::215:5dff:fef1:508/64 scope link
 valid_lft forever preferred_lft forever
4: ETH01: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc mq qlen 1000
 link/ether 00:15:5d:f1:05:09 brd ff:ff:ff:ff:ff:ff
 inet 192.168.2.1/24 brd 192.168.2.255 scope global ETH01:00
 valid_lft forever preferred_lft forever
 inet6 2a01:e35:2e74:9560:215:5dff:fef1:509/64 scope global dynamic
 valid_lft 86156sec preferred_lft 86156sec
 inet6 fe80::215:5dff:fef1:509/64 scope link
 valid_lft forever preferred_lft forever
5: ETH02: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc mq qlen 1000
 link/ether 00:15:5d:f1:05:0b brd ff:ff:ff:ff:ff:ff
 inet 192.168.1.1/24 brd 192.168.1.255 scope global ETH02:00
 valid_lft forever preferred_lft forever
 inet6 fe80::215:5dff:fef1:50b/64 scope link
 valid_lft forever preferred_lft forever
6: ETH03: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc mq qlen 1000
 link/ether 00:15:5d:f1:05:0c brd ff:ff:ff:ff:ff:ff
 inet 192.168.241.254/24 brd 192.168.241.255 scope global ETH03:00
 valid_lft forever preferred_lft forever
 inet6 fe80::215:5dff:fef1:50c/64 scope link
 valid_lft forever preferred_lft forever
7: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noqueue
 link/ether 9e:3d:6a:0e:65:39 brd ff:ff:ff:ff:ff:ff
 inet 192.168.141.142/24 brd 192.168.141.255 scope global dummy0
 valid_lft forever preferred_lft forever
8: dummy1: <BROADCAST,NOARP,UP,10000> mtu 1500 qdisc noqueue
 link/ether ee:6e:6f:33:32:34 brd ff:ff:ff:ff:ff:ff
 inet 192.168.142.142/32 brd 192.168.142.255 scope global dummy1
 valid_lft forever preferred_lft forever
 inet6 fe80::ec6e:6fff:fe33:3234/64 scope link
 valid_lft forever preferred_lft forever
9: DEFAULTBR: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
 link/ether 0a:61:ef:f2:09:80 brd ff:ff:ff:ff:ff:ff
10: VPN99: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 10 0
 link/ether 1a:e8:0e:ee:78:aa brd ff:ff:ff:ff:ff:ff
 inet 192.168.250.254/24 brd 192.168.250.255 scope global VPN99:00
 valid_lft forever preferred_lft forever
11: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop
 link/ether 8e:65:6c:3d:76:e5 brd ff:ff:ff:ff:ff:ff
12: bond1: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop
 link/ether 1e:34:34:54:8d:48 brd ff:ff:ff:ff:ff:ff
13: bond2: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop
 link/ether 5a:bc:4c:86:83:dc brd ff:ff:ff:ff:ff:ff
14: bond3: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop
 link/ether 6e:81:53:3e:0a:ff brd ff:ff:ff:ff:ff:ff
15: bond4: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop
 link/ether 6a:35:c8:45:d1:ff brd ff:ff:ff:ff:ff:ff
16: bond5: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop
 link/ether ca:5d:10:21:02:30 brd ff:ff:ff:ff:ff:ff
17: bond6: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop
 link/ether 82:60:85:97:d4:90 brd ff:ff:ff:ff:ff:ff
18: bond7: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop
 link/ether b6:fc:c9:a5:06:73 brd ff:ff:ff:ff:ff:ff
19: bond8: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop
 link/ether ce:75:5d:e5:7d:69 brd ff:ff:ff:ff:ff:ff
20: bond9: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop
 link/ether 2e:ef:1e:89:26:1b brd ff:ff:ff:ff:ff:ff
root@rtr ~> ip route show
default via 192.168.1.254 dev ETH02
192.168.1.0/24 dev ETH02 proto kernel scope link src 192.168.1.1
192.168.2.0/24 dev ETH01 proto kernel scope link src 192.168.2.1
192.168.230.0/24 dev ETH00 proto kernel scope link src 192.168.230.254
192.168.240.0/24 via 192.168.241.21 dev ETH03
192.168.241.0/24 dev ETH03 proto kernel scope link src 192.168.241.254
192.168.250.0/24 dev VPN99 proto kernel scope link src 192.168.250.254

The problem with such a dual-homed setup is that the return packets originating from the webserver on the ETH0 network follow the default gateway, which is only correct for connections coming in through that interface. I've run into this myself.

The solution I use is to add an extra IP address to the webserver on the ETH0 network (I'm assuming 192.168.241.24), and use that as the DNAT target for connections coming in through the second WAN interface. Then add a routing rule that the second IP address gets routed out through the second WAN interface.

You need to understand a little bit about Linux policy based routing. Do ip rule show:

0: from all lookup local 
32766: from all lookup main 
32767: from all lookup default 

When you do ip route show you show the "main" table as default. You can show one of the other tables by adding table $name, e.g.:

$ ip route show table local
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
broadcast 192.168.0.0 dev eth0 proto kernel scope link src 192.168.1.27
local 192.168.1.27 dev eth0 proto kernel scope host src 192.168.1.27
broadcast 192.168.1.255 dev eth0 proto kernel scope link src 192.168.1.27

You can add your own tables; edit /etc/iproute2/rt_tables and add two lines:

11 WAN1
12 WAN2

Now you can add the default routes over ETH01 and ETH02 to the respective tables:

# ip route add default via 192.168.2.254 table WAN1
# ip route add default via 192.168.1.254 table WAN2

(You first describe ETH01 as having 192.168.1.254 as gateway and ETH02 and having 192.168.2.254 but then your ip route show output disagrees, so I'm going with the latter... aside from the ETH1 / ETH01 difference.)

Now you need to add a rule to use the WAN1 table for traffic coming in from the second webserver IP address:

# ip rule add from 192.168.241.24 lookup WAN1 prio 1000

Now, when traffic comes in from ETH01 and gets sent via DNAT to the second IP address of the webserver, the webserver will return packets from that address and the rule will match that address and send return traffic via ETH01.

You don't really need the WAN2 table in this scenario, however if the router system itself needs to be accessible from both WAN interfaces, or if you want to be able to choose what WAN interface for whatever then it's useful to have it.