mjhjmtu

I am not an R&S professional, so much of that realm of computing escapes me.
Could you lend your expertise in helping understand how to go about
configuring the separation of network traffic on two network interfaces?
I've seen the other posts, but I am still stumped.

As I understand thus far, static routes are used for network traffic that
is not designed to use a default gateway. The default gateway is used for
all traffic which is not destined for the local network and for which no
preferred route has been specified in a routing table.

The scenario is as follows.

• Each computer in the network has two network cards.


• The production interface for each is eth0 (GW = 10.10.10.1).


• The management interface for each is eth1 (GW = 192.168.100.1).


• Production and Management traffic should be totally separated.

I have posted, below, what things I have tried with Debian Wheezy.
And, my problem is that, although I have hosts set up in such a way that
they do communicate on both interfaces, individual hosts seem to "hear"
traffic on the wrong interface. For example:

Host 140

eth0 Link encap:Ethernet HWaddr 08:00:27:d1:b6:8f
 inet addr:10.10.10.140 Bcast:10.10.10.255 Mask:255.255.255.0
 inet6 addr: fe80::a00:27ff:fed1:b68f/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:1341 errors:0 dropped:0 overruns:0 frame:0
 TX packets:2530 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:641481 (626.4 KiB) TX bytes:241124 (235.4 KiB)

eth1 Link encap:Ethernet HWaddr 08:00:27:ad:14:b6
 inet addr:192.168.100.140 Bcast:192.168.100.255 Mask:255.255.255.0
 inet6 addr: fe80::a00:27ff:fead:14b6/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:7220 errors:0 dropped:0 overruns:0 frame:0
 TX packets:5257 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:602485 (588.3 KiB) TX bytes:1022906 (998.9 KiB)

From host 140, I execute this command: tcpdump -i eth0. In a separate
session on host 140, I execute ping 192.168.100.50.

19:17:29.301565 IP 192.168.100.140 > 192.168.100.50: ICMP echo request, id 1400, seq 10, length 64
19:17:30.301561 IP 192.168.100.140 > 192.168.100.50: ICMP echo request, id 1400, seq 11, length 64
19:17:31.301570 IP 192.168.100.140 > 192.168.100.50: ICMP echo request, id 1400, seq 12, length 64
19:17:32.301580 IP 192.168.100.140 > 192.168.100.50: ICMP echo request, id 1400, seq 13, length 64

Why do I see the above output on eth0? I think I should only see traffic for 10.10.10.140.
I also see this on eth1, as expected:

19:18:47.805408 IP 192.168.100.50 > 192.168.100.140: ICMP echo request, id 1605, seq 247, length 64

If I ping from Host 50 (same ifconfig results - just a different last quad),
then eth0 is silent, and I see the ICMP echos on eth1, as expected.

I would like to understand how to configure each interface to handle only
the traffic for which it is responsible in two major Linux varieties.
I think I am almost there, but I am missing something I just can't seem to find.

• Debian Wheezy (7.x) or Debian Jessie (8.x)


• Enterprise Linux (6.x) (RedHat/CentOS/Scientific/Oracle).

I know that a solution for Debian should be good for both Wheezy and Jessie,
and that a solution for an EL should be the same for all the EL 6.x versions.
I would like to avoid using an RC script to execute commands, opting instead
for using the configuration files.

In Debian the relevant configuration files that I know about are:

• /etc/iproute2/rt_tables


• /etc/network/interfaces

In EL 6.x, the relevant configuration files that I know about are:

• /etc/iproute2/rt_tables


• /etc/sysconfig/network


• /etc/sysconfig/network-scripts/ifcfg-eth0


• /etc/sysconfig/network-scripts/ifcfg-eth1


• /etc/sysconfig/network-scripts/route-eth0


• /etc/sysconfig/network-scripts/route-eth1


• /etc/sysconfig/network-scripts/rule-eth0


• /etc/sysconfig/network-scripts/rule-eth1

My Debian 8 "Jessie" /etc/iproute2/rt_tables file:

#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
1 mgmt

My Debian 8 "Jessie" /etc/network/interfaces file:

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# Production interface
auto eth0
allow-hotplug eth0
iface eth0 inet static
 address 10.10.10.140
 netmask 255.255.255.0
 gateway 10.10.10.1

# Management interface
auto eth1
allow-hotplug eth1
iface eth1 inet static
 address 192.168.100.140
 netmask 255.255.255.0
 post-up ip route add 192.168.100.0/24 dev eth1 src 192.168.100.140 table mgmt
 post-up ip route add default via 192.168.100.1 dev eth1 table mgmt
 post-up ip rule add from 192.168.100.140/32 table mgmt
 post-up ip rule add to 192.168.100.140/32 table mgmt

I think netstat -anr might illustrate the problem:

Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.10.10.1 0.0.0.0 UG 0 0 0 eth0
10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.100.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.100.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1

I'd love to know more about this topic to refine the configuration to be the best that it can be, but here's what I have so far. Even without enabling ARP filtering on all network interfaces (net.ipv4.conf.all.arp_filter = 0),
traffic seems to be completely separated in this configuration.

The file, /etc/iproute2/rt_tables, is the same in EL 6.x and DEB 7/8, at least.

#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
200 mgmt

The file, /etc/network/interfaces in DEB 7/8, at least:

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
 iface lo inet loopback

# The production network interface
auto eth0
allow-hotplug eth0
iface eth0 inet static
 address 10.10.10.140
 netmask 255.255.255.0
 gateway 10.10.10.1

# The management network interface
auto eth1
allow-hotplug eth1
iface eth1 inet static
 address 192.168.100.140
 netmask 255.255.255.0
 post-up ip route add 192.168.100.0/24 dev eth1 src 192.168.100.140 table mgmt
 post-up ip route add default via 192.168.100.1 dev eth1 table mgmt
 post-up ip rule add from 192.168.100.140/32 table mgmt
 post-up ip rule add to 192.168.100.140/32 table mgmt

The result of ip route show on Debian:

default via 10.10.10.1 dev eth0
10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.140
192.168.100.0/24 dev eth1 proto kernel scope link src 192.168.100.140

The EL 6.x /etc/sysconfig/network file:

NETWORKING=yes
HOSTNAME=localhost.localdomain
GATEWAY=10.10.10.1

THE EL 6.x /etc/sysconfig/network-scripts/ifcfg-eth0 file, without "HWADDR" and "UUID":

DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTOCOL=none
IPADDR=10.10.10.140
NETMASK=255.255.255.0
NETWORK=10.10.10.0
BROADCAST=10.10.10.255

THE EL 6.x /etc/sysconfig/network-scripts/ifcfg-eth1 file, without "HWADDR" and "UUID":

DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTOCOL=none
IPADDR=192.168.100.140
NETMASK=255.255.255.0
NETWORK=192.168.100.0
BROADCAST=192.168.100.255

The EL 6.x /etc/sysconfig/network-scripts/route-eth1 file:

192.168.100.0/24 dev eth1 table mgmt
default via 192.168.100.1 dev eth1 table mgmt

The EL 6.x /etc/sysconfig/network-scripts/rule-eth1 file:

from 192.168.100.0/24 lookup mgmt

The result of ip route show on EL 6.x:

192.168.100.0/24 dev eth1 proto kernel scope link src 192.168.100.160
10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.160
169.254.0.0/16 dev eth0 scope link metric 1002
169.254.0.0/16 dev eth1 scope link metric 1003
default via 10.10.10.1 dev eth0

I haven't read throughly all your post (sorry, can't really spend the time right now), but I believe it may be related to the way Linux implements the IP host model:

... The IPv4 implementation in Linux defaults to the weak host model. ...

From that same page:

... If the IP stack is implemented with a weak host model, it accepts any locally destined packet regardless of the network interface on which the packet was received. ...

That is, in Linux, by default, the IP addresses "belong to the host", not strictly "to the interface". You can change that behavior via the arp_filter, rp_filter, arp_announce, arp_ignore sysctls (got from LVS: The ARP Problem, seen here). Also, see ip-sysctl.txt.

Hope this helps.