Two ISPs and multipath gateway configuration
Clash Royale CLAN TAG#URR8PPP
up vote
1
down vote
favorite
I have two different ISPs. I want to set some kind of load balancing setup that will distribute packets to those providers. I know this can be done using different routing tables, but I wanted to use something called "multipath gateway".
I've configured both interfaces in the /etc/network/interfaces file. Both of the connections work separately. I replaced the default gateways with the one below:
# ip route add default
nexthop via 192.168.1.1 dev bond0 weight 1
nexthop via 10.143.105.17 dev wwan0 weight 1
I added masquerade targets in iptables on both of the interfaces:
iptables -t nat -A POSTROUTING -o wwan0 -j MASQUERADE
iptables -t nat -A POSTROUTING -o bond0 -j MASQUERADE
Also I enabled (partially) reverse path filtering via sysctl:
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
This setup works. Packets (connections) are sent via both interfaces. There's just one problem I don't get.
When I want to check my IP address using the following commands:
$ curl text.whatisyourip.org
$ curl eko.one.pl/host.php
The IP address is different in both cases, which means that the mechanism works well. Also I can see it working in wireshark. But when I'm trying to send, for instance, multiple requests to the first of the domains above, I always get the same IP address in response. So it looks like that packets that are destined to the specific IP address always go through the same interface. I'm just wondering why. Is there any mechanism that remembers the destination IP addresses of the previous requests and makes the next requests to the same addresses to go through the same interface?
debian routing
asked May 20 '16 at 10:45
Mikhail Morfikov
4,330114267
add a comment |Â
up vote
1
down vote
favorite
I have two different ISPs. I want to set some kind of load balancing setup that will distribute packets to those providers. I know this can be done using different routing tables, but I wanted to use something called "multipath gateway".
I've configured both interfaces in the /etc/network/interfaces file. Both of the connections work separately. I replaced the default gateways with the one below:
# ip route add default
nexthop via 192.168.1.1 dev bond0 weight 1
nexthop via 10.143.105.17 dev wwan0 weight 1
I added masquerade targets in iptables on both of the interfaces:
iptables -t nat -A POSTROUTING -o wwan0 -j MASQUERADE
iptables -t nat -A POSTROUTING -o bond0 -j MASQUERADE
Also I enabled (partially) reverse path filtering via sysctl:
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
This setup works. Packets (connections) are sent via both interfaces. There's just one problem I don't get.
When I want to check my IP address using the following commands:
$ curl text.whatisyourip.org
$ curl eko.one.pl/host.php
The IP address is different in both cases, which means that the mechanism works well. Also I can see it working in wireshark. But when I'm trying to send, for instance, multiple requests to the first of the domains above, I always get the same IP address in response. So it looks like that packets that are destined to the specific IP address always go through the same interface. I'm just wondering why. Is there any mechanism that remembers the destination IP addresses of the previous requests and makes the next requests to the same addresses to go through the same interface?
debian routing
asked May 20 '16 at 10:45
Mikhail Morfikov
4,330114267
what you mean "multiple requests"? IP-packets? What exactly do you do to produce the traffic?
– Serge
May 20 '16 at 11:50
Your kernel may be caching routes, if it's an older version. Doesip route show cachedisplay anything? BTW, be happy, quite a few websites assign a session to an IP address, so if after logging in you connect froma different source IP you need to login again. And things stored in the session aren't always accessible, etc. (I found this out when using a load-balanced proxy setup that connected to the outside with multiple source IPs).
– wurtel
May 20 '16 at 11:55
1
There's nothing inip route show cache. And by "multiple requests" I mean that I send the commands 10-20 times in a row, just by typing them into a terminal.
– Mikhail Morfikov
May 20 '16 at 11:58
add a comment |Â
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I have two different ISPs. I want to set some kind of load balancing setup that will distribute packets to those providers. I know this can be done using different routing tables, but I wanted to use something called "multipath gateway".
I've configured both interfaces in the /etc/network/interfaces file. Both of the connections work separately. I replaced the default gateways with the one below:
# ip route add default
nexthop via 192.168.1.1 dev bond0 weight 1
nexthop via 10.143.105.17 dev wwan0 weight 1
I added masquerade targets in iptables on both of the interfaces:
iptables -t nat -A POSTROUTING -o wwan0 -j MASQUERADE
iptables -t nat -A POSTROUTING -o bond0 -j MASQUERADE
Also I enabled (partially) reverse path filtering via sysctl:
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
This setup works. Packets (connections) are sent via both interfaces. There's just one problem I don't get.
When I want to check my IP address using the following commands:
$ curl text.whatisyourip.org
$ curl eko.one.pl/host.php
The IP address is different in both cases, which means that the mechanism works well. Also I can see it working in wireshark. But when I'm trying to send, for instance, multiple requests to the first of the domains above, I always get the same IP address in response. So it looks like that packets that are destined to the specific IP address always go through the same interface. I'm just wondering why. Is there any mechanism that remembers the destination IP addresses of the previous requests and makes the next requests to the same addresses to go through the same interface?
debian routing
asked May 20 '16 at 10:45
Mikhail Morfikov
4,330114267
I have two different ISPs. I want to set some kind of load balancing setup that will distribute packets to those providers. I know this can be done using different routing tables, but I wanted to use something called "multipath gateway".
I've configured both interfaces in the /etc/network/interfaces file. Both of the connections work separately. I replaced the default gateways with the one below:
# ip route add default
nexthop via 192.168.1.1 dev bond0 weight 1
nexthop via 10.143.105.17 dev wwan0 weight 1
I added masquerade targets in iptables on both of the interfaces:
iptables -t nat -A POSTROUTING -o wwan0 -j MASQUERADE
iptables -t nat -A POSTROUTING -o bond0 -j MASQUERADE
Also I enabled (partially) reverse path filtering via sysctl:
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
This setup works. Packets (connections) are sent via both interfaces. There's just one problem I don't get.
When I want to check my IP address using the following commands:
$ curl text.whatisyourip.org
$ curl eko.one.pl/host.php
The IP address is different in both cases, which means that the mechanism works well. Also I can see it working in wireshark. But when I'm trying to send, for instance, multiple requests to the first of the domains above, I always get the same IP address in response. So it looks like that packets that are destined to the specific IP address always go through the same interface. I'm just wondering why. Is there any mechanism that remembers the destination IP addresses of the previous requests and makes the next requests to the same addresses to go through the same interface?
debian routing
debian routing
asked May 20 '16 at 10:45
Mikhail Morfikov
4,330114267
asked May 20 '16 at 10:45
Mikhail Morfikov
4,330114267
asked May 20 '16 at 10:45
Mikhail Morfikov
4,330114267
asked May 20 '16 at 10:45
Mikhail Morfikov
4,330114267
asked May 20 '16 at 10:45
Mikhail Morfikov
4,330114267
4,330114267
what you mean "multiple requests"? IP-packets? What exactly do you do to produce the traffic?
– Serge
May 20 '16 at 11:50
Your kernel may be caching routes, if it's an older version. Doesip route show cachedisplay anything? BTW, be happy, quite a few websites assign a session to an IP address, so if after logging in you connect froma different source IP you need to login again. And things stored in the session aren't always accessible, etc. (I found this out when using a load-balanced proxy setup that connected to the outside with multiple source IPs).
– wurtel
May 20 '16 at 11:55
1
There's nothing inip route show cache. And by "multiple requests" I mean that I send the commands 10-20 times in a row, just by typing them into a terminal.
– Mikhail Morfikov
May 20 '16 at 11:58
add a comment |Â
what you mean "multiple requests"? IP-packets? What exactly do you do to produce the traffic?
– Serge
May 20 '16 at 11:50
Your kernel may be caching routes, if it's an older version. Doesip route show cachedisplay anything? BTW, be happy, quite a few websites assign a session to an IP address, so if after logging in you connect froma different source IP you need to login again. And things stored in the session aren't always accessible, etc. (I found this out when using a load-balanced proxy setup that connected to the outside with multiple source IPs).
– wurtel
May 20 '16 at 11:55
1
There's nothing inip route show cache. And by "multiple requests" I mean that I send the commands 10-20 times in a row, just by typing them into a terminal.
– Mikhail Morfikov
May 20 '16 at 11:58
what you mean "multiple requests"? IP-packets? What exactly do you do to produce the traffic?
– Serge
May 20 '16 at 11:50
what you mean "multiple requests"? IP-packets? What exactly do you do to produce the traffic?
– Serge
May 20 '16 at 11:50
Your kernel may be caching routes, if it's an older version. Does
ip route show cache display anything? BTW, be happy, quite a few websites assign a session to an IP address, so if after logging in you connect froma different source IP you need to login again. And things stored in the session aren't always accessible, etc. (I found this out when using a load-balanced proxy setup that connected to the outside with multiple source IPs).– wurtel
May 20 '16 at 11:55
Your kernel may be caching routes, if it's an older version. Does
ip route show cache display anything? BTW, be happy, quite a few websites assign a session to an IP address, so if after logging in you connect froma different source IP you need to login again. And things stored in the session aren't always accessible, etc. (I found this out when using a load-balanced proxy setup that connected to the outside with multiple source IPs).– wurtel
May 20 '16 at 11:55
1
1
There's nothing in
ip route show cache . And by "multiple requests" I mean that I send the commands 10-20 times in a row, just by typing them into a terminal.– Mikhail Morfikov
May 20 '16 at 11:58
There's nothing in
ip route show cache . And by "multiple requests" I mean that I send the commands 10-20 times in a row, just by typing them into a terminal.– Mikhail Morfikov
May 20 '16 at 11:58
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
3
down vote
accepted
I've managed to solve the problem. In this link you can read the following:
IPv4: Hash-based multipath routing. When the routing cache was removed
in 3.6, the IPv4 multipath algorithm changed from more or less being
destination-based into being quasi-random per-packet scheduling. This
increased the risk of out-of-order packets and made it impossible to
use multipath together with anycast services. In this release, the
multipath routing implementation is replaced with a flow-based load
balancing based on a hash over the source and destination addresses
merge commit
So even though the cache was removed in kernel 3.6, the requests are still being cached. Now the source and the destination addresses matter. So that's why the packets go always through the same interface.
edited Aug 15 at 15:13
AdamKalisz
19315
answered May 20 '16 at 16:57
Mikhail Morfikov
4,330114267
You should also mention why a hash over source and destination is needed: Normal internet protocols (TCP/UDP) are not "multihomed", and your two ISPs will give you different public addresses. The protocols can't deal with that, so packets randomly distributed to the "other" ISP would just count as packet loss. You need a deterministic choiced based on the connection.
– dirkt
Aug 15 at 17:40
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
3
down vote
accepted
I've managed to solve the problem. In this link you can read the following:
IPv4: Hash-based multipath routing. When the routing cache was removed
in 3.6, the IPv4 multipath algorithm changed from more or less being
destination-based into being quasi-random per-packet scheduling. This
increased the risk of out-of-order packets and made it impossible to
use multipath together with anycast services. In this release, the
multipath routing implementation is replaced with a flow-based load
balancing based on a hash over the source and destination addresses
merge commit
So even though the cache was removed in kernel 3.6, the requests are still being cached. Now the source and the destination addresses matter. So that's why the packets go always through the same interface.
edited Aug 15 at 15:13
AdamKalisz
19315
answered May 20 '16 at 16:57
Mikhail Morfikov
4,330114267
You should also mention why a hash over source and destination is needed: Normal internet protocols (TCP/UDP) are not "multihomed", and your two ISPs will give you different public addresses. The protocols can't deal with that, so packets randomly distributed to the "other" ISP would just count as packet loss. You need a deterministic choiced based on the connection.
– dirkt
Aug 15 at 17:40
add a comment |Â
up vote
3
down vote
accepted
I've managed to solve the problem. In this link you can read the following:
IPv4: Hash-based multipath routing. When the routing cache was removed
in 3.6, the IPv4 multipath algorithm changed from more or less being
destination-based into being quasi-random per-packet scheduling. This
increased the risk of out-of-order packets and made it impossible to
use multipath together with anycast services. In this release, the
multipath routing implementation is replaced with a flow-based load
balancing based on a hash over the source and destination addresses
merge commit
So even though the cache was removed in kernel 3.6, the requests are still being cached. Now the source and the destination addresses matter. So that's why the packets go always through the same interface.
edited Aug 15 at 15:13
AdamKalisz
19315
answered May 20 '16 at 16:57
Mikhail Morfikov
4,330114267
You should also mention why a hash over source and destination is needed: Normal internet protocols (TCP/UDP) are not "multihomed", and your two ISPs will give you different public addresses. The protocols can't deal with that, so packets randomly distributed to the "other" ISP would just count as packet loss. You need a deterministic choiced based on the connection.
– dirkt
Aug 15 at 17:40
add a comment |Â
up vote
3
down vote
accepted
up vote
3
down vote
accepted
I've managed to solve the problem. In this link you can read the following:
IPv4: Hash-based multipath routing. When the routing cache was removed
in 3.6, the IPv4 multipath algorithm changed from more or less being
destination-based into being quasi-random per-packet scheduling. This
increased the risk of out-of-order packets and made it impossible to
use multipath together with anycast services. In this release, the
multipath routing implementation is replaced with a flow-based load
balancing based on a hash over the source and destination addresses
merge commit
So even though the cache was removed in kernel 3.6, the requests are still being cached. Now the source and the destination addresses matter. So that's why the packets go always through the same interface.
edited Aug 15 at 15:13
AdamKalisz
19315
answered May 20 '16 at 16:57
Mikhail Morfikov
4,330114267
I've managed to solve the problem. In this link you can read the following:
IPv4: Hash-based multipath routing. When the routing cache was removed
in 3.6, the IPv4 multipath algorithm changed from more or less being
destination-based into being quasi-random per-packet scheduling. This
increased the risk of out-of-order packets and made it impossible to
use multipath together with anycast services. In this release, the
multipath routing implementation is replaced with a flow-based load
balancing based on a hash over the source and destination addresses
merge commit
So even though the cache was removed in kernel 3.6, the requests are still being cached. Now the source and the destination addresses matter. So that's why the packets go always through the same interface.
edited Aug 15 at 15:13
AdamKalisz
19315
answered May 20 '16 at 16:57
Mikhail Morfikov
4,330114267
edited Aug 15 at 15:13
AdamKalisz
19315
edited Aug 15 at 15:13
AdamKalisz
19315
edited Aug 15 at 15:13
AdamKalisz
19315
19315
answered May 20 '16 at 16:57
Mikhail Morfikov
4,330114267
answered May 20 '16 at 16:57
Mikhail Morfikov
4,330114267
answered May 20 '16 at 16:57
Mikhail Morfikov
4,330114267
4,330114267
You should also mention why a hash over source and destination is needed: Normal internet protocols (TCP/UDP) are not "multihomed", and your two ISPs will give you different public addresses. The protocols can't deal with that, so packets randomly distributed to the "other" ISP would just count as packet loss. You need a deterministic choiced based on the connection.
– dirkt
Aug 15 at 17:40
add a comment |Â
You should also mention why a hash over source and destination is needed: Normal internet protocols (TCP/UDP) are not "multihomed", and your two ISPs will give you different public addresses. The protocols can't deal with that, so packets randomly distributed to the "other" ISP would just count as packet loss. You need a deterministic choiced based on the connection.
– dirkt
Aug 15 at 17:40
You should also mention why a hash over source and destination is needed: Normal internet protocols (TCP/UDP) are not "multihomed", and your two ISPs will give you different public addresses. The protocols can't deal with that, so packets randomly distributed to the "other" ISP would just count as packet loss. You need a deterministic choiced based on the connection.
– dirkt
Aug 15 at 17:40
You should also mention why a hash over source and destination is needed: Normal internet protocols (TCP/UDP) are not "multihomed", and your two ISPs will give you different public addresses. The protocols can't deal with that, so packets randomly distributed to the "other" ISP would just count as packet loss. You need a deterministic choiced based on the connection.
– dirkt
Aug 15 at 17:40
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f284416%2ftwo-isps-and-multipath-gateway-configuration%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
what you mean "multiple requests"? IP-packets? What exactly do you do to produce the traffic?
– Serge
May 20 '16 at 11:50
Your kernel may be caching routes, if it's an older version. Does
ip route show cachedisplay anything? BTW, be happy, quite a few websites assign a session to an IP address, so if after logging in you connect froma different source IP you need to login again. And things stored in the session aren't always accessible, etc. (I found this out when using a load-balanced proxy setup that connected to the outside with multiple source IPs).– wurtel
May 20 '16 at 11:55
1
There's nothing in
ip route show cache. And by "multiple requests" I mean that I send the commands 10-20 times in a row, just by typing them into a terminal.– Mikhail Morfikov
May 20 '16 at 11:58