Infiniband interface doesn't route IPoIB traffic

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
0
down vote

favorite












I have blocks of hosts that I'm provisioning using Puppet in exactly the same way, they have identical hardware (same blade chassis), and are definitely connected in all the same ways where interfaces on some are not working the same as others. These are all Infiniband interfaces, so I'm able to test them with commands like ibping and ibsysstat, which shows that they have working UVERBS/RDMA connections. For example:



master# ibsysstat 29
sysstat ping succeeded


where the node with that LID that isn't working quite right has:



node10# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.11.1250
Hardware version: 1
Node GUID: 0x...
System image GUID: 0x...
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 29
LMC: 0
SM lid: 26
Capability mask: 0x02594868
Port GUID: 0x...
Link layer: InfiniBand


but, when I just do a simple ping to the IPoIB IP address it sits there not connecting. Other commands like ibping are also definitely passing traffic, and data shows up when adding -d showing debug output. I can see the pings go out when I watch the interface using tcpdump, but nothing coming in. Meanwhile, right next to it is a host with the same everything that works just fine. The routing tables all like right to me also, and match hosts that work. On a host that doesn't work:



default via 10.10.0.1 dev em1 proto dhcp metric 100 
10.10.0.0/24 dev em1 proto kernel scope link src 10.10.0.110 metric 100
10.11.0.0/24 dev ib0 proto kernel scope link src 10.11.0.110
169.254.0.0/16 dev ib0 scope link metric 1005


and on one that does:



default via 10.10.0.1 dev em1 proto dhcp metric 100 
10.10.0.0/24 dev em1 proto kernel scope link src 10.10.0.108 metric 100
10.11.0.0/24 dev ib0 proto kernel scope link src 10.11.0.108
169.254.0.0/16 dev ib0 scope link metric 1004


The only thing different is the metric in the last route, but that shouldn't matter. Also of note, these hosts worked before they were reprovisioned. So I'm almost positive it's not hardware.



I'm at a bit of a loss now and any ideas would be appreciated.



Edit: Update with dmesg error



I found something in the output of dmesg for the interface in question that only exists on the hosts that don't work. The error



ib0: failed to modify QP to RTR: -22


unfortunately this isn't very helpful, and there's not much that comes up related in searches.



Perhaps also worth noting, the hosts in question can ping the switch IP address, and the switch can ping the hosts on their associated IPs.










share|improve this question























  • While probably not related, I would uninstall avahi. Zeroconf should have no place on enterprise servers.
    – Rui F Ribeiro
    Aug 16 at 21:00










  • It's not actually installed, the script /etc/sysconfig/network-scripts/ifup-ib from OFED adds a route in for some reason. I've removed that and no change.
    – geoffjay
    Aug 16 at 21:30










  • 169.254 is usually a sign it is installed.
    – Rui F Ribeiro
    Aug 16 at 21:45














up vote
0
down vote

favorite












I have blocks of hosts that I'm provisioning using Puppet in exactly the same way, they have identical hardware (same blade chassis), and are definitely connected in all the same ways where interfaces on some are not working the same as others. These are all Infiniband interfaces, so I'm able to test them with commands like ibping and ibsysstat, which shows that they have working UVERBS/RDMA connections. For example:



master# ibsysstat 29
sysstat ping succeeded


where the node with that LID that isn't working quite right has:



node10# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.11.1250
Hardware version: 1
Node GUID: 0x...
System image GUID: 0x...
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 29
LMC: 0
SM lid: 26
Capability mask: 0x02594868
Port GUID: 0x...
Link layer: InfiniBand


but, when I just do a simple ping to the IPoIB IP address it sits there not connecting. Other commands like ibping are also definitely passing traffic, and data shows up when adding -d showing debug output. I can see the pings go out when I watch the interface using tcpdump, but nothing coming in. Meanwhile, right next to it is a host with the same everything that works just fine. The routing tables all like right to me also, and match hosts that work. On a host that doesn't work:



default via 10.10.0.1 dev em1 proto dhcp metric 100 
10.10.0.0/24 dev em1 proto kernel scope link src 10.10.0.110 metric 100
10.11.0.0/24 dev ib0 proto kernel scope link src 10.11.0.110
169.254.0.0/16 dev ib0 scope link metric 1005


and on one that does:



default via 10.10.0.1 dev em1 proto dhcp metric 100 
10.10.0.0/24 dev em1 proto kernel scope link src 10.10.0.108 metric 100
10.11.0.0/24 dev ib0 proto kernel scope link src 10.11.0.108
169.254.0.0/16 dev ib0 scope link metric 1004


The only thing different is the metric in the last route, but that shouldn't matter. Also of note, these hosts worked before they were reprovisioned. So I'm almost positive it's not hardware.



I'm at a bit of a loss now and any ideas would be appreciated.



Edit: Update with dmesg error



I found something in the output of dmesg for the interface in question that only exists on the hosts that don't work. The error



ib0: failed to modify QP to RTR: -22


unfortunately this isn't very helpful, and there's not much that comes up related in searches.



Perhaps also worth noting, the hosts in question can ping the switch IP address, and the switch can ping the hosts on their associated IPs.










share|improve this question























  • While probably not related, I would uninstall avahi. Zeroconf should have no place on enterprise servers.
    – Rui F Ribeiro
    Aug 16 at 21:00










  • It's not actually installed, the script /etc/sysconfig/network-scripts/ifup-ib from OFED adds a route in for some reason. I've removed that and no change.
    – geoffjay
    Aug 16 at 21:30










  • 169.254 is usually a sign it is installed.
    – Rui F Ribeiro
    Aug 16 at 21:45












up vote
0
down vote

favorite









up vote
0
down vote

favorite











I have blocks of hosts that I'm provisioning using Puppet in exactly the same way, they have identical hardware (same blade chassis), and are definitely connected in all the same ways where interfaces on some are not working the same as others. These are all Infiniband interfaces, so I'm able to test them with commands like ibping and ibsysstat, which shows that they have working UVERBS/RDMA connections. For example:



master# ibsysstat 29
sysstat ping succeeded


where the node with that LID that isn't working quite right has:



node10# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.11.1250
Hardware version: 1
Node GUID: 0x...
System image GUID: 0x...
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 29
LMC: 0
SM lid: 26
Capability mask: 0x02594868
Port GUID: 0x...
Link layer: InfiniBand


but, when I just do a simple ping to the IPoIB IP address it sits there not connecting. Other commands like ibping are also definitely passing traffic, and data shows up when adding -d showing debug output. I can see the pings go out when I watch the interface using tcpdump, but nothing coming in. Meanwhile, right next to it is a host with the same everything that works just fine. The routing tables all like right to me also, and match hosts that work. On a host that doesn't work:



default via 10.10.0.1 dev em1 proto dhcp metric 100 
10.10.0.0/24 dev em1 proto kernel scope link src 10.10.0.110 metric 100
10.11.0.0/24 dev ib0 proto kernel scope link src 10.11.0.110
169.254.0.0/16 dev ib0 scope link metric 1005


and on one that does:



default via 10.10.0.1 dev em1 proto dhcp metric 100 
10.10.0.0/24 dev em1 proto kernel scope link src 10.10.0.108 metric 100
10.11.0.0/24 dev ib0 proto kernel scope link src 10.11.0.108
169.254.0.0/16 dev ib0 scope link metric 1004


The only thing different is the metric in the last route, but that shouldn't matter. Also of note, these hosts worked before they were reprovisioned. So I'm almost positive it's not hardware.



I'm at a bit of a loss now and any ideas would be appreciated.



Edit: Update with dmesg error



I found something in the output of dmesg for the interface in question that only exists on the hosts that don't work. The error



ib0: failed to modify QP to RTR: -22


unfortunately this isn't very helpful, and there's not much that comes up related in searches.



Perhaps also worth noting, the hosts in question can ping the switch IP address, and the switch can ping the hosts on their associated IPs.










share|improve this question















I have blocks of hosts that I'm provisioning using Puppet in exactly the same way, they have identical hardware (same blade chassis), and are definitely connected in all the same ways where interfaces on some are not working the same as others. These are all Infiniband interfaces, so I'm able to test them with commands like ibping and ibsysstat, which shows that they have working UVERBS/RDMA connections. For example:



master# ibsysstat 29
sysstat ping succeeded


where the node with that LID that isn't working quite right has:



node10# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.11.1250
Hardware version: 1
Node GUID: 0x...
System image GUID: 0x...
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 29
LMC: 0
SM lid: 26
Capability mask: 0x02594868
Port GUID: 0x...
Link layer: InfiniBand


but, when I just do a simple ping to the IPoIB IP address it sits there not connecting. Other commands like ibping are also definitely passing traffic, and data shows up when adding -d showing debug output. I can see the pings go out when I watch the interface using tcpdump, but nothing coming in. Meanwhile, right next to it is a host with the same everything that works just fine. The routing tables all like right to me also, and match hosts that work. On a host that doesn't work:



default via 10.10.0.1 dev em1 proto dhcp metric 100 
10.10.0.0/24 dev em1 proto kernel scope link src 10.10.0.110 metric 100
10.11.0.0/24 dev ib0 proto kernel scope link src 10.11.0.110
169.254.0.0/16 dev ib0 scope link metric 1005


and on one that does:



default via 10.10.0.1 dev em1 proto dhcp metric 100 
10.10.0.0/24 dev em1 proto kernel scope link src 10.10.0.108 metric 100
10.11.0.0/24 dev ib0 proto kernel scope link src 10.11.0.108
169.254.0.0/16 dev ib0 scope link metric 1004


The only thing different is the metric in the last route, but that shouldn't matter. Also of note, these hosts worked before they were reprovisioned. So I'm almost positive it's not hardware.



I'm at a bit of a loss now and any ideas would be appreciated.



Edit: Update with dmesg error



I found something in the output of dmesg for the interface in question that only exists on the hosts that don't work. The error



ib0: failed to modify QP to RTR: -22


unfortunately this isn't very helpful, and there's not much that comes up related in searches.



Perhaps also worth noting, the hosts in question can ping the switch IP address, and the switch can ping the hosts on their associated IPs.







routing infiniband






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Aug 21 at 4:23

























asked Aug 16 at 20:41









geoffjay

32




32











  • While probably not related, I would uninstall avahi. Zeroconf should have no place on enterprise servers.
    – Rui F Ribeiro
    Aug 16 at 21:00










  • It's not actually installed, the script /etc/sysconfig/network-scripts/ifup-ib from OFED adds a route in for some reason. I've removed that and no change.
    – geoffjay
    Aug 16 at 21:30










  • 169.254 is usually a sign it is installed.
    – Rui F Ribeiro
    Aug 16 at 21:45
















  • While probably not related, I would uninstall avahi. Zeroconf should have no place on enterprise servers.
    – Rui F Ribeiro
    Aug 16 at 21:00










  • It's not actually installed, the script /etc/sysconfig/network-scripts/ifup-ib from OFED adds a route in for some reason. I've removed that and no change.
    – geoffjay
    Aug 16 at 21:30










  • 169.254 is usually a sign it is installed.
    – Rui F Ribeiro
    Aug 16 at 21:45















While probably not related, I would uninstall avahi. Zeroconf should have no place on enterprise servers.
– Rui F Ribeiro
Aug 16 at 21:00




While probably not related, I would uninstall avahi. Zeroconf should have no place on enterprise servers.
– Rui F Ribeiro
Aug 16 at 21:00












It's not actually installed, the script /etc/sysconfig/network-scripts/ifup-ib from OFED adds a route in for some reason. I've removed that and no change.
– geoffjay
Aug 16 at 21:30




It's not actually installed, the script /etc/sysconfig/network-scripts/ifup-ib from OFED adds a route in for some reason. I've removed that and no change.
– geoffjay
Aug 16 at 21:30












169.254 is usually a sign it is installed.
– Rui F Ribeiro
Aug 16 at 21:45




169.254 is usually a sign it is installed.
– Rui F Ribeiro
Aug 16 at 21:45










1 Answer
1






active

oldest

votes

















up vote
0
down vote



accepted










This is a known issue in kernel 3.10.0-862.11.1 to 3.10.0-862.11.6 (see here and here).



Essentially, if you update the kernel to 862.11.1-862.11.6, a bug in drivers/infiniband/core/verbs.c where a semi-colon was left out causes all reliable connected (rc) messages to fail while unreliable datagram messages will work. You can either patch this driver, or boot from an earlier kernel to work-around this issue until the updated kernel resolves this issue.






share|improve this answer




















  • Holy hell, that's exactly what it was. I spent a bunch of time trying to fix this and never thought to check the kernel versions because the ones that didn't work were built within a day of the ones that did. Thanks.
    – geoffjay
    Aug 28 at 19:07










Your Answer







StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f463067%2finfiniband-interface-doesnt-route-ipoib-traffic%23new-answer', 'question_page');

);

Post as a guest






























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
0
down vote



accepted










This is a known issue in kernel 3.10.0-862.11.1 to 3.10.0-862.11.6 (see here and here).



Essentially, if you update the kernel to 862.11.1-862.11.6, a bug in drivers/infiniband/core/verbs.c where a semi-colon was left out causes all reliable connected (rc) messages to fail while unreliable datagram messages will work. You can either patch this driver, or boot from an earlier kernel to work-around this issue until the updated kernel resolves this issue.






share|improve this answer




















  • Holy hell, that's exactly what it was. I spent a bunch of time trying to fix this and never thought to check the kernel versions because the ones that didn't work were built within a day of the ones that did. Thanks.
    – geoffjay
    Aug 28 at 19:07














up vote
0
down vote



accepted










This is a known issue in kernel 3.10.0-862.11.1 to 3.10.0-862.11.6 (see here and here).



Essentially, if you update the kernel to 862.11.1-862.11.6, a bug in drivers/infiniband/core/verbs.c where a semi-colon was left out causes all reliable connected (rc) messages to fail while unreliable datagram messages will work. You can either patch this driver, or boot from an earlier kernel to work-around this issue until the updated kernel resolves this issue.






share|improve this answer




















  • Holy hell, that's exactly what it was. I spent a bunch of time trying to fix this and never thought to check the kernel versions because the ones that didn't work were built within a day of the ones that did. Thanks.
    – geoffjay
    Aug 28 at 19:07












up vote
0
down vote



accepted







up vote
0
down vote



accepted






This is a known issue in kernel 3.10.0-862.11.1 to 3.10.0-862.11.6 (see here and here).



Essentially, if you update the kernel to 862.11.1-862.11.6, a bug in drivers/infiniband/core/verbs.c where a semi-colon was left out causes all reliable connected (rc) messages to fail while unreliable datagram messages will work. You can either patch this driver, or boot from an earlier kernel to work-around this issue until the updated kernel resolves this issue.






share|improve this answer












This is a known issue in kernel 3.10.0-862.11.1 to 3.10.0-862.11.6 (see here and here).



Essentially, if you update the kernel to 862.11.1-862.11.6, a bug in drivers/infiniband/core/verbs.c where a semi-colon was left out causes all reliable connected (rc) messages to fail while unreliable datagram messages will work. You can either patch this driver, or boot from an earlier kernel to work-around this issue until the updated kernel resolves this issue.







share|improve this answer












share|improve this answer



share|improve this answer










answered Aug 27 at 15:57









Austin Ewens

161




161











  • Holy hell, that's exactly what it was. I spent a bunch of time trying to fix this and never thought to check the kernel versions because the ones that didn't work were built within a day of the ones that did. Thanks.
    – geoffjay
    Aug 28 at 19:07
















  • Holy hell, that's exactly what it was. I spent a bunch of time trying to fix this and never thought to check the kernel versions because the ones that didn't work were built within a day of the ones that did. Thanks.
    – geoffjay
    Aug 28 at 19:07















Holy hell, that's exactly what it was. I spent a bunch of time trying to fix this and never thought to check the kernel versions because the ones that didn't work were built within a day of the ones that did. Thanks.
– geoffjay
Aug 28 at 19:07




Holy hell, that's exactly what it was. I spent a bunch of time trying to fix this and never thought to check the kernel versions because the ones that didn't work were built within a day of the ones that did. Thanks.
– geoffjay
Aug 28 at 19:07

















 

draft saved


draft discarded















































 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f463067%2finfiniband-interface-doesnt-route-ipoib-traffic%23new-answer', 'question_page');

);

Post as a guest













































































Popular posts from this blog

How to check contact read email or not when send email to Individual?

Displaying single band from multi-band raster using QGIS

How many registers does an x86_64 CPU actually have?