Using ethtool filters to balance NIC queues

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
1
down vote

favorite












I have amulti-threaded server application running in Linux 4.4.0 with a X540-AT2 NIC (the server has one thread per core). Since Linux has RSS enabled, it uses one NIC receive queue for each core in the system (16 cores and therefore 16 RX queues).



My objective is to let a client application running on a separate host to "hint" to what queue should packets be directed (i.e., some kind of client-directed receive queue load balancing).



To achieve this, I've been playing with the NIC's flow director table with no luck (any ideas?):




  • VLAN tag:
    The server host assigns each VLAN identifier to a separate RX queue (using ethtool --config-ntuple), and the client app assigns a VLAN tag to each packet in order to identify the target receive queue (therefore achieving the client-directed balancing I want).



    Unfortunately, the server never receives any of the client's packets, since it is listening on the main NIC interface, which has no VLANs assigned to it in the system. Is there a way to drop the VLAN tags once packets are received, so that they'll only be used for my balancing needs?




  • TOS field (bits 8-15 of IPv4 header):
    I've also tried using IPv4's TOS field to do the same. Te server host uses ethtool to direct each TOS value to a separate queue, and the client crafts sent packets to have a TOS value according to the desired target receive queue on the server.



    Unfortunately, it seems that ethtool is ignoring my TOS values on the filters (TOS is always 0 on the rules):




    $ sudo ethtool -U em2 flow-type tcp4 tos 1 action 10
    Added rule with ID 2045
    $ ethtool --show-ntuple em2
    16 RX rings available
    Total 1 rules
    Filter: 2045
    Rule Type: TCP over IPv4
    Src IP addr: 0.0.0.0 mask: 255.255.255.255
    Dest IP addr: 0.0.0.0 mask: 255.255.255.255
    TOS: 0x0 mask: 0xff
    Src port: 0 mask: 0xffff
    Dest port: 0 mask: 0xffff
    VLAN EtherType: 0x0 mask: 0xffff
    VLAN: 0x0 mask: 0xffff
    User-defined: 0x0 mask: 0xffffffffffffffff
    Action: Direct to queue 10



  • user-def:
    I've also tried with user-def to overcome the TOS field "problem", but it seems I can only use that on the last two bytes:




    $ sudo ethtool -U em2 flow-type tcp4 user-def 2 action 10
    Added rule with ID 2045
    $ ethtool --show-ntuple em2
    16 RX rings available
    Total 1 rules
    Filter: 2045
    Rule Type: TCP over IPv4
    Src IP addr: 0.0.0.0 mask: 255.255.255.255
    Dest IP addr: 0.0.0.0 mask: 255.255.255.255
    TOS: 0x0 mask: 0xff
    Src port: 0 mask: 0xffff
    Dest port: 0 mask: 0xffff
    VLAN EtherType: 0x0 mask: 0xffff
    VLAN: 0x0 mask: 0xffff
    User-defined: 0x2 mask: 0xffffffffffffff00
    Action: Direct to queue 10


    And when I try to match some other bytes, it's simply ignored (user-defined is always zero and the mask is full):




    $ sudo ethtool -U em2 flow-type tcp4 user-def 2 m 0xf0ffffffffffffff action 10
    Added rule with ID 2045
    $ ethtool --show-ntuple em2
    16 RX rings available
    Total 1 rules
    Filter: 2045
    Rule Type: TCP over IPv4
    Src IP addr: 0.0.0.0 mask: 255.255.255.255
    Dest IP addr: 0.0.0.0 mask: 255.255.255.255
    TOS: 0x0 mask: 0xff
    Src port: 0 mask: 0xffff
    Dest port: 0 mask: 0xffff
    VLAN EtherType: 0x0 mask: 0xffff
    VLAN: 0x0 mask: 0xffff
    User-defined: 0x0 mask: 0xffffffffffffffff
    Action: Direct to queue 10


Any idea how I can solve the problems above? (any of the VLAN or TOS approaches would work for me).



Edit: Clarified question as requested by @Hauke Laging.







share|improve this question


























    up vote
    1
    down vote

    favorite












    I have amulti-threaded server application running in Linux 4.4.0 with a X540-AT2 NIC (the server has one thread per core). Since Linux has RSS enabled, it uses one NIC receive queue for each core in the system (16 cores and therefore 16 RX queues).



    My objective is to let a client application running on a separate host to "hint" to what queue should packets be directed (i.e., some kind of client-directed receive queue load balancing).



    To achieve this, I've been playing with the NIC's flow director table with no luck (any ideas?):




    • VLAN tag:
      The server host assigns each VLAN identifier to a separate RX queue (using ethtool --config-ntuple), and the client app assigns a VLAN tag to each packet in order to identify the target receive queue (therefore achieving the client-directed balancing I want).



      Unfortunately, the server never receives any of the client's packets, since it is listening on the main NIC interface, which has no VLANs assigned to it in the system. Is there a way to drop the VLAN tags once packets are received, so that they'll only be used for my balancing needs?




    • TOS field (bits 8-15 of IPv4 header):
      I've also tried using IPv4's TOS field to do the same. Te server host uses ethtool to direct each TOS value to a separate queue, and the client crafts sent packets to have a TOS value according to the desired target receive queue on the server.



      Unfortunately, it seems that ethtool is ignoring my TOS values on the filters (TOS is always 0 on the rules):




      $ sudo ethtool -U em2 flow-type tcp4 tos 1 action 10
      Added rule with ID 2045
      $ ethtool --show-ntuple em2
      16 RX rings available
      Total 1 rules
      Filter: 2045
      Rule Type: TCP over IPv4
      Src IP addr: 0.0.0.0 mask: 255.255.255.255
      Dest IP addr: 0.0.0.0 mask: 255.255.255.255
      TOS: 0x0 mask: 0xff
      Src port: 0 mask: 0xffff
      Dest port: 0 mask: 0xffff
      VLAN EtherType: 0x0 mask: 0xffff
      VLAN: 0x0 mask: 0xffff
      User-defined: 0x0 mask: 0xffffffffffffffff
      Action: Direct to queue 10



    • user-def:
      I've also tried with user-def to overcome the TOS field "problem", but it seems I can only use that on the last two bytes:




      $ sudo ethtool -U em2 flow-type tcp4 user-def 2 action 10
      Added rule with ID 2045
      $ ethtool --show-ntuple em2
      16 RX rings available
      Total 1 rules
      Filter: 2045
      Rule Type: TCP over IPv4
      Src IP addr: 0.0.0.0 mask: 255.255.255.255
      Dest IP addr: 0.0.0.0 mask: 255.255.255.255
      TOS: 0x0 mask: 0xff
      Src port: 0 mask: 0xffff
      Dest port: 0 mask: 0xffff
      VLAN EtherType: 0x0 mask: 0xffff
      VLAN: 0x0 mask: 0xffff
      User-defined: 0x2 mask: 0xffffffffffffff00
      Action: Direct to queue 10


      And when I try to match some other bytes, it's simply ignored (user-defined is always zero and the mask is full):




      $ sudo ethtool -U em2 flow-type tcp4 user-def 2 m 0xf0ffffffffffffff action 10
      Added rule with ID 2045
      $ ethtool --show-ntuple em2
      16 RX rings available
      Total 1 rules
      Filter: 2045
      Rule Type: TCP over IPv4
      Src IP addr: 0.0.0.0 mask: 255.255.255.255
      Dest IP addr: 0.0.0.0 mask: 255.255.255.255
      TOS: 0x0 mask: 0xff
      Src port: 0 mask: 0xffff
      Dest port: 0 mask: 0xffff
      VLAN EtherType: 0x0 mask: 0xffff
      VLAN: 0x0 mask: 0xffff
      User-defined: 0x0 mask: 0xffffffffffffffff
      Action: Direct to queue 10


    Any idea how I can solve the problems above? (any of the VLAN or TOS approaches would work for me).



    Edit: Clarified question as requested by @Hauke Laging.







    share|improve this question
























      up vote
      1
      down vote

      favorite









      up vote
      1
      down vote

      favorite











      I have amulti-threaded server application running in Linux 4.4.0 with a X540-AT2 NIC (the server has one thread per core). Since Linux has RSS enabled, it uses one NIC receive queue for each core in the system (16 cores and therefore 16 RX queues).



      My objective is to let a client application running on a separate host to "hint" to what queue should packets be directed (i.e., some kind of client-directed receive queue load balancing).



      To achieve this, I've been playing with the NIC's flow director table with no luck (any ideas?):




      • VLAN tag:
        The server host assigns each VLAN identifier to a separate RX queue (using ethtool --config-ntuple), and the client app assigns a VLAN tag to each packet in order to identify the target receive queue (therefore achieving the client-directed balancing I want).



        Unfortunately, the server never receives any of the client's packets, since it is listening on the main NIC interface, which has no VLANs assigned to it in the system. Is there a way to drop the VLAN tags once packets are received, so that they'll only be used for my balancing needs?




      • TOS field (bits 8-15 of IPv4 header):
        I've also tried using IPv4's TOS field to do the same. Te server host uses ethtool to direct each TOS value to a separate queue, and the client crafts sent packets to have a TOS value according to the desired target receive queue on the server.



        Unfortunately, it seems that ethtool is ignoring my TOS values on the filters (TOS is always 0 on the rules):




        $ sudo ethtool -U em2 flow-type tcp4 tos 1 action 10
        Added rule with ID 2045
        $ ethtool --show-ntuple em2
        16 RX rings available
        Total 1 rules
        Filter: 2045
        Rule Type: TCP over IPv4
        Src IP addr: 0.0.0.0 mask: 255.255.255.255
        Dest IP addr: 0.0.0.0 mask: 255.255.255.255
        TOS: 0x0 mask: 0xff
        Src port: 0 mask: 0xffff
        Dest port: 0 mask: 0xffff
        VLAN EtherType: 0x0 mask: 0xffff
        VLAN: 0x0 mask: 0xffff
        User-defined: 0x0 mask: 0xffffffffffffffff
        Action: Direct to queue 10



      • user-def:
        I've also tried with user-def to overcome the TOS field "problem", but it seems I can only use that on the last two bytes:




        $ sudo ethtool -U em2 flow-type tcp4 user-def 2 action 10
        Added rule with ID 2045
        $ ethtool --show-ntuple em2
        16 RX rings available
        Total 1 rules
        Filter: 2045
        Rule Type: TCP over IPv4
        Src IP addr: 0.0.0.0 mask: 255.255.255.255
        Dest IP addr: 0.0.0.0 mask: 255.255.255.255
        TOS: 0x0 mask: 0xff
        Src port: 0 mask: 0xffff
        Dest port: 0 mask: 0xffff
        VLAN EtherType: 0x0 mask: 0xffff
        VLAN: 0x0 mask: 0xffff
        User-defined: 0x2 mask: 0xffffffffffffff00
        Action: Direct to queue 10


        And when I try to match some other bytes, it's simply ignored (user-defined is always zero and the mask is full):




        $ sudo ethtool -U em2 flow-type tcp4 user-def 2 m 0xf0ffffffffffffff action 10
        Added rule with ID 2045
        $ ethtool --show-ntuple em2
        16 RX rings available
        Total 1 rules
        Filter: 2045
        Rule Type: TCP over IPv4
        Src IP addr: 0.0.0.0 mask: 255.255.255.255
        Dest IP addr: 0.0.0.0 mask: 255.255.255.255
        TOS: 0x0 mask: 0xff
        Src port: 0 mask: 0xffff
        Dest port: 0 mask: 0xffff
        VLAN EtherType: 0x0 mask: 0xffff
        VLAN: 0x0 mask: 0xffff
        User-defined: 0x0 mask: 0xffffffffffffffff
        Action: Direct to queue 10


      Any idea how I can solve the problems above? (any of the VLAN or TOS approaches would work for me).



      Edit: Clarified question as requested by @Hauke Laging.







      share|improve this question














      I have amulti-threaded server application running in Linux 4.4.0 with a X540-AT2 NIC (the server has one thread per core). Since Linux has RSS enabled, it uses one NIC receive queue for each core in the system (16 cores and therefore 16 RX queues).



      My objective is to let a client application running on a separate host to "hint" to what queue should packets be directed (i.e., some kind of client-directed receive queue load balancing).



      To achieve this, I've been playing with the NIC's flow director table with no luck (any ideas?):




      • VLAN tag:
        The server host assigns each VLAN identifier to a separate RX queue (using ethtool --config-ntuple), and the client app assigns a VLAN tag to each packet in order to identify the target receive queue (therefore achieving the client-directed balancing I want).



        Unfortunately, the server never receives any of the client's packets, since it is listening on the main NIC interface, which has no VLANs assigned to it in the system. Is there a way to drop the VLAN tags once packets are received, so that they'll only be used for my balancing needs?




      • TOS field (bits 8-15 of IPv4 header):
        I've also tried using IPv4's TOS field to do the same. Te server host uses ethtool to direct each TOS value to a separate queue, and the client crafts sent packets to have a TOS value according to the desired target receive queue on the server.



        Unfortunately, it seems that ethtool is ignoring my TOS values on the filters (TOS is always 0 on the rules):




        $ sudo ethtool -U em2 flow-type tcp4 tos 1 action 10
        Added rule with ID 2045
        $ ethtool --show-ntuple em2
        16 RX rings available
        Total 1 rules
        Filter: 2045
        Rule Type: TCP over IPv4
        Src IP addr: 0.0.0.0 mask: 255.255.255.255
        Dest IP addr: 0.0.0.0 mask: 255.255.255.255
        TOS: 0x0 mask: 0xff
        Src port: 0 mask: 0xffff
        Dest port: 0 mask: 0xffff
        VLAN EtherType: 0x0 mask: 0xffff
        VLAN: 0x0 mask: 0xffff
        User-defined: 0x0 mask: 0xffffffffffffffff
        Action: Direct to queue 10



      • user-def:
        I've also tried with user-def to overcome the TOS field "problem", but it seems I can only use that on the last two bytes:




        $ sudo ethtool -U em2 flow-type tcp4 user-def 2 action 10
        Added rule with ID 2045
        $ ethtool --show-ntuple em2
        16 RX rings available
        Total 1 rules
        Filter: 2045
        Rule Type: TCP over IPv4
        Src IP addr: 0.0.0.0 mask: 255.255.255.255
        Dest IP addr: 0.0.0.0 mask: 255.255.255.255
        TOS: 0x0 mask: 0xff
        Src port: 0 mask: 0xffff
        Dest port: 0 mask: 0xffff
        VLAN EtherType: 0x0 mask: 0xffff
        VLAN: 0x0 mask: 0xffff
        User-defined: 0x2 mask: 0xffffffffffffff00
        Action: Direct to queue 10


        And when I try to match some other bytes, it's simply ignored (user-defined is always zero and the mask is full):




        $ sudo ethtool -U em2 flow-type tcp4 user-def 2 m 0xf0ffffffffffffff action 10
        Added rule with ID 2045
        $ ethtool --show-ntuple em2
        16 RX rings available
        Total 1 rules
        Filter: 2045
        Rule Type: TCP over IPv4
        Src IP addr: 0.0.0.0 mask: 255.255.255.255
        Dest IP addr: 0.0.0.0 mask: 255.255.255.255
        TOS: 0x0 mask: 0xff
        Src port: 0 mask: 0xffff
        Dest port: 0 mask: 0xffff
        VLAN EtherType: 0x0 mask: 0xffff
        VLAN: 0x0 mask: 0xffff
        User-defined: 0x0 mask: 0xffffffffffffffff
        Action: Direct to queue 10


      Any idea how I can solve the problems above? (any of the VLAN or TOS approaches would work for me).



      Edit: Clarified question as requested by @Hauke Laging.









      share|improve this question













      share|improve this question




      share|improve this question








      edited Oct 15 '17 at 16:02

























      asked Oct 15 '17 at 8:27









      Lluís Vilanova

      1065




      1065




















          1 Answer
          1






          active

          oldest

          votes

















          up vote
          0
          down vote













          You should make clear what kind of queues you want to balance and with which effect.



          In addition you should mention what you can configure on the server. You mentioned VLAN. It seems you cannot configure VLAN on the server.



          There are several possibilities to send packets to the same system in different ways:



          1. Using different TCP/UDP ports (if TCP/UDP is used). This can be done transparently to the application by using DNAT on both the client and the server.

          2. Using different IP addresses on the server. This can be done transparently to the application by using DNAT on both the client and the server.

          3. Creating several tunnel interfaces on client and server (ip link add type ipip) and sending the packets over several interfaces.

          4. Ingress queueing can be done with the ifb device and traffic shaping (based on the above "information").





          share|improve this answer




















          • I've just rewritten the question in a way that I hope is much clearer. Thanks!
            – Lluís Vilanova
            Oct 15 '17 at 16:02










          • Oh, and for the record, I can configure whatever necessary on both the client and server hosts.
            – Lluís Vilanova
            Oct 15 '17 at 17:10










          Your Answer







          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "106"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          convertImagesToLinks: false,
          noModals: false,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













           

          draft saved


          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f398209%2fusing-ethtool-filters-to-balance-nic-queues%23new-answer', 'question_page');

          );

          Post as a guest






























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes








          up vote
          0
          down vote













          You should make clear what kind of queues you want to balance and with which effect.



          In addition you should mention what you can configure on the server. You mentioned VLAN. It seems you cannot configure VLAN on the server.



          There are several possibilities to send packets to the same system in different ways:



          1. Using different TCP/UDP ports (if TCP/UDP is used). This can be done transparently to the application by using DNAT on both the client and the server.

          2. Using different IP addresses on the server. This can be done transparently to the application by using DNAT on both the client and the server.

          3. Creating several tunnel interfaces on client and server (ip link add type ipip) and sending the packets over several interfaces.

          4. Ingress queueing can be done with the ifb device and traffic shaping (based on the above "information").





          share|improve this answer




















          • I've just rewritten the question in a way that I hope is much clearer. Thanks!
            – Lluís Vilanova
            Oct 15 '17 at 16:02










          • Oh, and for the record, I can configure whatever necessary on both the client and server hosts.
            – Lluís Vilanova
            Oct 15 '17 at 17:10














          up vote
          0
          down vote













          You should make clear what kind of queues you want to balance and with which effect.



          In addition you should mention what you can configure on the server. You mentioned VLAN. It seems you cannot configure VLAN on the server.



          There are several possibilities to send packets to the same system in different ways:



          1. Using different TCP/UDP ports (if TCP/UDP is used). This can be done transparently to the application by using DNAT on both the client and the server.

          2. Using different IP addresses on the server. This can be done transparently to the application by using DNAT on both the client and the server.

          3. Creating several tunnel interfaces on client and server (ip link add type ipip) and sending the packets over several interfaces.

          4. Ingress queueing can be done with the ifb device and traffic shaping (based on the above "information").





          share|improve this answer




















          • I've just rewritten the question in a way that I hope is much clearer. Thanks!
            – Lluís Vilanova
            Oct 15 '17 at 16:02










          • Oh, and for the record, I can configure whatever necessary on both the client and server hosts.
            – Lluís Vilanova
            Oct 15 '17 at 17:10












          up vote
          0
          down vote










          up vote
          0
          down vote









          You should make clear what kind of queues you want to balance and with which effect.



          In addition you should mention what you can configure on the server. You mentioned VLAN. It seems you cannot configure VLAN on the server.



          There are several possibilities to send packets to the same system in different ways:



          1. Using different TCP/UDP ports (if TCP/UDP is used). This can be done transparently to the application by using DNAT on both the client and the server.

          2. Using different IP addresses on the server. This can be done transparently to the application by using DNAT on both the client and the server.

          3. Creating several tunnel interfaces on client and server (ip link add type ipip) and sending the packets over several interfaces.

          4. Ingress queueing can be done with the ifb device and traffic shaping (based on the above "information").





          share|improve this answer












          You should make clear what kind of queues you want to balance and with which effect.



          In addition you should mention what you can configure on the server. You mentioned VLAN. It seems you cannot configure VLAN on the server.



          There are several possibilities to send packets to the same system in different ways:



          1. Using different TCP/UDP ports (if TCP/UDP is used). This can be done transparently to the application by using DNAT on both the client and the server.

          2. Using different IP addresses on the server. This can be done transparently to the application by using DNAT on both the client and the server.

          3. Creating several tunnel interfaces on client and server (ip link add type ipip) and sending the packets over several interfaces.

          4. Ingress queueing can be done with the ifb device and traffic shaping (based on the above "information").






          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Oct 15 '17 at 11:31









          Hauke Laging

          53.6k1282130




          53.6k1282130











          • I've just rewritten the question in a way that I hope is much clearer. Thanks!
            – Lluís Vilanova
            Oct 15 '17 at 16:02










          • Oh, and for the record, I can configure whatever necessary on both the client and server hosts.
            – Lluís Vilanova
            Oct 15 '17 at 17:10
















          • I've just rewritten the question in a way that I hope is much clearer. Thanks!
            – Lluís Vilanova
            Oct 15 '17 at 16:02










          • Oh, and for the record, I can configure whatever necessary on both the client and server hosts.
            – Lluís Vilanova
            Oct 15 '17 at 17:10















          I've just rewritten the question in a way that I hope is much clearer. Thanks!
          – Lluís Vilanova
          Oct 15 '17 at 16:02




          I've just rewritten the question in a way that I hope is much clearer. Thanks!
          – Lluís Vilanova
          Oct 15 '17 at 16:02












          Oh, and for the record, I can configure whatever necessary on both the client and server hosts.
          – Lluís Vilanova
          Oct 15 '17 at 17:10




          Oh, and for the record, I can configure whatever necessary on both the client and server hosts.
          – Lluís Vilanova
          Oct 15 '17 at 17:10

















           

          draft saved


          draft discarded















































           


          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f398209%2fusing-ethtool-filters-to-balance-nic-queues%23new-answer', 'question_page');

          );

          Post as a guest













































































          Popular posts from this blog

          How to check contact read email or not when send email to Individual?

          Displaying single band from multi-band raster using QGIS

          How many registers does an x86_64 CPU actually have?