Is it possible to manage total threads for pipes of multi-threaded commands?
Clash Royale CLAN TAG#URR8PPP
I am running several tools that have multi-threading capabilities. For example:
tool1 --threads=4 | tool2 --threads=4 | tool3 --threads=4
In an ideal case, I assume this should utilize 12 (4*3) threads. It's possible that the first tool is the slowest, so then the downstream tools would only utilize a fraction of the requested threads. In that case, you may want to give the first tool more threads. Is there a way to manage that?
pipe multithreading
asked Feb 11 at 17:32
burgerburger
1375
add a comment |
I am running several tools that have multi-threading capabilities. For example:
tool1 --threads=4 | tool2 --threads=4 | tool3 --threads=4
In an ideal case, I assume this should utilize 12 (4*3) threads. It's possible that the first tool is the slowest, so then the downstream tools would only utilize a fraction of the requested threads. In that case, you may want to give the first tool more threads. Is there a way to manage that?
pipe multithreading
asked Feb 11 at 17:32
burgerburger
1375
Do you want to dynamically change the number of threads? Then your tools need an interface to do this and to report their thread usage. And you probably need a controlling process that checks the threads of all tools and adjusts their thread limits. Or do you want to use the statistics from one run to calculate the parameters for the next run? This would be the only option if all you have is a command line option--threads=number
– Bodo
Feb 11 at 17:41
add a comment |
I am running several tools that have multi-threading capabilities. For example:
tool1 --threads=4 | tool2 --threads=4 | tool3 --threads=4
In an ideal case, I assume this should utilize 12 (4*3) threads. It's possible that the first tool is the slowest, so then the downstream tools would only utilize a fraction of the requested threads. In that case, you may want to give the first tool more threads. Is there a way to manage that?
pipe multithreading
asked Feb 11 at 17:32
burgerburger
1375
I am running several tools that have multi-threading capabilities. For example:
tool1 --threads=4 | tool2 --threads=4 | tool3 --threads=4
In an ideal case, I assume this should utilize 12 (4*3) threads. It's possible that the first tool is the slowest, so then the downstream tools would only utilize a fraction of the requested threads. In that case, you may want to give the first tool more threads. Is there a way to manage that?
pipe multithreading
pipe multithreading
asked Feb 11 at 17:32
burgerburger
1375
asked Feb 11 at 17:32
burgerburger
1375
asked Feb 11 at 17:32
burgerburger
1375
asked Feb 11 at 17:32
burgerburger
1375
asked Feb 11 at 17:32
burgerburger
1375
1375
Do you want to dynamically change the number of threads? Then your tools need an interface to do this and to report their thread usage. And you probably need a controlling process that checks the threads of all tools and adjusts their thread limits. Or do you want to use the statistics from one run to calculate the parameters for the next run? This would be the only option if all you have is a command line option--threads=number
– Bodo
Feb 11 at 17:41
add a comment |
Do you want to dynamically change the number of threads? Then your tools need an interface to do this and to report their thread usage. And you probably need a controlling process that checks the threads of all tools and adjusts their thread limits. Or do you want to use the statistics from one run to calculate the parameters for the next run? This would be the only option if all you have is a command line option--threads=number
– Bodo
Feb 11 at 17:41
Do you want to dynamically change the number of threads? Then your tools need an interface to do this and to report their thread usage. And you probably need a controlling process that checks the threads of all tools and adjusts their thread limits. Or do you want to use the statistics from one run to calculate the parameters for the next run? This would be the only option if all you have is a command line option
--threads=number– Bodo
Feb 11 at 17:41
Do you want to dynamically change the number of threads? Then your tools need an interface to do this and to report their thread usage. And you probably need a controlling process that checks the threads of all tools and adjusts their thread limits. Or do you want to use the statistics from one run to calculate the parameters for the next run? This would be the only option if all you have is a command line option
--threads=number– Bodo
Feb 11 at 17:41
add a comment |
1 Answer
1
active
oldest
votes
Based on your example, the thread management is built right into the tools you're using in your pipeline.
It doesn't matter how many threads your process is using, each process has precisely one each of standard input, output, and error. All of tool1's standard output - no matter what the value provised to the --threads argument -- will be sent to the standard input of tool2, and so forth.
answered Feb 11 at 17:40
DopeGhotiDopeGhoti
46k56089
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f499991%2fis-it-possible-to-manage-total-threads-for-pipes-of-multi-threaded-commands%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Based on your example, the thread management is built right into the tools you're using in your pipeline.
It doesn't matter how many threads your process is using, each process has precisely one each of standard input, output, and error. All of tool1's standard output - no matter what the value provised to the --threads argument -- will be sent to the standard input of tool2, and so forth.
answered Feb 11 at 17:40
DopeGhotiDopeGhoti
46k56089
add a comment |
Based on your example, the thread management is built right into the tools you're using in your pipeline.
It doesn't matter how many threads your process is using, each process has precisely one each of standard input, output, and error. All of tool1's standard output - no matter what the value provised to the --threads argument -- will be sent to the standard input of tool2, and so forth.
answered Feb 11 at 17:40
DopeGhotiDopeGhoti
46k56089
add a comment |
Based on your example, the thread management is built right into the tools you're using in your pipeline.
It doesn't matter how many threads your process is using, each process has precisely one each of standard input, output, and error. All of tool1's standard output - no matter what the value provised to the --threads argument -- will be sent to the standard input of tool2, and so forth.
answered Feb 11 at 17:40
DopeGhotiDopeGhoti
46k56089
Based on your example, the thread management is built right into the tools you're using in your pipeline.
It doesn't matter how many threads your process is using, each process has precisely one each of standard input, output, and error. All of tool1's standard output - no matter what the value provised to the --threads argument -- will be sent to the standard input of tool2, and so forth.
answered Feb 11 at 17:40
DopeGhotiDopeGhoti
46k56089
answered Feb 11 at 17:40
DopeGhotiDopeGhoti
46k56089
answered Feb 11 at 17:40
DopeGhotiDopeGhoti
46k56089
answered Feb 11 at 17:40
DopeGhotiDopeGhoti
46k56089
46k56089
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f499991%2fis-it-possible-to-manage-total-threads-for-pipes-of-multi-threaded-commands%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Do you want to dynamically change the number of threads? Then your tools need an interface to do this and to report their thread usage. And you probably need a controlling process that checks the threads of all tools and adjusts their thread limits. Or do you want to use the statistics from one run to calculate the parameters for the next run? This would be the only option if all you have is a command line option
--threads=number– Bodo
Feb 11 at 17:41