Concurrent requests from node.js process are far too slow - is the OS limiting them
Clash Royale CLAN TAG#URR8PPP
I have a node.js script, it makes a http request to a url N number of times using two different techniques. First it does it serially, then it does it using the node.js Promise api (all the requests go off at nearly the same time and we wait for the responses back).
If I run this script on my mac, I get output of:
node dl.js 70
count: 70
serialHttp ...
serialHttp: 8749.417ms
parallelHttp ...
parallelHttp: 5218.728ms
If I run it on my Pop_OS 18.10 I get:
# running 40 - 70 was taking too long...
count: 40
serialHttp ...
serialHttp: 4938.666ms
parallelHttp ...
# after about 1 minute it times out
So I'm guessing that the OS must be restricting the node process in some way and preventing it from making the requests that it needs.
Would anyone have an idea of what I could tweak to improve this? I've been tinkering with the ulimit etc, but it hasn't helped.
OS Info:
ulimit -a
Maximum size of core files created (kB, -c) 0
Maximum size of a process’s data segment (kB, -d) unlimited
Maximum size of files created by the shell (kB, -f) unlimited
Maximum size that may be locked into memory (kB, -l) 16384
Maximum resident set size (kB, -m) unlimited
Maximum number of open file descriptors (-n) 524288
Maximum stack size (kB, -s) 8192
Maximum amount of cpu time in seconds (seconds, -t) unlimited
Maximum number of processes available to a single user (-u) 524288
Maximum amount of virtual memory available to the shell (kB, -v) unlimited
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Pop!_OS 18.10
Release: 18.10
Codename: cosmic
uname -a
Linux pop-os 4.20.11-042011-generic #201902200535 SMP Wed Feb 20 10:37:40 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
linux ubuntu process http ulimit
asked Feb 21 at 16:42
ed.ed.
1011
add a comment |
I have a node.js script, it makes a http request to a url N number of times using two different techniques. First it does it serially, then it does it using the node.js Promise api (all the requests go off at nearly the same time and we wait for the responses back).
If I run this script on my mac, I get output of:
node dl.js 70
count: 70
serialHttp ...
serialHttp: 8749.417ms
parallelHttp ...
parallelHttp: 5218.728ms
If I run it on my Pop_OS 18.10 I get:
# running 40 - 70 was taking too long...
count: 40
serialHttp ...
serialHttp: 4938.666ms
parallelHttp ...
# after about 1 minute it times out
So I'm guessing that the OS must be restricting the node process in some way and preventing it from making the requests that it needs.
Would anyone have an idea of what I could tweak to improve this? I've been tinkering with the ulimit etc, but it hasn't helped.
OS Info:
ulimit -a
Maximum size of core files created (kB, -c) 0
Maximum size of a process’s data segment (kB, -d) unlimited
Maximum size of files created by the shell (kB, -f) unlimited
Maximum size that may be locked into memory (kB, -l) 16384
Maximum resident set size (kB, -m) unlimited
Maximum number of open file descriptors (-n) 524288
Maximum stack size (kB, -s) 8192
Maximum amount of cpu time in seconds (seconds, -t) unlimited
Maximum number of processes available to a single user (-u) 524288
Maximum amount of virtual memory available to the shell (kB, -v) unlimited
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Pop!_OS 18.10
Release: 18.10
Codename: cosmic
uname -a
Linux pop-os 4.20.11-042011-generic #201902200535 SMP Wed Feb 20 10:37:40 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
linux ubuntu process http ulimit
asked Feb 21 at 16:42
ed.ed.
1011
add a comment |
I have a node.js script, it makes a http request to a url N number of times using two different techniques. First it does it serially, then it does it using the node.js Promise api (all the requests go off at nearly the same time and we wait for the responses back).
If I run this script on my mac, I get output of:
node dl.js 70
count: 70
serialHttp ...
serialHttp: 8749.417ms
parallelHttp ...
parallelHttp: 5218.728ms
If I run it on my Pop_OS 18.10 I get:
# running 40 - 70 was taking too long...
count: 40
serialHttp ...
serialHttp: 4938.666ms
parallelHttp ...
# after about 1 minute it times out
So I'm guessing that the OS must be restricting the node process in some way and preventing it from making the requests that it needs.
Would anyone have an idea of what I could tweak to improve this? I've been tinkering with the ulimit etc, but it hasn't helped.
OS Info:
ulimit -a
Maximum size of core files created (kB, -c) 0
Maximum size of a process’s data segment (kB, -d) unlimited
Maximum size of files created by the shell (kB, -f) unlimited
Maximum size that may be locked into memory (kB, -l) 16384
Maximum resident set size (kB, -m) unlimited
Maximum number of open file descriptors (-n) 524288
Maximum stack size (kB, -s) 8192
Maximum amount of cpu time in seconds (seconds, -t) unlimited
Maximum number of processes available to a single user (-u) 524288
Maximum amount of virtual memory available to the shell (kB, -v) unlimited
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Pop!_OS 18.10
Release: 18.10
Codename: cosmic
uname -a
Linux pop-os 4.20.11-042011-generic #201902200535 SMP Wed Feb 20 10:37:40 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
linux ubuntu process http ulimit
asked Feb 21 at 16:42
ed.ed.
1011
I have a node.js script, it makes a http request to a url N number of times using two different techniques. First it does it serially, then it does it using the node.js Promise api (all the requests go off at nearly the same time and we wait for the responses back).
If I run this script on my mac, I get output of:
node dl.js 70
count: 70
serialHttp ...
serialHttp: 8749.417ms
parallelHttp ...
parallelHttp: 5218.728ms
If I run it on my Pop_OS 18.10 I get:
# running 40 - 70 was taking too long...
count: 40
serialHttp ...
serialHttp: 4938.666ms
parallelHttp ...
# after about 1 minute it times out
So I'm guessing that the OS must be restricting the node process in some way and preventing it from making the requests that it needs.
Would anyone have an idea of what I could tweak to improve this? I've been tinkering with the ulimit etc, but it hasn't helped.
OS Info:
ulimit -a
Maximum size of core files created (kB, -c) 0
Maximum size of a process’s data segment (kB, -d) unlimited
Maximum size of files created by the shell (kB, -f) unlimited
Maximum size that may be locked into memory (kB, -l) 16384
Maximum resident set size (kB, -m) unlimited
Maximum number of open file descriptors (-n) 524288
Maximum stack size (kB, -s) 8192
Maximum amount of cpu time in seconds (seconds, -t) unlimited
Maximum number of processes available to a single user (-u) 524288
Maximum amount of virtual memory available to the shell (kB, -v) unlimited
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Pop!_OS 18.10
Release: 18.10
Codename: cosmic
uname -a
Linux pop-os 4.20.11-042011-generic #201902200535 SMP Wed Feb 20 10:37:40 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
linux ubuntu process http ulimit
linux ubuntu process http ulimit
asked Feb 21 at 16:42
ed.ed.
1011
asked Feb 21 at 16:42
ed.ed.
1011
asked Feb 21 at 16:42
ed.ed.
1011
asked Feb 21 at 16:42
ed.ed.
1011
asked Feb 21 at 16:42
ed.ed.
1011
1011
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f502124%2fconcurrent-requests-from-node-js-process-are-far-too-slow-is-the-os-limiting-t%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f502124%2fconcurrent-requests-from-node-js-process-are-far-too-slow-is-the-os-limiting-t%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
