Perl script running infinitely - how to debug what happened?
Clash Royale CLAN TAG#URR8PPP
up vote
6
down vote
favorite
I am running a Perl script on a Linux machine via a cron job. However, from time to time (around 1% of all cases), the script gets stuck and runs infinitely. If I list processes, I can see its PID. However, I don't want to kill it right away; I would rather know what went wrong.
Is there a way how to show what lines are being executed from the script? Something like step-by-step debugging of the script based on PID.
linux perl
edited Sep 22 at 20:50
Jeff Schaller
33.3k849111
asked Sep 22 at 9:07
Martin Perry
1506
add a comment |Â
up vote
6
down vote
favorite
I am running a Perl script on a Linux machine via a cron job. However, from time to time (around 1% of all cases), the script gets stuck and runs infinitely. If I list processes, I can see its PID. However, I don't want to kill it right away; I would rather know what went wrong.
Is there a way how to show what lines are being executed from the script? Something like step-by-step debugging of the script based on PID.
linux perl
edited Sep 22 at 20:50
Jeff Schaller
33.3k849111
asked Sep 22 at 9:07
Martin Perry
1506
add a comment |Â
up vote
6
down vote
favorite
up vote
6
down vote
favorite
I am running a Perl script on a Linux machine via a cron job. However, from time to time (around 1% of all cases), the script gets stuck and runs infinitely. If I list processes, I can see its PID. However, I don't want to kill it right away; I would rather know what went wrong.
Is there a way how to show what lines are being executed from the script? Something like step-by-step debugging of the script based on PID.
linux perl
edited Sep 22 at 20:50
Jeff Schaller
33.3k849111
asked Sep 22 at 9:07
Martin Perry
1506
I am running a Perl script on a Linux machine via a cron job. However, from time to time (around 1% of all cases), the script gets stuck and runs infinitely. If I list processes, I can see its PID. However, I don't want to kill it right away; I would rather know what went wrong.
Is there a way how to show what lines are being executed from the script? Something like step-by-step debugging of the script based on PID.
linux perl
linux perl
edited Sep 22 at 20:50
Jeff Schaller
33.3k849111
asked Sep 22 at 9:07
Martin Perry
1506
edited Sep 22 at 20:50
Jeff Schaller
33.3k849111
asked Sep 22 at 9:07
Martin Perry
1506
edited Sep 22 at 20:50
Jeff Schaller
33.3k849111
edited Sep 22 at 20:50
Jeff Schaller
33.3k849111
edited Sep 22 at 20:50
Jeff Schaller
33.3k849111
33.3k849111
asked Sep 22 at 9:07
Martin Perry
1506
asked Sep 22 at 9:07
Martin Perry
1506
asked Sep 22 at 9:07
Martin Perry
1506
1506
add a comment |Â
add a comment |Â
2 Answers
2
active
oldest
votes
up vote
10
down vote
accepted
Try to follow these steps:
- find the process pid of the shell, you may use a command like:
ps -ef | grep <your_script_name>
Let's set this pid in the shell variable $PID. Find all the child processes of this $PID by run the command:
ps --ppid $PID
You might find one or more (if for example it's stuck in a pipelined series of commands). Repeat this command couple of times. If it doesn't change this means the script is stuck in certain command. In this case, you may attach trace command to the running child process:
sudo strace -p $PID
This will show you what is being executed, either indefinite loop (like reading from a pipe), or waiting on some event that never happens.
In case you find ps --ppid $PID changes, this indicates that your script is advancing but it's stuck somewhere, e.g. local loop in the script. From the changing commands, it can give you a hint where in the script it's looping.
Finally, a very simple method to debug a perl is to use perl debugger:
perl -d script.pl
More: 1, 2, 3, 4, 5
edited 2 days ago
Jeff Schaller
33.3k849111
answered Sep 22 at 9:14
Goro
6,20152763
Why are you looking for child processes of the script?perldoesn't need to run child processes unless you use things likesystem()to execute external commands.
– Barmar
Sep 22 at 18:20
Did you misread the question to be asking about abashscript rather thanperl? Your Stack Overflow links suggest so.
– Barmar
Sep 22 at 18:22
Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
– Goro
Sep 22 at 18:48
1
The "More" links at the end are all about debugging bash, not perl.
– Barmar
Sep 22 at 19:39
Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit inwait()while waiting for it to finish. But you should add thestraceoption to traces forked children.
– Barmar
Sep 22 at 19:40
 |Â
show 1 more comment
up vote
3
down vote
For the next runs of your script you can try out the package Devel::Trace.
From description: "This module will print a message to standard error just before each line is executed."
Run either with
perl -d:Trace program
or use in your script with
import Devel::Trace 'trace';
trace 'on'; # Enable
trace 'off'; # Disable
answered Sep 22 at 9:17
Marvin
1464
But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably trystraceas in Goro's answer first.
– Henrik
Sep 22 at 9:59
@Henrik While debugging, you can runbash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$'to capture the output. And it doesn't matter much if many of generated files here are useless. Thestraceapproach is universal but the output there must be harder to match with the actual Perl code.
– Kirill Bulygin
Sep 22 at 12:39
add a comment |Â
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
10
down vote
accepted
Try to follow these steps:
- find the process pid of the shell, you may use a command like:
ps -ef | grep <your_script_name>
Let's set this pid in the shell variable $PID. Find all the child processes of this $PID by run the command:
ps --ppid $PID
You might find one or more (if for example it's stuck in a pipelined series of commands). Repeat this command couple of times. If it doesn't change this means the script is stuck in certain command. In this case, you may attach trace command to the running child process:
sudo strace -p $PID
This will show you what is being executed, either indefinite loop (like reading from a pipe), or waiting on some event that never happens.
In case you find ps --ppid $PID changes, this indicates that your script is advancing but it's stuck somewhere, e.g. local loop in the script. From the changing commands, it can give you a hint where in the script it's looping.
Finally, a very simple method to debug a perl is to use perl debugger:
perl -d script.pl
More: 1, 2, 3, 4, 5
edited 2 days ago
Jeff Schaller
33.3k849111
answered Sep 22 at 9:14
Goro
6,20152763
Why are you looking for child processes of the script?perldoesn't need to run child processes unless you use things likesystem()to execute external commands.
– Barmar
Sep 22 at 18:20
Did you misread the question to be asking about abashscript rather thanperl? Your Stack Overflow links suggest so.
– Barmar
Sep 22 at 18:22
Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
– Goro
Sep 22 at 18:48
1
The "More" links at the end are all about debugging bash, not perl.
– Barmar
Sep 22 at 19:39
Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit inwait()while waiting for it to finish. But you should add thestraceoption to traces forked children.
– Barmar
Sep 22 at 19:40
 |Â
show 1 more comment
up vote
10
down vote
accepted
Try to follow these steps:
- find the process pid of the shell, you may use a command like:
ps -ef | grep <your_script_name>
Let's set this pid in the shell variable $PID. Find all the child processes of this $PID by run the command:
ps --ppid $PID
You might find one or more (if for example it's stuck in a pipelined series of commands). Repeat this command couple of times. If it doesn't change this means the script is stuck in certain command. In this case, you may attach trace command to the running child process:
sudo strace -p $PID
This will show you what is being executed, either indefinite loop (like reading from a pipe), or waiting on some event that never happens.
In case you find ps --ppid $PID changes, this indicates that your script is advancing but it's stuck somewhere, e.g. local loop in the script. From the changing commands, it can give you a hint where in the script it's looping.
Finally, a very simple method to debug a perl is to use perl debugger:
perl -d script.pl
More: 1, 2, 3, 4, 5
edited 2 days ago
Jeff Schaller
33.3k849111
answered Sep 22 at 9:14
Goro
6,20152763
Why are you looking for child processes of the script?perldoesn't need to run child processes unless you use things likesystem()to execute external commands.
– Barmar
Sep 22 at 18:20
Did you misread the question to be asking about abashscript rather thanperl? Your Stack Overflow links suggest so.
– Barmar
Sep 22 at 18:22
Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
– Goro
Sep 22 at 18:48
1
The "More" links at the end are all about debugging bash, not perl.
– Barmar
Sep 22 at 19:39
Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit inwait()while waiting for it to finish. But you should add thestraceoption to traces forked children.
– Barmar
Sep 22 at 19:40
 |Â
show 1 more comment
up vote
10
down vote
accepted
up vote
10
down vote
accepted
Try to follow these steps:
- find the process pid of the shell, you may use a command like:
ps -ef | grep <your_script_name>
Let's set this pid in the shell variable $PID. Find all the child processes of this $PID by run the command:
ps --ppid $PID
You might find one or more (if for example it's stuck in a pipelined series of commands). Repeat this command couple of times. If it doesn't change this means the script is stuck in certain command. In this case, you may attach trace command to the running child process:
sudo strace -p $PID
This will show you what is being executed, either indefinite loop (like reading from a pipe), or waiting on some event that never happens.
In case you find ps --ppid $PID changes, this indicates that your script is advancing but it's stuck somewhere, e.g. local loop in the script. From the changing commands, it can give you a hint where in the script it's looping.
Finally, a very simple method to debug a perl is to use perl debugger:
perl -d script.pl
More: 1, 2, 3, 4, 5
edited 2 days ago
Jeff Schaller
33.3k849111
answered Sep 22 at 9:14
Goro
6,20152763
Try to follow these steps:
- find the process pid of the shell, you may use a command like:
ps -ef | grep <your_script_name>
Let's set this pid in the shell variable $PID. Find all the child processes of this $PID by run the command:
ps --ppid $PID
You might find one or more (if for example it's stuck in a pipelined series of commands). Repeat this command couple of times. If it doesn't change this means the script is stuck in certain command. In this case, you may attach trace command to the running child process:
sudo strace -p $PID
This will show you what is being executed, either indefinite loop (like reading from a pipe), or waiting on some event that never happens.
In case you find ps --ppid $PID changes, this indicates that your script is advancing but it's stuck somewhere, e.g. local loop in the script. From the changing commands, it can give you a hint where in the script it's looping.
Finally, a very simple method to debug a perl is to use perl debugger:
perl -d script.pl
More: 1, 2, 3, 4, 5
edited 2 days ago
Jeff Schaller
33.3k849111
answered Sep 22 at 9:14
Goro
6,20152763
edited 2 days ago
Jeff Schaller
33.3k849111
edited 2 days ago
Jeff Schaller
33.3k849111
edited 2 days ago
Jeff Schaller
33.3k849111
33.3k849111
answered Sep 22 at 9:14
Goro
6,20152763
answered Sep 22 at 9:14
Goro
6,20152763
answered Sep 22 at 9:14
Goro
6,20152763
6,20152763
Why are you looking for child processes of the script?perldoesn't need to run child processes unless you use things likesystem()to execute external commands.
– Barmar
Sep 22 at 18:20
Did you misread the question to be asking about abashscript rather thanperl? Your Stack Overflow links suggest so.
– Barmar
Sep 22 at 18:22
Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
– Goro
Sep 22 at 18:48
1
The "More" links at the end are all about debugging bash, not perl.
– Barmar
Sep 22 at 19:39
Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit inwait()while waiting for it to finish. But you should add thestraceoption to traces forked children.
– Barmar
Sep 22 at 19:40
 |Â
show 1 more comment
Why are you looking for child processes of the script?perldoesn't need to run child processes unless you use things likesystem()to execute external commands.
– Barmar
Sep 22 at 18:20
Did you misread the question to be asking about abashscript rather thanperl? Your Stack Overflow links suggest so.
– Barmar
Sep 22 at 18:22
Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
– Goro
Sep 22 at 18:48
1
The "More" links at the end are all about debugging bash, not perl.
– Barmar
Sep 22 at 19:39
Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit inwait()while waiting for it to finish. But you should add thestraceoption to traces forked children.
– Barmar
Sep 22 at 19:40
Why are you looking for child processes of the script?
perl doesn't need to run child processes unless you use things like system() to execute external commands.– Barmar
Sep 22 at 18:20
Why are you looking for child processes of the script?
perl doesn't need to run child processes unless you use things like system() to execute external commands.– Barmar
Sep 22 at 18:20
Did you misread the question to be asking about a
bash script rather than perl? Your Stack Overflow links suggest so.– Barmar
Sep 22 at 18:22
Did you misread the question to be asking about a
bash script rather than perl? Your Stack Overflow links suggest so.– Barmar
Sep 22 at 18:22
Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
– Goro
Sep 22 at 18:48
Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
– Goro
Sep 22 at 18:48
1
1
The "More" links at the end are all about debugging bash, not perl.
– Barmar
Sep 22 at 19:39
The "More" links at the end are all about debugging bash, not perl.
– Barmar
Sep 22 at 19:39
Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit in
wait() while waiting for it to finish. But you should add the strace option to traces forked children.– Barmar
Sep 22 at 19:40
Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit in
wait() while waiting for it to finish. But you should add the strace option to traces forked children.– Barmar
Sep 22 at 19:40
 |Â
show 1 more comment
up vote
3
down vote
For the next runs of your script you can try out the package Devel::Trace.
From description: "This module will print a message to standard error just before each line is executed."
Run either with
perl -d:Trace program
or use in your script with
import Devel::Trace 'trace';
trace 'on'; # Enable
trace 'off'; # Disable
answered Sep 22 at 9:17
Marvin
1464
But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably trystraceas in Goro's answer first.
– Henrik
Sep 22 at 9:59
@Henrik While debugging, you can runbash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$'to capture the output. And it doesn't matter much if many of generated files here are useless. Thestraceapproach is universal but the output there must be harder to match with the actual Perl code.
– Kirill Bulygin
Sep 22 at 12:39
add a comment |Â
up vote
3
down vote
For the next runs of your script you can try out the package Devel::Trace.
From description: "This module will print a message to standard error just before each line is executed."
Run either with
perl -d:Trace program
or use in your script with
import Devel::Trace 'trace';
trace 'on'; # Enable
trace 'off'; # Disable
answered Sep 22 at 9:17
Marvin
1464
But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably trystraceas in Goro's answer first.
– Henrik
Sep 22 at 9:59
@Henrik While debugging, you can runbash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$'to capture the output. And it doesn't matter much if many of generated files here are useless. Thestraceapproach is universal but the output there must be harder to match with the actual Perl code.
– Kirill Bulygin
Sep 22 at 12:39
add a comment |Â
up vote
3
down vote
up vote
3
down vote
For the next runs of your script you can try out the package Devel::Trace.
From description: "This module will print a message to standard error just before each line is executed."
Run either with
perl -d:Trace program
or use in your script with
import Devel::Trace 'trace';
trace 'on'; # Enable
trace 'off'; # Disable
answered Sep 22 at 9:17
Marvin
1464
For the next runs of your script you can try out the package Devel::Trace.
From description: "This module will print a message to standard error just before each line is executed."
Run either with
perl -d:Trace program
or use in your script with
import Devel::Trace 'trace';
trace 'on'; # Enable
trace 'off'; # Disable
answered Sep 22 at 9:17
Marvin
1464
answered Sep 22 at 9:17
Marvin
1464
answered Sep 22 at 9:17
Marvin
1464
answered Sep 22 at 9:17
Marvin
1464
1464
But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably trystraceas in Goro's answer first.
– Henrik
Sep 22 at 9:59
@Henrik While debugging, you can runbash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$'to capture the output. And it doesn't matter much if many of generated files here are useless. Thestraceapproach is universal but the output there must be harder to match with the actual Perl code.
– Kirill Bulygin
Sep 22 at 12:39
add a comment |Â
But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably trystraceas in Goro's answer first.
– Henrik
Sep 22 at 9:59
@Henrik While debugging, you can runbash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$'to capture the output. And it doesn't matter much if many of generated files here are useless. Thestraceapproach is universal but the output there must be harder to match with the actual Perl code.
– Kirill Bulygin
Sep 22 at 12:39
But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably try
strace as in Goro's answer first.– Henrik
Sep 22 at 9:59
But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably try
strace as in Goro's answer first.– Henrik
Sep 22 at 9:59
@Henrik While debugging, you can run
bash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$' to capture the output. And it doesn't matter much if many of generated files here are useless. The strace approach is universal but the output there must be harder to match with the actual Perl code.– Kirill Bulygin
Sep 22 at 12:39
@Henrik While debugging, you can run
bash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$' to capture the output. And it doesn't matter much if many of generated files here are useless. The strace approach is universal but the output there must be harder to match with the actual Perl code.– Kirill Bulygin
Sep 22 at 12:39
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f470681%2fperl-script-running-infinitely-how-to-debug-what-happened%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
