Perl script running infinitely - how to debug what happened?

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
6
down vote

favorite
1












I am running a Perl script on a Linux machine via a cron job. However, from time to time (around 1% of all cases), the script gets stuck and runs infinitely. If I list processes, I can see its PID. However, I don't want to kill it right away; I would rather know what went wrong.



Is there a way how to show what lines are being executed from the script? Something like step-by-step debugging of the script based on PID.










share|improve this question



























    up vote
    6
    down vote

    favorite
    1












    I am running a Perl script on a Linux machine via a cron job. However, from time to time (around 1% of all cases), the script gets stuck and runs infinitely. If I list processes, I can see its PID. However, I don't want to kill it right away; I would rather know what went wrong.



    Is there a way how to show what lines are being executed from the script? Something like step-by-step debugging of the script based on PID.










    share|improve this question

























      up vote
      6
      down vote

      favorite
      1









      up vote
      6
      down vote

      favorite
      1






      1





      I am running a Perl script on a Linux machine via a cron job. However, from time to time (around 1% of all cases), the script gets stuck and runs infinitely. If I list processes, I can see its PID. However, I don't want to kill it right away; I would rather know what went wrong.



      Is there a way how to show what lines are being executed from the script? Something like step-by-step debugging of the script based on PID.










      share|improve this question















      I am running a Perl script on a Linux machine via a cron job. However, from time to time (around 1% of all cases), the script gets stuck and runs infinitely. If I list processes, I can see its PID. However, I don't want to kill it right away; I would rather know what went wrong.



      Is there a way how to show what lines are being executed from the script? Something like step-by-step debugging of the script based on PID.







      linux perl






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Sep 22 at 20:50









      Jeff Schaller

      33.3k849111




      33.3k849111










      asked Sep 22 at 9:07









      Martin Perry

      1506




      1506




















          2 Answers
          2






          active

          oldest

          votes

















          up vote
          10
          down vote



          accepted










          Try to follow these steps:
          - find the process pid of the shell, you may use a command like:



           ps -ef | grep <your_script_name> 



          • Let's set this pid in the shell variable $PID. Find all the child processes of this $PID by run the command:



            ps --ppid $PID


          You might find one or more (if for example it's stuck in a pipelined series of commands). Repeat this command couple of times. If it doesn't change this means the script is stuck in certain command. In this case, you may attach trace command to the running child process:



           sudo strace -p $PID


          This will show you what is being executed, either indefinite loop (like reading from a pipe), or waiting on some event that never happens.



          In case you find ps --ppid $PID changes, this indicates that your script is advancing but it's stuck somewhere, e.g. local loop in the script. From the changing commands, it can give you a hint where in the script it's looping.



          Finally, a very simple method to debug a perl is to use perl debugger:



          perl -d script.pl


          More: 1, 2, 3, 4, 5






          share|improve this answer






















          • Why are you looking for child processes of the script? perl doesn't need to run child processes unless you use things like system() to execute external commands.
            – Barmar
            Sep 22 at 18:20










          • Did you misread the question to be asking about a bash script rather than perl? Your Stack Overflow links suggest so.
            – Barmar
            Sep 22 at 18:22











          • Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
            – Goro
            Sep 22 at 18:48







          • 1




            The "More" links at the end are all about debugging bash, not perl.
            – Barmar
            Sep 22 at 19:39










          • Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit in wait() while waiting for it to finish. But you should add the strace option to traces forked children.
            – Barmar
            Sep 22 at 19:40


















          up vote
          3
          down vote













          For the next runs of your script you can try out the package Devel::Trace.



          From description: "This module will print a message to standard error just before each line is executed."



          Run either with



          perl -d:Trace program


          or use in your script with



          import Devel::Trace 'trace';

          trace 'on'; # Enable
          trace 'off'; # Disable





          share|improve this answer




















          • But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably try strace as in Goro's answer first.
            – Henrik
            Sep 22 at 9:59










          • @Henrik While debugging, you can run bash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$' to capture the output. And it doesn't matter much if many of generated files here are useless. The strace approach is universal but the output there must be harder to match with the actual Perl code.
            – Kirill Bulygin
            Sep 22 at 12:39










          Your Answer







          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "106"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          convertImagesToLinks: false,
          noModals: false,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













           

          draft saved


          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f470681%2fperl-script-running-infinitely-how-to-debug-what-happened%23new-answer', 'question_page');

          );

          Post as a guest






























          2 Answers
          2






          active

          oldest

          votes








          2 Answers
          2






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes








          up vote
          10
          down vote



          accepted










          Try to follow these steps:
          - find the process pid of the shell, you may use a command like:



           ps -ef | grep <your_script_name> 



          • Let's set this pid in the shell variable $PID. Find all the child processes of this $PID by run the command:



            ps --ppid $PID


          You might find one or more (if for example it's stuck in a pipelined series of commands). Repeat this command couple of times. If it doesn't change this means the script is stuck in certain command. In this case, you may attach trace command to the running child process:



           sudo strace -p $PID


          This will show you what is being executed, either indefinite loop (like reading from a pipe), or waiting on some event that never happens.



          In case you find ps --ppid $PID changes, this indicates that your script is advancing but it's stuck somewhere, e.g. local loop in the script. From the changing commands, it can give you a hint where in the script it's looping.



          Finally, a very simple method to debug a perl is to use perl debugger:



          perl -d script.pl


          More: 1, 2, 3, 4, 5






          share|improve this answer






















          • Why are you looking for child processes of the script? perl doesn't need to run child processes unless you use things like system() to execute external commands.
            – Barmar
            Sep 22 at 18:20










          • Did you misread the question to be asking about a bash script rather than perl? Your Stack Overflow links suggest so.
            – Barmar
            Sep 22 at 18:22











          • Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
            – Goro
            Sep 22 at 18:48







          • 1




            The "More" links at the end are all about debugging bash, not perl.
            – Barmar
            Sep 22 at 19:39










          • Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit in wait() while waiting for it to finish. But you should add the strace option to traces forked children.
            – Barmar
            Sep 22 at 19:40















          up vote
          10
          down vote



          accepted










          Try to follow these steps:
          - find the process pid of the shell, you may use a command like:



           ps -ef | grep <your_script_name> 



          • Let's set this pid in the shell variable $PID. Find all the child processes of this $PID by run the command:



            ps --ppid $PID


          You might find one or more (if for example it's stuck in a pipelined series of commands). Repeat this command couple of times. If it doesn't change this means the script is stuck in certain command. In this case, you may attach trace command to the running child process:



           sudo strace -p $PID


          This will show you what is being executed, either indefinite loop (like reading from a pipe), or waiting on some event that never happens.



          In case you find ps --ppid $PID changes, this indicates that your script is advancing but it's stuck somewhere, e.g. local loop in the script. From the changing commands, it can give you a hint where in the script it's looping.



          Finally, a very simple method to debug a perl is to use perl debugger:



          perl -d script.pl


          More: 1, 2, 3, 4, 5






          share|improve this answer






















          • Why are you looking for child processes of the script? perl doesn't need to run child processes unless you use things like system() to execute external commands.
            – Barmar
            Sep 22 at 18:20










          • Did you misread the question to be asking about a bash script rather than perl? Your Stack Overflow links suggest so.
            – Barmar
            Sep 22 at 18:22











          • Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
            – Goro
            Sep 22 at 18:48







          • 1




            The "More" links at the end are all about debugging bash, not perl.
            – Barmar
            Sep 22 at 19:39










          • Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit in wait() while waiting for it to finish. But you should add the strace option to traces forked children.
            – Barmar
            Sep 22 at 19:40













          up vote
          10
          down vote



          accepted







          up vote
          10
          down vote



          accepted






          Try to follow these steps:
          - find the process pid of the shell, you may use a command like:



           ps -ef | grep <your_script_name> 



          • Let's set this pid in the shell variable $PID. Find all the child processes of this $PID by run the command:



            ps --ppid $PID


          You might find one or more (if for example it's stuck in a pipelined series of commands). Repeat this command couple of times. If it doesn't change this means the script is stuck in certain command. In this case, you may attach trace command to the running child process:



           sudo strace -p $PID


          This will show you what is being executed, either indefinite loop (like reading from a pipe), or waiting on some event that never happens.



          In case you find ps --ppid $PID changes, this indicates that your script is advancing but it's stuck somewhere, e.g. local loop in the script. From the changing commands, it can give you a hint where in the script it's looping.



          Finally, a very simple method to debug a perl is to use perl debugger:



          perl -d script.pl


          More: 1, 2, 3, 4, 5






          share|improve this answer














          Try to follow these steps:
          - find the process pid of the shell, you may use a command like:



           ps -ef | grep <your_script_name> 



          • Let's set this pid in the shell variable $PID. Find all the child processes of this $PID by run the command:



            ps --ppid $PID


          You might find one or more (if for example it's stuck in a pipelined series of commands). Repeat this command couple of times. If it doesn't change this means the script is stuck in certain command. In this case, you may attach trace command to the running child process:



           sudo strace -p $PID


          This will show you what is being executed, either indefinite loop (like reading from a pipe), or waiting on some event that never happens.



          In case you find ps --ppid $PID changes, this indicates that your script is advancing but it's stuck somewhere, e.g. local loop in the script. From the changing commands, it can give you a hint where in the script it's looping.



          Finally, a very simple method to debug a perl is to use perl debugger:



          perl -d script.pl


          More: 1, 2, 3, 4, 5







          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited 2 days ago









          Jeff Schaller

          33.3k849111




          33.3k849111










          answered Sep 22 at 9:14









          Goro

          6,20152763




          6,20152763











          • Why are you looking for child processes of the script? perl doesn't need to run child processes unless you use things like system() to execute external commands.
            – Barmar
            Sep 22 at 18:20










          • Did you misread the question to be asking about a bash script rather than perl? Your Stack Overflow links suggest so.
            – Barmar
            Sep 22 at 18:22











          • Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
            – Goro
            Sep 22 at 18:48







          • 1




            The "More" links at the end are all about debugging bash, not perl.
            – Barmar
            Sep 22 at 19:39










          • Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit in wait() while waiting for it to finish. But you should add the strace option to traces forked children.
            – Barmar
            Sep 22 at 19:40

















          • Why are you looking for child processes of the script? perl doesn't need to run child processes unless you use things like system() to execute external commands.
            – Barmar
            Sep 22 at 18:20










          • Did you misread the question to be asking about a bash script rather than perl? Your Stack Overflow links suggest so.
            – Barmar
            Sep 22 at 18:22











          • Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
            – Goro
            Sep 22 at 18:48







          • 1




            The "More" links at the end are all about debugging bash, not perl.
            – Barmar
            Sep 22 at 19:39










          • Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit in wait() while waiting for it to finish. But you should add the strace option to traces forked children.
            – Barmar
            Sep 22 at 19:40
















          Why are you looking for child processes of the script? perl doesn't need to run child processes unless you use things like system() to execute external commands.
          – Barmar
          Sep 22 at 18:20




          Why are you looking for child processes of the script? perl doesn't need to run child processes unless you use things like system() to execute external commands.
          – Barmar
          Sep 22 at 18:20












          Did you misread the question to be asking about a bash script rather than perl? Your Stack Overflow links suggest so.
          – Barmar
          Sep 22 at 18:22





          Did you misread the question to be asking about a bash script rather than perl? Your Stack Overflow links suggest so.
          – Barmar
          Sep 22 at 18:22













          Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
          – Goro
          Sep 22 at 18:48





          Hi @Barmer. Thanks ;-) Indeed, I assumed that the perl relies of something like system() because it is very rare for simple programs to hang out like the OP description. For the link I am not sure what do you mean!
          – Goro
          Sep 22 at 18:48





          1




          1




          The "More" links at the end are all about debugging bash, not perl.
          – Barmar
          Sep 22 at 19:39




          The "More" links at the end are all about debugging bash, not perl.
          – Barmar
          Sep 22 at 19:39












          Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit in wait() while waiting for it to finish. But you should add the strace option to traces forked children.
          – Barmar
          Sep 22 at 19:40





          Tracing the perl process itself would show you which processes it's exec'ing, and then you'll see it sit in wait() while waiting for it to finish. But you should add the strace option to traces forked children.
          – Barmar
          Sep 22 at 19:40













          up vote
          3
          down vote













          For the next runs of your script you can try out the package Devel::Trace.



          From description: "This module will print a message to standard error just before each line is executed."



          Run either with



          perl -d:Trace program


          or use in your script with



          import Devel::Trace 'trace';

          trace 'on'; # Enable
          trace 'off'; # Disable





          share|improve this answer




















          • But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably try strace as in Goro's answer first.
            – Henrik
            Sep 22 at 9:59










          • @Henrik While debugging, you can run bash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$' to capture the output. And it doesn't matter much if many of generated files here are useless. The strace approach is universal but the output there must be harder to match with the actual Perl code.
            – Kirill Bulygin
            Sep 22 at 12:39














          up vote
          3
          down vote













          For the next runs of your script you can try out the package Devel::Trace.



          From description: "This module will print a message to standard error just before each line is executed."



          Run either with



          perl -d:Trace program


          or use in your script with



          import Devel::Trace 'trace';

          trace 'on'; # Enable
          trace 'off'; # Disable





          share|improve this answer




















          • But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably try strace as in Goro's answer first.
            – Henrik
            Sep 22 at 9:59










          • @Henrik While debugging, you can run bash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$' to capture the output. And it doesn't matter much if many of generated files here are useless. The strace approach is universal but the output there must be harder to match with the actual Perl code.
            – Kirill Bulygin
            Sep 22 at 12:39












          up vote
          3
          down vote










          up vote
          3
          down vote









          For the next runs of your script you can try out the package Devel::Trace.



          From description: "This module will print a message to standard error just before each line is executed."



          Run either with



          perl -d:Trace program


          or use in your script with



          import Devel::Trace 'trace';

          trace 'on'; # Enable
          trace 'off'; # Disable





          share|improve this answer












          For the next runs of your script you can try out the package Devel::Trace.



          From description: "This module will print a message to standard error just before each line is executed."



          Run either with



          perl -d:Trace program


          or use in your script with



          import Devel::Trace 'trace';

          trace 'on'; # Enable
          trace 'off'; # Disable






          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Sep 22 at 9:17









          Marvin

          1464




          1464











          • But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably try strace as in Goro's answer first.
            – Henrik
            Sep 22 at 9:59










          • @Henrik While debugging, you can run bash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$' to capture the output. And it doesn't matter much if many of generated files here are useless. The strace approach is universal but the output there must be harder to match with the actual Perl code.
            – Kirill Bulygin
            Sep 22 at 12:39
















          • But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably try strace as in Goro's answer first.
            – Henrik
            Sep 22 at 9:59










          • @Henrik While debugging, you can run bash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$' to capture the output. And it doesn't matter much if many of generated files here are useless. The strace approach is universal but the output there must be harder to match with the actual Perl code.
            – Kirill Bulygin
            Sep 22 at 12:39















          But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably try strace as in Goro's answer first.
          – Henrik
          Sep 22 at 9:59




          But the output from the script will be captured by cron, so to actually see this you either need to look at the file descriptors used for that communication or convince cron that the script is done so it will send a mail with the output, hopefully killing it will be enough for that, but I've never tried. And as the OP said it was only about 1% of the invocations that didn't terminate in time, this could generate a lot of output before anything useful. I would probably try strace as in Goro's answer first.
          – Henrik
          Sep 22 at 9:59












          @Henrik While debugging, you can run bash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$' to capture the output. And it doesn't matter much if many of generated files here are useless. The strace approach is universal but the output there must be harder to match with the actual Perl code.
          – Kirill Bulygin
          Sep 22 at 12:39




          @Henrik While debugging, you can run bash -c 'perl -d:Trace program 2>&1 | tee /tmp/program.$$' to capture the output. And it doesn't matter much if many of generated files here are useless. The strace approach is universal but the output there must be harder to match with the actual Perl code.
          – Kirill Bulygin
          Sep 22 at 12:39

















           

          draft saved


          draft discarded















































           


          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f470681%2fperl-script-running-infinitely-how-to-debug-what-happened%23new-answer', 'question_page');

          );

          Post as a guest













































































          Popular posts from this blog

          How to check contact read email or not when send email to Individual?

          Displaying single band from multi-band raster using QGIS

          How many registers does an x86_64 CPU actually have?