Expect, Command, Pipes and Gzip

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
1
down vote

favorite












I am trying to work with the expect command, everything works well until I try to pipe the results of a command into a Gzip file. Here is the snippet of the bash file:



XYZ=$(expect -c "
spawn python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z | gzip >> /data/sources/results-$strt2-$end2.json.gz

expect "Password:"

send "$passr"

interact

")

echo "$XYZ"


$start - variable (start date)
$end - variable (end date)
$strt2 - variable (start date with 00:00)
$end2 - variable (end date with 23:59)
$pass - Password entered by the user previously.



If I remove



| gzip >> /data/sources/results-$strt2-$end2.json.gz


from the above it works as expected. When I add it I get the following error:



send: spawn id exp7 not open
while executing

usage: API [-h] [-s START] [-e END] [-d DAY] [-i INTERVAL]
[-a LOGGER] [-l LOGFILE] [-f FIELDS] [-q QUERY]
[-o OUTPUT] [-t FORMAT] [-u USER] [-p PASS] [-z]
API: error: unrecognized arguments: | gzip >> /data/sources/results-20180604-20180604.json.gz






share|improve this question

























    up vote
    1
    down vote

    favorite












    I am trying to work with the expect command, everything works well until I try to pipe the results of a command into a Gzip file. Here is the snippet of the bash file:



    XYZ=$(expect -c "
    spawn python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z | gzip >> /data/sources/results-$strt2-$end2.json.gz

    expect "Password:"

    send "$passr"

    interact

    ")

    echo "$XYZ"


    $start - variable (start date)
    $end - variable (end date)
    $strt2 - variable (start date with 00:00)
    $end2 - variable (end date with 23:59)
    $pass - Password entered by the user previously.



    If I remove



    | gzip >> /data/sources/results-$strt2-$end2.json.gz


    from the above it works as expected. When I add it I get the following error:



    send: spawn id exp7 not open
    while executing

    usage: API [-h] [-s START] [-e END] [-d DAY] [-i INTERVAL]
    [-a LOGGER] [-l LOGFILE] [-f FIELDS] [-q QUERY]
    [-o OUTPUT] [-t FORMAT] [-u USER] [-p PASS] [-z]
    API: error: unrecognized arguments: | gzip >> /data/sources/results-20180604-20180604.json.gz






    share|improve this question























      up vote
      1
      down vote

      favorite









      up vote
      1
      down vote

      favorite











      I am trying to work with the expect command, everything works well until I try to pipe the results of a command into a Gzip file. Here is the snippet of the bash file:



      XYZ=$(expect -c "
      spawn python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z | gzip >> /data/sources/results-$strt2-$end2.json.gz

      expect "Password:"

      send "$passr"

      interact

      ")

      echo "$XYZ"


      $start - variable (start date)
      $end - variable (end date)
      $strt2 - variable (start date with 00:00)
      $end2 - variable (end date with 23:59)
      $pass - Password entered by the user previously.



      If I remove



      | gzip >> /data/sources/results-$strt2-$end2.json.gz


      from the above it works as expected. When I add it I get the following error:



      send: spawn id exp7 not open
      while executing

      usage: API [-h] [-s START] [-e END] [-d DAY] [-i INTERVAL]
      [-a LOGGER] [-l LOGFILE] [-f FIELDS] [-q QUERY]
      [-o OUTPUT] [-t FORMAT] [-u USER] [-p PASS] [-z]
      API: error: unrecognized arguments: | gzip >> /data/sources/results-20180604-20180604.json.gz






      share|improve this question













      I am trying to work with the expect command, everything works well until I try to pipe the results of a command into a Gzip file. Here is the snippet of the bash file:



      XYZ=$(expect -c "
      spawn python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z | gzip >> /data/sources/results-$strt2-$end2.json.gz

      expect "Password:"

      send "$passr"

      interact

      ")

      echo "$XYZ"


      $start - variable (start date)
      $end - variable (end date)
      $strt2 - variable (start date with 00:00)
      $end2 - variable (end date with 23:59)
      $pass - Password entered by the user previously.



      If I remove



      | gzip >> /data/sources/results-$strt2-$end2.json.gz


      from the above it works as expected. When I add it I get the following error:



      send: spawn id exp7 not open
      while executing

      usage: API [-h] [-s START] [-e END] [-d DAY] [-i INTERVAL]
      [-a LOGGER] [-l LOGFILE] [-f FIELDS] [-q QUERY]
      [-o OUTPUT] [-t FORMAT] [-u USER] [-p PASS] [-z]
      API: error: unrecognized arguments: | gzip >> /data/sources/results-20180604-20180604.json.gz








      share|improve this question












      share|improve this question




      share|improve this question








      edited Jun 6 at 12:48
























      asked Jun 5 at 18:51









      ximian

      84




      84




















          1 Answer
          1






          active

          oldest

          votes

















          up vote
          2
          down vote



          accepted










          You might want to use a shell to interpret the pipe and redirection: I'll use a heredoc to make quoting easier



          XYZ=$(expect <<END_EXPECT
          set timeout -1
          spawn sh -c python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z
          expect "Password:"
          send "$passr"
          expect eof
          END_EXPECT
          )

          echo "$XYZ"





          share|improve this answer























          • Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
            – ximian
            Jun 6 at 12:18











          • It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
            – ximian
            Jun 6 at 12:48










          • If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
            – glenn jackman
            Jun 6 at 12:56











          • It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
            – ximian
            Jun 6 at 13:08










          • -1 means "do not time out". Put it before spawn
            – glenn jackman
            Jun 6 at 13:43










          Your Answer







          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "106"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          convertImagesToLinks: false,
          noModals: false,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );








           

          draft saved


          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f448053%2fexpect-command-pipes-and-gzip%23new-answer', 'question_page');

          );

          Post as a guest






























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes








          up vote
          2
          down vote



          accepted










          You might want to use a shell to interpret the pipe and redirection: I'll use a heredoc to make quoting easier



          XYZ=$(expect <<END_EXPECT
          set timeout -1
          spawn sh -c python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z
          expect "Password:"
          send "$passr"
          expect eof
          END_EXPECT
          )

          echo "$XYZ"





          share|improve this answer























          • Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
            – ximian
            Jun 6 at 12:18











          • It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
            – ximian
            Jun 6 at 12:48










          • If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
            – glenn jackman
            Jun 6 at 12:56











          • It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
            – ximian
            Jun 6 at 13:08










          • -1 means "do not time out". Put it before spawn
            – glenn jackman
            Jun 6 at 13:43














          up vote
          2
          down vote



          accepted










          You might want to use a shell to interpret the pipe and redirection: I'll use a heredoc to make quoting easier



          XYZ=$(expect <<END_EXPECT
          set timeout -1
          spawn sh -c python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z
          expect "Password:"
          send "$passr"
          expect eof
          END_EXPECT
          )

          echo "$XYZ"





          share|improve this answer























          • Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
            – ximian
            Jun 6 at 12:18











          • It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
            – ximian
            Jun 6 at 12:48










          • If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
            – glenn jackman
            Jun 6 at 12:56











          • It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
            – ximian
            Jun 6 at 13:08










          • -1 means "do not time out". Put it before spawn
            – glenn jackman
            Jun 6 at 13:43












          up vote
          2
          down vote



          accepted







          up vote
          2
          down vote



          accepted






          You might want to use a shell to interpret the pipe and redirection: I'll use a heredoc to make quoting easier



          XYZ=$(expect <<END_EXPECT
          set timeout -1
          spawn sh -c python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z
          expect "Password:"
          send "$passr"
          expect eof
          END_EXPECT
          )

          echo "$XYZ"





          share|improve this answer















          You might want to use a shell to interpret the pipe and redirection: I'll use a heredoc to make quoting easier



          XYZ=$(expect <<END_EXPECT
          set timeout -1
          spawn sh -c python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z
          expect "Password:"
          send "$passr"
          expect eof
          END_EXPECT
          )

          echo "$XYZ"






          share|improve this answer















          share|improve this answer



          share|improve this answer








          edited Jun 6 at 21:21


























          answered Jun 5 at 19:04









          glenn jackman

          45.6k265100




          45.6k265100











          • Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
            – ximian
            Jun 6 at 12:18











          • It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
            – ximian
            Jun 6 at 12:48










          • If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
            – glenn jackman
            Jun 6 at 12:56











          • It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
            – ximian
            Jun 6 at 13:08










          • -1 means "do not time out". Put it before spawn
            – glenn jackman
            Jun 6 at 13:43
















          • Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
            – ximian
            Jun 6 at 12:18











          • It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
            – ximian
            Jun 6 at 12:48










          • If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
            – glenn jackman
            Jun 6 at 12:56











          • It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
            – ximian
            Jun 6 at 13:08










          • -1 means "do not time out". Put it before spawn
            – glenn jackman
            Jun 6 at 13:43















          Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
          – ximian
          Jun 6 at 12:18





          Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
          – ximian
          Jun 6 at 12:18













          It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
          – ximian
          Jun 6 at 12:48




          It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
          – ximian
          Jun 6 at 12:48












          If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
          – glenn jackman
          Jun 6 at 12:56





          If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
          – glenn jackman
          Jun 6 at 12:56













          It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
          – ximian
          Jun 6 at 13:08




          It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
          – ximian
          Jun 6 at 13:08












          -1 means "do not time out". Put it before spawn
          – glenn jackman
          Jun 6 at 13:43




          -1 means "do not time out". Put it before spawn
          – glenn jackman
          Jun 6 at 13:43












           

          draft saved


          draft discarded


























           


          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f448053%2fexpect-command-pipes-and-gzip%23new-answer', 'question_page');

          );

          Post as a guest













































































          Popular posts from this blog

          How to check contact read email or not when send email to Individual?

          Christian Cage

          How to properly install USB display driver for Fresco Logic FL2000DX on Ubuntu?