Expect, Command, Pipes and Gzip

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
1
down vote

favorite












I am trying to work with the expect command, everything works well until I try to pipe the results of a command into a Gzip file. Here is the snippet of the bash file:



XYZ=$(expect -c "
spawn python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z | gzip >> /data/sources/results-$strt2-$end2.json.gz

expect "Password:"

send "$passr"

interact

")

echo "$XYZ"


$start - variable (start date)
$end - variable (end date)
$strt2 - variable (start date with 00:00)
$end2 - variable (end date with 23:59)
$pass - Password entered by the user previously.



If I remove



| gzip >> /data/sources/results-$strt2-$end2.json.gz


from the above it works as expected. When I add it I get the following error:



send: spawn id exp7 not open
while executing

usage: API [-h] [-s START] [-e END] [-d DAY] [-i INTERVAL]
[-a LOGGER] [-l LOGFILE] [-f FIELDS] [-q QUERY]
[-o OUTPUT] [-t FORMAT] [-u USER] [-p PASS] [-z]
API: error: unrecognized arguments: | gzip >> /data/sources/results-20180604-20180604.json.gz






share|improve this question

























    up vote
    1
    down vote

    favorite












    I am trying to work with the expect command, everything works well until I try to pipe the results of a command into a Gzip file. Here is the snippet of the bash file:



    XYZ=$(expect -c "
    spawn python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z | gzip >> /data/sources/results-$strt2-$end2.json.gz

    expect "Password:"

    send "$passr"

    interact

    ")

    echo "$XYZ"


    $start - variable (start date)
    $end - variable (end date)
    $strt2 - variable (start date with 00:00)
    $end2 - variable (end date with 23:59)
    $pass - Password entered by the user previously.



    If I remove



    | gzip >> /data/sources/results-$strt2-$end2.json.gz


    from the above it works as expected. When I add it I get the following error:



    send: spawn id exp7 not open
    while executing

    usage: API [-h] [-s START] [-e END] [-d DAY] [-i INTERVAL]
    [-a LOGGER] [-l LOGFILE] [-f FIELDS] [-q QUERY]
    [-o OUTPUT] [-t FORMAT] [-u USER] [-p PASS] [-z]
    API: error: unrecognized arguments: | gzip >> /data/sources/results-20180604-20180604.json.gz






    share|improve this question























      up vote
      1
      down vote

      favorite









      up vote
      1
      down vote

      favorite











      I am trying to work with the expect command, everything works well until I try to pipe the results of a command into a Gzip file. Here is the snippet of the bash file:



      XYZ=$(expect -c "
      spawn python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z | gzip >> /data/sources/results-$strt2-$end2.json.gz

      expect "Password:"

      send "$passr"

      interact

      ")

      echo "$XYZ"


      $start - variable (start date)
      $end - variable (end date)
      $strt2 - variable (start date with 00:00)
      $end2 - variable (end date with 23:59)
      $pass - Password entered by the user previously.



      If I remove



      | gzip >> /data/sources/results-$strt2-$end2.json.gz


      from the above it works as expected. When I add it I get the following error:



      send: spawn id exp7 not open
      while executing

      usage: API [-h] [-s START] [-e END] [-d DAY] [-i INTERVAL]
      [-a LOGGER] [-l LOGFILE] [-f FIELDS] [-q QUERY]
      [-o OUTPUT] [-t FORMAT] [-u USER] [-p PASS] [-z]
      API: error: unrecognized arguments: | gzip >> /data/sources/results-20180604-20180604.json.gz






      share|improve this question













      I am trying to work with the expect command, everything works well until I try to pipe the results of a command into a Gzip file. Here is the snippet of the bash file:



      XYZ=$(expect -c "
      spawn python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z | gzip >> /data/sources/results-$strt2-$end2.json.gz

      expect "Password:"

      send "$passr"

      interact

      ")

      echo "$XYZ"


      $start - variable (start date)
      $end - variable (end date)
      $strt2 - variable (start date with 00:00)
      $end2 - variable (end date with 23:59)
      $pass - Password entered by the user previously.



      If I remove



      | gzip >> /data/sources/results-$strt2-$end2.json.gz


      from the above it works as expected. When I add it I get the following error:



      send: spawn id exp7 not open
      while executing

      usage: API [-h] [-s START] [-e END] [-d DAY] [-i INTERVAL]
      [-a LOGGER] [-l LOGFILE] [-f FIELDS] [-q QUERY]
      [-o OUTPUT] [-t FORMAT] [-u USER] [-p PASS] [-z]
      API: error: unrecognized arguments: | gzip >> /data/sources/results-20180604-20180604.json.gz








      share|improve this question












      share|improve this question




      share|improve this question








      edited Jun 6 at 12:48
























      asked Jun 5 at 18:51









      ximian

      84




      84




















          1 Answer
          1






          active

          oldest

          votes

















          up vote
          2
          down vote



          accepted










          You might want to use a shell to interpret the pipe and redirection: I'll use a heredoc to make quoting easier



          XYZ=$(expect <<END_EXPECT
          set timeout -1
          spawn sh -c python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z
          expect "Password:"
          send "$passr"
          expect eof
          END_EXPECT
          )

          echo "$XYZ"





          share|improve this answer























          • Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
            – ximian
            Jun 6 at 12:18











          • It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
            – ximian
            Jun 6 at 12:48










          • If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
            – glenn jackman
            Jun 6 at 12:56











          • It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
            – ximian
            Jun 6 at 13:08










          • -1 means "do not time out". Put it before spawn
            – glenn jackman
            Jun 6 at 13:43










          Your Answer







          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "106"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          convertImagesToLinks: false,
          noModals: false,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );








           

          draft saved


          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f448053%2fexpect-command-pipes-and-gzip%23new-answer', 'question_page');

          );

          Post as a guest






























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes








          up vote
          2
          down vote



          accepted










          You might want to use a shell to interpret the pipe and redirection: I'll use a heredoc to make quoting easier



          XYZ=$(expect <<END_EXPECT
          set timeout -1
          spawn sh -c python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z
          expect "Password:"
          send "$passr"
          expect eof
          END_EXPECT
          )

          echo "$XYZ"





          share|improve this answer























          • Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
            – ximian
            Jun 6 at 12:18











          • It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
            – ximian
            Jun 6 at 12:48










          • If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
            – glenn jackman
            Jun 6 at 12:56











          • It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
            – ximian
            Jun 6 at 13:08










          • -1 means "do not time out". Put it before spawn
            – glenn jackman
            Jun 6 at 13:43














          up vote
          2
          down vote



          accepted










          You might want to use a shell to interpret the pipe and redirection: I'll use a heredoc to make quoting easier



          XYZ=$(expect <<END_EXPECT
          set timeout -1
          spawn sh -c python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z
          expect "Password:"
          send "$passr"
          expect eof
          END_EXPECT
          )

          echo "$XYZ"





          share|improve this answer























          • Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
            – ximian
            Jun 6 at 12:18











          • It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
            – ximian
            Jun 6 at 12:48










          • If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
            – glenn jackman
            Jun 6 at 12:56











          • It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
            – ximian
            Jun 6 at 13:08










          • -1 means "do not time out". Put it before spawn
            – glenn jackman
            Jun 6 at 13:43












          up vote
          2
          down vote



          accepted







          up vote
          2
          down vote



          accepted






          You might want to use a shell to interpret the pipe and redirection: I'll use a heredoc to make quoting easier



          XYZ=$(expect <<END_EXPECT
          set timeout -1
          spawn sh -c python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z
          expect "Password:"
          send "$passr"
          expect eof
          END_EXPECT
          )

          echo "$XYZ"





          share|improve this answer















          You might want to use a shell to interpret the pipe and redirection: I'll use a heredoc to make quoting easier



          XYZ=$(expect <<END_EXPECT
          set timeout -1
          spawn sh -c python log-connector.py -s $strt -e $end -i 600 -a https://server:9000 -u someaccount -f ./conf/firewall-fields2.txt -q ./conf/query.txt -z
          expect "Password:"
          send "$passr"
          expect eof
          END_EXPECT
          )

          echo "$XYZ"






          share|improve this answer















          share|improve this answer



          share|improve this answer








          edited Jun 6 at 21:21


























          answered Jun 5 at 19:04









          glenn jackman

          45.6k265100




          45.6k265100











          • Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
            – ximian
            Jun 6 at 12:18











          • It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
            – ximian
            Jun 6 at 12:48










          • If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
            – glenn jackman
            Jun 6 at 12:56











          • It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
            – ximian
            Jun 6 at 13:08










          • -1 means "do not time out". Put it before spawn
            – glenn jackman
            Jun 6 at 13:43
















          • Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
            – ximian
            Jun 6 at 12:18











          • It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
            – ximian
            Jun 6 at 12:48










          • If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
            – glenn jackman
            Jun 6 at 12:56











          • It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
            – ximian
            Jun 6 at 13:08










          • -1 means "do not time out". Put it before spawn
            – glenn jackman
            Jun 6 at 13:43















          Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
          – ximian
          Jun 6 at 12:18





          Well, I thought this did the trick and it sort of does, It runs the the python script once, gets the first chunk of a 10 min log, but does not continue getting the others. So its not really executing to completeness
          – ximian
          Jun 6 at 12:18













          It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
          – ximian
          Jun 6 at 12:48




          It should go until I get to the $end2 - variable (end date with 23:59), is there a way to check this in the expect ?
          – ximian
          Jun 6 at 12:48












          If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
          – glenn jackman
          Jun 6 at 12:56





          If it takes longer than 10 seconds then set timeout -1 at the top of the expect body
          – glenn jackman
          Jun 6 at 12:56













          It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
          – ximian
          Jun 6 at 13:08




          It can take 10 minutes like it could take a few hours for the logs to be parsed, so will the set timeout -1 handle this ? Also at the top of the expect body, you mean before XYZ=$.
          – ximian
          Jun 6 at 13:08












          -1 means "do not time out". Put it before spawn
          – glenn jackman
          Jun 6 at 13:43




          -1 means "do not time out". Put it before spawn
          – glenn jackman
          Jun 6 at 13:43












           

          draft saved


          draft discarded


























           


          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f448053%2fexpect-command-pipes-and-gzip%23new-answer', 'question_page');

          );

          Post as a guest













































































          Popular posts from this blog

          How to check contact read email or not when send email to Individual?

          Bahrain

          Postfix configuration issue with fips on centos 7; mailgun relay