Extract part of the logs to other file

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
0
down vote

favorite












1) We have a process.log file in which we have lot of text data and in between we have some XML data published.

2) There of thousands of different XML published in the logs along with other text data.

3) Now i need to select only the XML files which are published after Outgoing XML: value

4) Also the XML file which must be selected and copied to a new file should be the one which matches the value in the ALERTID tag.

5) The ALERTID value will be provided in the script input. So in our case mGMjhgHgffHhhFdH1u4 will be provided in the input and we need to select the full XML file published for this alertid. Starting tag is from <xml version..> and ending tag is </Alert>

5) So i need to select the relevant Outgoing XML file in a new file based on a particular ALERTID so it can be replayed in different environments.



Format of the log file is like below:



Info Jan 11 17:30:26.12122 The process is not responding to heartbeats
Debug Jan 11 17:30:26.12123 Incoming XML :<xml version "1.0" encoding ="UTF-8"?>
<Alert trigger = "true" >
<Alerttype>orderReject</Alerttype>
<AlertID>ghghfsjUtYuu78T1</AlertID>
<Order>uusingas</Order>
<Quantity>1254</Quanity>
</Alert> (CreateInitEventHandler. C:356)
Debug Jan 11 17:30:26.12199 The process is going down with warnings
Debug Jan 11 17:30:26.148199 Outgoing XML: <xml version "1.0" encoding ="UTF-8"?>
<Alert trigger = "true" >
<Alerttype>orderheld</Alerttype>
<AlertID>mGMjhgHgffHhhFdH1u4</AlertID>
<Order>uwiofhdf</Order>
<Quantity>7651</Quanity>
</Alert>(CreateEventHandler. C:723)
Debug Jan 11 17:30:26.13214 The process has restarted and thread opened
Debug Jan 11 17:30:26.13215 The heartbeat is recieved from alertlistener process


Now the requirement is to take AlertID in the input, scan the process log and extract the matching outgoing XML in a separate file.



Using awk i am able to extract all the outgoing xml files but not sure how to extract the one related to a particular AlertID.



Also i cannot install/use any new XML parser as per the company policy.
This needs to be achieved using shell/perl/awk/sed



Eg:



awk '/Outgoing/p=1; s=$0 P & & /</Alert>/ print $0 FS s; s="" ;p=0p' 1.log>2.log






share|improve this question


















  • 1




    there's no Outgoing word in your input - extend your input and post the expected result file contents
    – RomanPerekhrest
    Jan 15 at 16:34






  • 1




    wasted my time answering, this is a duplicate of unix.stackexchange.com/questions/416658/…
    – Gerard H. Pille
    Jan 15 at 17:06










  • Thanks Gerard for answering... Your script works and that's the answer I was looking for.. It wasn't answered in the other thread and was marked as duplicate of some other issue.. So posted again... But thanks a lot
    – abhishek chaudhry
    Jan 15 at 20:07














up vote
0
down vote

favorite












1) We have a process.log file in which we have lot of text data and in between we have some XML data published.

2) There of thousands of different XML published in the logs along with other text data.

3) Now i need to select only the XML files which are published after Outgoing XML: value

4) Also the XML file which must be selected and copied to a new file should be the one which matches the value in the ALERTID tag.

5) The ALERTID value will be provided in the script input. So in our case mGMjhgHgffHhhFdH1u4 will be provided in the input and we need to select the full XML file published for this alertid. Starting tag is from <xml version..> and ending tag is </Alert>

5) So i need to select the relevant Outgoing XML file in a new file based on a particular ALERTID so it can be replayed in different environments.



Format of the log file is like below:



Info Jan 11 17:30:26.12122 The process is not responding to heartbeats
Debug Jan 11 17:30:26.12123 Incoming XML :<xml version "1.0" encoding ="UTF-8"?>
<Alert trigger = "true" >
<Alerttype>orderReject</Alerttype>
<AlertID>ghghfsjUtYuu78T1</AlertID>
<Order>uusingas</Order>
<Quantity>1254</Quanity>
</Alert> (CreateInitEventHandler. C:356)
Debug Jan 11 17:30:26.12199 The process is going down with warnings
Debug Jan 11 17:30:26.148199 Outgoing XML: <xml version "1.0" encoding ="UTF-8"?>
<Alert trigger = "true" >
<Alerttype>orderheld</Alerttype>
<AlertID>mGMjhgHgffHhhFdH1u4</AlertID>
<Order>uwiofhdf</Order>
<Quantity>7651</Quanity>
</Alert>(CreateEventHandler. C:723)
Debug Jan 11 17:30:26.13214 The process has restarted and thread opened
Debug Jan 11 17:30:26.13215 The heartbeat is recieved from alertlistener process


Now the requirement is to take AlertID in the input, scan the process log and extract the matching outgoing XML in a separate file.



Using awk i am able to extract all the outgoing xml files but not sure how to extract the one related to a particular AlertID.



Also i cannot install/use any new XML parser as per the company policy.
This needs to be achieved using shell/perl/awk/sed



Eg:



awk '/Outgoing/p=1; s=$0 P & & /</Alert>/ print $0 FS s; s="" ;p=0p' 1.log>2.log






share|improve this question


















  • 1




    there's no Outgoing word in your input - extend your input and post the expected result file contents
    – RomanPerekhrest
    Jan 15 at 16:34






  • 1




    wasted my time answering, this is a duplicate of unix.stackexchange.com/questions/416658/…
    – Gerard H. Pille
    Jan 15 at 17:06










  • Thanks Gerard for answering... Your script works and that's the answer I was looking for.. It wasn't answered in the other thread and was marked as duplicate of some other issue.. So posted again... But thanks a lot
    – abhishek chaudhry
    Jan 15 at 20:07












up vote
0
down vote

favorite









up vote
0
down vote

favorite











1) We have a process.log file in which we have lot of text data and in between we have some XML data published.

2) There of thousands of different XML published in the logs along with other text data.

3) Now i need to select only the XML files which are published after Outgoing XML: value

4) Also the XML file which must be selected and copied to a new file should be the one which matches the value in the ALERTID tag.

5) The ALERTID value will be provided in the script input. So in our case mGMjhgHgffHhhFdH1u4 will be provided in the input and we need to select the full XML file published for this alertid. Starting tag is from <xml version..> and ending tag is </Alert>

5) So i need to select the relevant Outgoing XML file in a new file based on a particular ALERTID so it can be replayed in different environments.



Format of the log file is like below:



Info Jan 11 17:30:26.12122 The process is not responding to heartbeats
Debug Jan 11 17:30:26.12123 Incoming XML :<xml version "1.0" encoding ="UTF-8"?>
<Alert trigger = "true" >
<Alerttype>orderReject</Alerttype>
<AlertID>ghghfsjUtYuu78T1</AlertID>
<Order>uusingas</Order>
<Quantity>1254</Quanity>
</Alert> (CreateInitEventHandler. C:356)
Debug Jan 11 17:30:26.12199 The process is going down with warnings
Debug Jan 11 17:30:26.148199 Outgoing XML: <xml version "1.0" encoding ="UTF-8"?>
<Alert trigger = "true" >
<Alerttype>orderheld</Alerttype>
<AlertID>mGMjhgHgffHhhFdH1u4</AlertID>
<Order>uwiofhdf</Order>
<Quantity>7651</Quanity>
</Alert>(CreateEventHandler. C:723)
Debug Jan 11 17:30:26.13214 The process has restarted and thread opened
Debug Jan 11 17:30:26.13215 The heartbeat is recieved from alertlistener process


Now the requirement is to take AlertID in the input, scan the process log and extract the matching outgoing XML in a separate file.



Using awk i am able to extract all the outgoing xml files but not sure how to extract the one related to a particular AlertID.



Also i cannot install/use any new XML parser as per the company policy.
This needs to be achieved using shell/perl/awk/sed



Eg:



awk '/Outgoing/p=1; s=$0 P & & /</Alert>/ print $0 FS s; s="" ;p=0p' 1.log>2.log






share|improve this question














1) We have a process.log file in which we have lot of text data and in between we have some XML data published.

2) There of thousands of different XML published in the logs along with other text data.

3) Now i need to select only the XML files which are published after Outgoing XML: value

4) Also the XML file which must be selected and copied to a new file should be the one which matches the value in the ALERTID tag.

5) The ALERTID value will be provided in the script input. So in our case mGMjhgHgffHhhFdH1u4 will be provided in the input and we need to select the full XML file published for this alertid. Starting tag is from <xml version..> and ending tag is </Alert>

5) So i need to select the relevant Outgoing XML file in a new file based on a particular ALERTID so it can be replayed in different environments.



Format of the log file is like below:



Info Jan 11 17:30:26.12122 The process is not responding to heartbeats
Debug Jan 11 17:30:26.12123 Incoming XML :<xml version "1.0" encoding ="UTF-8"?>
<Alert trigger = "true" >
<Alerttype>orderReject</Alerttype>
<AlertID>ghghfsjUtYuu78T1</AlertID>
<Order>uusingas</Order>
<Quantity>1254</Quanity>
</Alert> (CreateInitEventHandler. C:356)
Debug Jan 11 17:30:26.12199 The process is going down with warnings
Debug Jan 11 17:30:26.148199 Outgoing XML: <xml version "1.0" encoding ="UTF-8"?>
<Alert trigger = "true" >
<Alerttype>orderheld</Alerttype>
<AlertID>mGMjhgHgffHhhFdH1u4</AlertID>
<Order>uwiofhdf</Order>
<Quantity>7651</Quanity>
</Alert>(CreateEventHandler. C:723)
Debug Jan 11 17:30:26.13214 The process has restarted and thread opened
Debug Jan 11 17:30:26.13215 The heartbeat is recieved from alertlistener process


Now the requirement is to take AlertID in the input, scan the process log and extract the matching outgoing XML in a separate file.



Using awk i am able to extract all the outgoing xml files but not sure how to extract the one related to a particular AlertID.



Also i cannot install/use any new XML parser as per the company policy.
This needs to be achieved using shell/perl/awk/sed



Eg:



awk '/Outgoing/p=1; s=$0 P & & /</Alert>/ print $0 FS s; s="" ;p=0p' 1.log>2.log








share|improve this question













share|improve this question




share|improve this question








edited Jan 15 at 16:59









Philippos

5,90211545




5,90211545










asked Jan 15 at 16:01









abhishek chaudhry

35




35







  • 1




    there's no Outgoing word in your input - extend your input and post the expected result file contents
    – RomanPerekhrest
    Jan 15 at 16:34






  • 1




    wasted my time answering, this is a duplicate of unix.stackexchange.com/questions/416658/…
    – Gerard H. Pille
    Jan 15 at 17:06










  • Thanks Gerard for answering... Your script works and that's the answer I was looking for.. It wasn't answered in the other thread and was marked as duplicate of some other issue.. So posted again... But thanks a lot
    – abhishek chaudhry
    Jan 15 at 20:07












  • 1




    there's no Outgoing word in your input - extend your input and post the expected result file contents
    – RomanPerekhrest
    Jan 15 at 16:34






  • 1




    wasted my time answering, this is a duplicate of unix.stackexchange.com/questions/416658/…
    – Gerard H. Pille
    Jan 15 at 17:06










  • Thanks Gerard for answering... Your script works and that's the answer I was looking for.. It wasn't answered in the other thread and was marked as duplicate of some other issue.. So posted again... But thanks a lot
    – abhishek chaudhry
    Jan 15 at 20:07







1




1




there's no Outgoing word in your input - extend your input and post the expected result file contents
– RomanPerekhrest
Jan 15 at 16:34




there's no Outgoing word in your input - extend your input and post the expected result file contents
– RomanPerekhrest
Jan 15 at 16:34




1




1




wasted my time answering, this is a duplicate of unix.stackexchange.com/questions/416658/…
– Gerard H. Pille
Jan 15 at 17:06




wasted my time answering, this is a duplicate of unix.stackexchange.com/questions/416658/…
– Gerard H. Pille
Jan 15 at 17:06












Thanks Gerard for answering... Your script works and that's the answer I was looking for.. It wasn't answered in the other thread and was marked as duplicate of some other issue.. So posted again... But thanks a lot
– abhishek chaudhry
Jan 15 at 20:07




Thanks Gerard for answering... Your script works and that's the answer I was looking for.. It wasn't answered in the other thread and was marked as duplicate of some other issue.. So posted again... But thanks a lot
– abhishek chaudhry
Jan 15 at 20:07










2 Answers
2






active

oldest

votes

















up vote
0
down vote



accepted










Supposing your ID is given in a variable called ALERTID:



sed -e '/Outgoing XML/!d;:a' -e '$d;N;s/.*(<xml version.*</Alert>).*/1/;Ta' -e "/$ALERTID/!d" yourfile.log


Explanation:




  • /Outgoing XML/!d;:a delete stuff until the Outgoing XML line and start a loop then


  • $d to delete an unfinished record at the end of the file


  • N;s/.*(<xml version.*</Alert>).*/1/;Ta appends lines until the </Alert> tag is found and removes everything before and after the desired block
    "/$ALERTID/!ddeletes blocks without the$ALERTID`

Maybe better to read:



sed '/Outgoing XML/!d;:a
$d;N
s/.*(<xml version.*</Alert>).*/1/;Ta
/'$ALERTID'/!d' yourfile.log





share|improve this answer






















  • Thanks a lot Phillippos... This solution works really well for me.. Have a great day ahead
    – abhishek chaudhry
    Jan 15 at 20:10

















up vote
0
down vote













create a shell script, getalert.sh, with following contents:



awk '
/^Debug .* Outgoing XML/
sub(/^.* Outgoing XML: /,"")
H=$0
LC=0
next

/</Alert>/
sub(/Alert>.*$/,"Alert>")
if (LC>0) print
LC=0
next


/<AlertID>'$1'</AlertID>/
print H
print
LC=1
next


/<AlertID>.*</AlertID>/
H=""
LC=0
next


if (LC > 0)
print
else
H = H $0

' $2


run it as



getalert.sh mGMjhgHgffHhhFdH1u4 process.log





share|improve this answer




















    Your Answer







    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "106"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    convertImagesToLinks: false,
    noModals: false,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );








     

    draft saved


    draft discarded


















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f417277%2fextract-part-of-the-logs-to-other-file%23new-answer', 'question_page');

    );

    Post as a guest






























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes








    up vote
    0
    down vote



    accepted










    Supposing your ID is given in a variable called ALERTID:



    sed -e '/Outgoing XML/!d;:a' -e '$d;N;s/.*(<xml version.*</Alert>).*/1/;Ta' -e "/$ALERTID/!d" yourfile.log


    Explanation:




    • /Outgoing XML/!d;:a delete stuff until the Outgoing XML line and start a loop then


    • $d to delete an unfinished record at the end of the file


    • N;s/.*(<xml version.*</Alert>).*/1/;Ta appends lines until the </Alert> tag is found and removes everything before and after the desired block
      "/$ALERTID/!ddeletes blocks without the$ALERTID`

    Maybe better to read:



    sed '/Outgoing XML/!d;:a
    $d;N
    s/.*(<xml version.*</Alert>).*/1/;Ta
    /'$ALERTID'/!d' yourfile.log





    share|improve this answer






















    • Thanks a lot Phillippos... This solution works really well for me.. Have a great day ahead
      – abhishek chaudhry
      Jan 15 at 20:10














    up vote
    0
    down vote



    accepted










    Supposing your ID is given in a variable called ALERTID:



    sed -e '/Outgoing XML/!d;:a' -e '$d;N;s/.*(<xml version.*</Alert>).*/1/;Ta' -e "/$ALERTID/!d" yourfile.log


    Explanation:




    • /Outgoing XML/!d;:a delete stuff until the Outgoing XML line and start a loop then


    • $d to delete an unfinished record at the end of the file


    • N;s/.*(<xml version.*</Alert>).*/1/;Ta appends lines until the </Alert> tag is found and removes everything before and after the desired block
      "/$ALERTID/!ddeletes blocks without the$ALERTID`

    Maybe better to read:



    sed '/Outgoing XML/!d;:a
    $d;N
    s/.*(<xml version.*</Alert>).*/1/;Ta
    /'$ALERTID'/!d' yourfile.log





    share|improve this answer






















    • Thanks a lot Phillippos... This solution works really well for me.. Have a great day ahead
      – abhishek chaudhry
      Jan 15 at 20:10












    up vote
    0
    down vote



    accepted







    up vote
    0
    down vote



    accepted






    Supposing your ID is given in a variable called ALERTID:



    sed -e '/Outgoing XML/!d;:a' -e '$d;N;s/.*(<xml version.*</Alert>).*/1/;Ta' -e "/$ALERTID/!d" yourfile.log


    Explanation:




    • /Outgoing XML/!d;:a delete stuff until the Outgoing XML line and start a loop then


    • $d to delete an unfinished record at the end of the file


    • N;s/.*(<xml version.*</Alert>).*/1/;Ta appends lines until the </Alert> tag is found and removes everything before and after the desired block
      "/$ALERTID/!ddeletes blocks without the$ALERTID`

    Maybe better to read:



    sed '/Outgoing XML/!d;:a
    $d;N
    s/.*(<xml version.*</Alert>).*/1/;Ta
    /'$ALERTID'/!d' yourfile.log





    share|improve this answer














    Supposing your ID is given in a variable called ALERTID:



    sed -e '/Outgoing XML/!d;:a' -e '$d;N;s/.*(<xml version.*</Alert>).*/1/;Ta' -e "/$ALERTID/!d" yourfile.log


    Explanation:




    • /Outgoing XML/!d;:a delete stuff until the Outgoing XML line and start a loop then


    • $d to delete an unfinished record at the end of the file


    • N;s/.*(<xml version.*</Alert>).*/1/;Ta appends lines until the </Alert> tag is found and removes everything before and after the desired block
      "/$ALERTID/!ddeletes blocks without the$ALERTID`

    Maybe better to read:



    sed '/Outgoing XML/!d;:a
    $d;N
    s/.*(<xml version.*</Alert>).*/1/;Ta
    /'$ALERTID'/!d' yourfile.log






    share|improve this answer














    share|improve this answer



    share|improve this answer








    edited Jan 15 at 17:20

























    answered Jan 15 at 17:13









    Philippos

    5,90211545




    5,90211545











    • Thanks a lot Phillippos... This solution works really well for me.. Have a great day ahead
      – abhishek chaudhry
      Jan 15 at 20:10
















    • Thanks a lot Phillippos... This solution works really well for me.. Have a great day ahead
      – abhishek chaudhry
      Jan 15 at 20:10















    Thanks a lot Phillippos... This solution works really well for me.. Have a great day ahead
    – abhishek chaudhry
    Jan 15 at 20:10




    Thanks a lot Phillippos... This solution works really well for me.. Have a great day ahead
    – abhishek chaudhry
    Jan 15 at 20:10












    up vote
    0
    down vote













    create a shell script, getalert.sh, with following contents:



    awk '
    /^Debug .* Outgoing XML/
    sub(/^.* Outgoing XML: /,"")
    H=$0
    LC=0
    next

    /</Alert>/
    sub(/Alert>.*$/,"Alert>")
    if (LC>0) print
    LC=0
    next


    /<AlertID>'$1'</AlertID>/
    print H
    print
    LC=1
    next


    /<AlertID>.*</AlertID>/
    H=""
    LC=0
    next


    if (LC > 0)
    print
    else
    H = H $0

    ' $2


    run it as



    getalert.sh mGMjhgHgffHhhFdH1u4 process.log





    share|improve this answer
























      up vote
      0
      down vote













      create a shell script, getalert.sh, with following contents:



      awk '
      /^Debug .* Outgoing XML/
      sub(/^.* Outgoing XML: /,"")
      H=$0
      LC=0
      next

      /</Alert>/
      sub(/Alert>.*$/,"Alert>")
      if (LC>0) print
      LC=0
      next


      /<AlertID>'$1'</AlertID>/
      print H
      print
      LC=1
      next


      /<AlertID>.*</AlertID>/
      H=""
      LC=0
      next


      if (LC > 0)
      print
      else
      H = H $0

      ' $2


      run it as



      getalert.sh mGMjhgHgffHhhFdH1u4 process.log





      share|improve this answer






















        up vote
        0
        down vote










        up vote
        0
        down vote









        create a shell script, getalert.sh, with following contents:



        awk '
        /^Debug .* Outgoing XML/
        sub(/^.* Outgoing XML: /,"")
        H=$0
        LC=0
        next

        /</Alert>/
        sub(/Alert>.*$/,"Alert>")
        if (LC>0) print
        LC=0
        next


        /<AlertID>'$1'</AlertID>/
        print H
        print
        LC=1
        next


        /<AlertID>.*</AlertID>/
        H=""
        LC=0
        next


        if (LC > 0)
        print
        else
        H = H $0

        ' $2


        run it as



        getalert.sh mGMjhgHgffHhhFdH1u4 process.log





        share|improve this answer












        create a shell script, getalert.sh, with following contents:



        awk '
        /^Debug .* Outgoing XML/
        sub(/^.* Outgoing XML: /,"")
        H=$0
        LC=0
        next

        /</Alert>/
        sub(/Alert>.*$/,"Alert>")
        if (LC>0) print
        LC=0
        next


        /<AlertID>'$1'</AlertID>/
        print H
        print
        LC=1
        next


        /<AlertID>.*</AlertID>/
        H=""
        LC=0
        next


        if (LC > 0)
        print
        else
        H = H $0

        ' $2


        run it as



        getalert.sh mGMjhgHgffHhhFdH1u4 process.log






        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Jan 15 at 16:51









        Gerard H. Pille

        1,219212




        1,219212






















             

            draft saved


            draft discarded


























             


            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f417277%2fextract-part-of-the-logs-to-other-file%23new-answer', 'question_page');

            );

            Post as a guest













































































            Popular posts from this blog

            Peggy Mitchell

            Palaiologos

            The Forum (Inglewood, California)