Inject string in tee output
Clash Royale CLAN TAG#URR8PPP
I am using pipelines to implement some functionality with bash on Centos 6.
During these pipes I want to export my data in a log file, but this data comes from different pipes in my pipeline and I want to differentiate between them, by injecting a tag before each log line. To be more precise I want to do the following:
COMMAND1 | (create a line in the log file like: "command1: " + output of the command) | COMMAND2 ...
After doing a lot of research, I reached the following point:
COMMAND1 | tee >(ifne echo -n "command1: " >> out.log) | tee -a out.log | COMMAND2
This worked but with a problem. The second tee writes first to the file and then writes the first tee. So I get:
(output of command 1)
command1:
instead of
command1: (output of command 1)
I suspect that ifne delays enough for the second tee to be executed before the first one. If I remove the ifne, it works fine. The problem is that I need the ifne, because I often have null output, that I don't want to log.
How can I make the second tee wait for the first one to finish?
bash shell-script text-processing tee
edited Mar 7 at 0:07
Gilles
545k12911071623
asked Mar 6 at 13:15
Stefanos KargasStefanos Kargas
1084
add a comment |
I am using pipelines to implement some functionality with bash on Centos 6.
During these pipes I want to export my data in a log file, but this data comes from different pipes in my pipeline and I want to differentiate between them, by injecting a tag before each log line. To be more precise I want to do the following:
COMMAND1 | (create a line in the log file like: "command1: " + output of the command) | COMMAND2 ...
After doing a lot of research, I reached the following point:
COMMAND1 | tee >(ifne echo -n "command1: " >> out.log) | tee -a out.log | COMMAND2
This worked but with a problem. The second tee writes first to the file and then writes the first tee. So I get:
(output of command 1)
command1:
instead of
command1: (output of command 1)
I suspect that ifne delays enough for the second tee to be executed before the first one. If I remove the ifne, it works fine. The problem is that I need the ifne, because I often have null output, that I don't want to log.
How can I make the second tee wait for the first one to finish?
bash shell-script text-processing tee
edited Mar 7 at 0:07
Gilles
545k12911071623
asked Mar 6 at 13:15
Stefanos KargasStefanos Kargas
1084
1
Commands in a pipeline are always started concurrently. If you don't want this, then you would have to not use a pipeline.
– Kusalananda♦
Mar 6 at 13:18
add a comment |
I am using pipelines to implement some functionality with bash on Centos 6.
During these pipes I want to export my data in a log file, but this data comes from different pipes in my pipeline and I want to differentiate between them, by injecting a tag before each log line. To be more precise I want to do the following:
COMMAND1 | (create a line in the log file like: "command1: " + output of the command) | COMMAND2 ...
After doing a lot of research, I reached the following point:
COMMAND1 | tee >(ifne echo -n "command1: " >> out.log) | tee -a out.log | COMMAND2
This worked but with a problem. The second tee writes first to the file and then writes the first tee. So I get:
(output of command 1)
command1:
instead of
command1: (output of command 1)
I suspect that ifne delays enough for the second tee to be executed before the first one. If I remove the ifne, it works fine. The problem is that I need the ifne, because I often have null output, that I don't want to log.
How can I make the second tee wait for the first one to finish?
bash shell-script text-processing tee
edited Mar 7 at 0:07
Gilles
545k12911071623
asked Mar 6 at 13:15
Stefanos KargasStefanos Kargas
1084
I am using pipelines to implement some functionality with bash on Centos 6.
During these pipes I want to export my data in a log file, but this data comes from different pipes in my pipeline and I want to differentiate between them, by injecting a tag before each log line. To be more precise I want to do the following:
COMMAND1 | (create a line in the log file like: "command1: " + output of the command) | COMMAND2 ...
After doing a lot of research, I reached the following point:
COMMAND1 | tee >(ifne echo -n "command1: " >> out.log) | tee -a out.log | COMMAND2
This worked but with a problem. The second tee writes first to the file and then writes the first tee. So I get:
(output of command 1)
command1:
instead of
command1: (output of command 1)
I suspect that ifne delays enough for the second tee to be executed before the first one. If I remove the ifne, it works fine. The problem is that I need the ifne, because I often have null output, that I don't want to log.
How can I make the second tee wait for the first one to finish?
bash shell-script text-processing tee
bash shell-script text-processing tee
edited Mar 7 at 0:07
Gilles
545k12911071623
asked Mar 6 at 13:15
Stefanos KargasStefanos Kargas
1084
edited Mar 7 at 0:07
Gilles
545k12911071623
asked Mar 6 at 13:15
Stefanos KargasStefanos Kargas
1084
edited Mar 7 at 0:07
Gilles
545k12911071623
edited Mar 7 at 0:07
Gilles
545k12911071623
edited Mar 7 at 0:07
Gilles
545k12911071623
545k12911071623
asked Mar 6 at 13:15
Stefanos KargasStefanos Kargas
1084
asked Mar 6 at 13:15
Stefanos KargasStefanos Kargas
1084
asked Mar 6 at 13:15
Stefanos KargasStefanos Kargas
1084
1084
1
Commands in a pipeline are always started concurrently. If you don't want this, then you would have to not use a pipeline.
– Kusalananda♦
Mar 6 at 13:18
add a comment |
1
Commands in a pipeline are always started concurrently. If you don't want this, then you would have to not use a pipeline.
– Kusalananda♦
Mar 6 at 13:18
1
1
Commands in a pipeline are always started concurrently. If you don't want this, then you would have to not use a pipeline.
– Kusalananda♦
Mar 6 at 13:18
Commands in a pipeline are always started concurrently. If you don't want this, then you would have to not use a pipeline.
– Kusalananda♦
Mar 6 at 13:18
add a comment |
1 Answer
1
active
oldest
votes
If I understand correctly, you want to log the output of COMMAND1 with a marker indicating that it comes from COMMAND1. Here's a way to do that:
COMMAND1 | tee >(sed 's/^/command1: /' >>out.log) | COMMAND2
Note that:
- This puts the
command1:marker at the beginning of each line, not just at the beginning of the first line, which I think is desirable but isn't what you seemed to be trying to do. command1must not contain any sed special characters (or else they need to be protected with a backslash).- Since
out.logis opened for appending, there's no risk that output will overlap even if multiple commands output to the same log file in parallel. - Since
out.logis only opened once in this snippet, all the lines will be in order. Now for the bad news: if you log to the same file multiple times in parallel, then there's no guarantee that the lines will appear in the desired order. For example,
echo hello | tee >(sed 's/^/command1: /' >>out.log) | tr a-z A-Z | tee >(sed 's/^/command2: /' >>out.log)may log
command2: HELLObeforecommand1: HELLO.- If you log to the same file multiple times in parallel, and the content of a line is too long, then lines from different instances may be interspersed. I think that common sed implementations will pass through lines that are less than
PIPE_BUFbytes long, but I'm not sure.
Given the ordering and atomicity limitations, I recommend logging to separate files, in which case you don't need the prefixes.
answered Mar 7 at 0:07
GillesGilles
545k12911071623
It works, but indeed it doesn't save the lines in the correct order. Also it seems as if the log of the first command sometimes holds until the second command finishes. In this case I have the log of the second command first and after killing the second command, I get the log of the first command. I might need to find another solution. Actually the first command provides me an input for the second command which creates an output. I want to be able to map input-output in the same log file.
– Stefanos Kargas
Mar 8 at 8:48
@StefanosKargas I still don't know what you define as “correct order”, but if your complaint is that the log of command1 appears later than you expect, turn off buffering.
– Gilles
Mar 8 at 9:10
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f504707%2finject-string-in-tee-output%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
If I understand correctly, you want to log the output of COMMAND1 with a marker indicating that it comes from COMMAND1. Here's a way to do that:
COMMAND1 | tee >(sed 's/^/command1: /' >>out.log) | COMMAND2
Note that:
- This puts the
command1:marker at the beginning of each line, not just at the beginning of the first line, which I think is desirable but isn't what you seemed to be trying to do. command1must not contain any sed special characters (or else they need to be protected with a backslash).- Since
out.logis opened for appending, there's no risk that output will overlap even if multiple commands output to the same log file in parallel. - Since
out.logis only opened once in this snippet, all the lines will be in order. Now for the bad news: if you log to the same file multiple times in parallel, then there's no guarantee that the lines will appear in the desired order. For example,
echo hello | tee >(sed 's/^/command1: /' >>out.log) | tr a-z A-Z | tee >(sed 's/^/command2: /' >>out.log)may log
command2: HELLObeforecommand1: HELLO.- If you log to the same file multiple times in parallel, and the content of a line is too long, then lines from different instances may be interspersed. I think that common sed implementations will pass through lines that are less than
PIPE_BUFbytes long, but I'm not sure.
Given the ordering and atomicity limitations, I recommend logging to separate files, in which case you don't need the prefixes.
answered Mar 7 at 0:07
GillesGilles
545k12911071623
It works, but indeed it doesn't save the lines in the correct order. Also it seems as if the log of the first command sometimes holds until the second command finishes. In this case I have the log of the second command first and after killing the second command, I get the log of the first command. I might need to find another solution. Actually the first command provides me an input for the second command which creates an output. I want to be able to map input-output in the same log file.
– Stefanos Kargas
Mar 8 at 8:48
@StefanosKargas I still don't know what you define as “correct order”, but if your complaint is that the log of command1 appears later than you expect, turn off buffering.
– Gilles
Mar 8 at 9:10
add a comment |
If I understand correctly, you want to log the output of COMMAND1 with a marker indicating that it comes from COMMAND1. Here's a way to do that:
COMMAND1 | tee >(sed 's/^/command1: /' >>out.log) | COMMAND2
Note that:
- This puts the
command1:marker at the beginning of each line, not just at the beginning of the first line, which I think is desirable but isn't what you seemed to be trying to do. command1must not contain any sed special characters (or else they need to be protected with a backslash).- Since
out.logis opened for appending, there's no risk that output will overlap even if multiple commands output to the same log file in parallel. - Since
out.logis only opened once in this snippet, all the lines will be in order. Now for the bad news: if you log to the same file multiple times in parallel, then there's no guarantee that the lines will appear in the desired order. For example,
echo hello | tee >(sed 's/^/command1: /' >>out.log) | tr a-z A-Z | tee >(sed 's/^/command2: /' >>out.log)may log
command2: HELLObeforecommand1: HELLO.- If you log to the same file multiple times in parallel, and the content of a line is too long, then lines from different instances may be interspersed. I think that common sed implementations will pass through lines that are less than
PIPE_BUFbytes long, but I'm not sure.
Given the ordering and atomicity limitations, I recommend logging to separate files, in which case you don't need the prefixes.
answered Mar 7 at 0:07
GillesGilles
545k12911071623
It works, but indeed it doesn't save the lines in the correct order. Also it seems as if the log of the first command sometimes holds until the second command finishes. In this case I have the log of the second command first and after killing the second command, I get the log of the first command. I might need to find another solution. Actually the first command provides me an input for the second command which creates an output. I want to be able to map input-output in the same log file.
– Stefanos Kargas
Mar 8 at 8:48
@StefanosKargas I still don't know what you define as “correct order”, but if your complaint is that the log of command1 appears later than you expect, turn off buffering.
– Gilles
Mar 8 at 9:10
add a comment |
If I understand correctly, you want to log the output of COMMAND1 with a marker indicating that it comes from COMMAND1. Here's a way to do that:
COMMAND1 | tee >(sed 's/^/command1: /' >>out.log) | COMMAND2
Note that:
- This puts the
command1:marker at the beginning of each line, not just at the beginning of the first line, which I think is desirable but isn't what you seemed to be trying to do. command1must not contain any sed special characters (or else they need to be protected with a backslash).- Since
out.logis opened for appending, there's no risk that output will overlap even if multiple commands output to the same log file in parallel. - Since
out.logis only opened once in this snippet, all the lines will be in order. Now for the bad news: if you log to the same file multiple times in parallel, then there's no guarantee that the lines will appear in the desired order. For example,
echo hello | tee >(sed 's/^/command1: /' >>out.log) | tr a-z A-Z | tee >(sed 's/^/command2: /' >>out.log)may log
command2: HELLObeforecommand1: HELLO.- If you log to the same file multiple times in parallel, and the content of a line is too long, then lines from different instances may be interspersed. I think that common sed implementations will pass through lines that are less than
PIPE_BUFbytes long, but I'm not sure.
Given the ordering and atomicity limitations, I recommend logging to separate files, in which case you don't need the prefixes.
answered Mar 7 at 0:07
GillesGilles
545k12911071623
If I understand correctly, you want to log the output of COMMAND1 with a marker indicating that it comes from COMMAND1. Here's a way to do that:
COMMAND1 | tee >(sed 's/^/command1: /' >>out.log) | COMMAND2
Note that:
- This puts the
command1:marker at the beginning of each line, not just at the beginning of the first line, which I think is desirable but isn't what you seemed to be trying to do. command1must not contain any sed special characters (or else they need to be protected with a backslash).- Since
out.logis opened for appending, there's no risk that output will overlap even if multiple commands output to the same log file in parallel. - Since
out.logis only opened once in this snippet, all the lines will be in order. Now for the bad news: if you log to the same file multiple times in parallel, then there's no guarantee that the lines will appear in the desired order. For example,
echo hello | tee >(sed 's/^/command1: /' >>out.log) | tr a-z A-Z | tee >(sed 's/^/command2: /' >>out.log)may log
command2: HELLObeforecommand1: HELLO.- If you log to the same file multiple times in parallel, and the content of a line is too long, then lines from different instances may be interspersed. I think that common sed implementations will pass through lines that are less than
PIPE_BUFbytes long, but I'm not sure.
Given the ordering and atomicity limitations, I recommend logging to separate files, in which case you don't need the prefixes.
answered Mar 7 at 0:07
GillesGilles
545k12911071623
edited Mar 7 at 7:55
answered Mar 7 at 0:07
GillesGilles
545k12911071623
answered Mar 7 at 0:07
GillesGilles
545k12911071623
answered Mar 7 at 0:07
GillesGilles
545k12911071623
545k12911071623
It works, but indeed it doesn't save the lines in the correct order. Also it seems as if the log of the first command sometimes holds until the second command finishes. In this case I have the log of the second command first and after killing the second command, I get the log of the first command. I might need to find another solution. Actually the first command provides me an input for the second command which creates an output. I want to be able to map input-output in the same log file.
– Stefanos Kargas
Mar 8 at 8:48
@StefanosKargas I still don't know what you define as “correct order”, but if your complaint is that the log of command1 appears later than you expect, turn off buffering.
– Gilles
Mar 8 at 9:10
add a comment |
It works, but indeed it doesn't save the lines in the correct order. Also it seems as if the log of the first command sometimes holds until the second command finishes. In this case I have the log of the second command first and after killing the second command, I get the log of the first command. I might need to find another solution. Actually the first command provides me an input for the second command which creates an output. I want to be able to map input-output in the same log file.
– Stefanos Kargas
Mar 8 at 8:48
@StefanosKargas I still don't know what you define as “correct order”, but if your complaint is that the log of command1 appears later than you expect, turn off buffering.
– Gilles
Mar 8 at 9:10
It works, but indeed it doesn't save the lines in the correct order. Also it seems as if the log of the first command sometimes holds until the second command finishes. In this case I have the log of the second command first and after killing the second command, I get the log of the first command. I might need to find another solution. Actually the first command provides me an input for the second command which creates an output. I want to be able to map input-output in the same log file.
– Stefanos Kargas
Mar 8 at 8:48
It works, but indeed it doesn't save the lines in the correct order. Also it seems as if the log of the first command sometimes holds until the second command finishes. In this case I have the log of the second command first and after killing the second command, I get the log of the first command. I might need to find another solution. Actually the first command provides me an input for the second command which creates an output. I want to be able to map input-output in the same log file.
– Stefanos Kargas
Mar 8 at 8:48
@StefanosKargas I still don't know what you define as “correct order”, but if your complaint is that the log of command1 appears later than you expect, turn off buffering.
– Gilles
Mar 8 at 9:10
@StefanosKargas I still don't know what you define as “correct order”, but if your complaint is that the log of command1 appears later than you expect, turn off buffering.
– Gilles
Mar 8 at 9:10
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f504707%2finject-string-in-tee-output%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Commands in a pipeline are always started concurrently. If you don't want this, then you would have to not use a pipeline.
– Kusalananda♦
Mar 6 at 13:18