Why do âjournalctl --disk-usageâ and âjournalctl | wcâ not correspond well?

Clash Royale CLAN TAG#URR8PPP
up vote
1
down vote
favorite
TL;DR how comes that journald stores about 770 MiB while displaying the logs only yields 90MiB of data?
#UPDATE#
I did a journalctl -a -o verbose which at least yielded some 530MiB, yet less than the 770MiB the journal uses on disk. Anyway I would have expected that the journal on disk is compressed in some sort, and should be less than the 530MiB and not bigger instead. what is going on?
#UPDATE2#
It seems that journald has an tendency to create mostly empty journal files.
$> cat .system@0000.journal~ | wc -c
8388608
#> cat .system@0000.journal~ | sed 's/x00//g' | wc -c
47181
which might be what the comment by @Mioriin hints at. Is there any reason why there are 8MiB , but almost empy journal files kept by journald?
long version with background
My system is quite chatty, with linux kernel dmesg output, gnome DE, etc. - filling up my /var/log directory via the system logger journald that I sadly use (being still on a standard systemd arch linux).
I looked via du -hs /var/log and found roughly 2GiB with most being of journald journals. I removed the data being older than 90 days via journalctl --vacuum-time=90d and alas with sacrfificing logs older than ~3 month reduced the disk space used for the journald logs to only about 770MiB as checked via
$> journalctl --disk-usage
Archived and active journals take up 768.0M in the file system.
which is great.
However 770MiB of data for 90 days on my arch linux box seems quite a bit so I tried investigating to see what for the 770MiB are used for. I did the imho first logical step to measure initial size of the log output via
$> journalctl | wc
584467 7662317 90227741
and to double check did
$ journalctl | dd of=/dev/null iflag=fullblock bs=1M
86+1 records in
86+1 records out
90244229 bytes (90 MB, 86 MiB) copied, 9.21893 s, 9.8 MB/s
showing that for some reason, reading the "binary-because-saves-space" journald in plain text is not yielding some < 770MiB of data but instead is only 90MiB.
Now my question is about why those two figures do not match? How can my beloved journald, end up using 770MiB for essentially storing 90MiB of data?
What, next to "the meauring and being confused about the result", have I done to help myself?
I have skimmed the lengthy man journactl but am clearly at a loss with the considerable intricacy and detail it offers, yet could not find an hint as to what might cause the counter-intuitive thing that the binary-compressed-journal-data uses more space than the actuall plain text reproduced log.
I guess there should be an "easy", explanation.
systemd-journald
 |Â
show 1 more comment
up vote
1
down vote
favorite
TL;DR how comes that journald stores about 770 MiB while displaying the logs only yields 90MiB of data?
#UPDATE#
I did a journalctl -a -o verbose which at least yielded some 530MiB, yet less than the 770MiB the journal uses on disk. Anyway I would have expected that the journal on disk is compressed in some sort, and should be less than the 530MiB and not bigger instead. what is going on?
#UPDATE2#
It seems that journald has an tendency to create mostly empty journal files.
$> cat .system@0000.journal~ | wc -c
8388608
#> cat .system@0000.journal~ | sed 's/x00//g' | wc -c
47181
which might be what the comment by @Mioriin hints at. Is there any reason why there are 8MiB , but almost empy journal files kept by journald?
long version with background
My system is quite chatty, with linux kernel dmesg output, gnome DE, etc. - filling up my /var/log directory via the system logger journald that I sadly use (being still on a standard systemd arch linux).
I looked via du -hs /var/log and found roughly 2GiB with most being of journald journals. I removed the data being older than 90 days via journalctl --vacuum-time=90d and alas with sacrfificing logs older than ~3 month reduced the disk space used for the journald logs to only about 770MiB as checked via
$> journalctl --disk-usage
Archived and active journals take up 768.0M in the file system.
which is great.
However 770MiB of data for 90 days on my arch linux box seems quite a bit so I tried investigating to see what for the 770MiB are used for. I did the imho first logical step to measure initial size of the log output via
$> journalctl | wc
584467 7662317 90227741
and to double check did
$ journalctl | dd of=/dev/null iflag=fullblock bs=1M
86+1 records in
86+1 records out
90244229 bytes (90 MB, 86 MiB) copied, 9.21893 s, 9.8 MB/s
showing that for some reason, reading the "binary-because-saves-space" journald in plain text is not yielding some < 770MiB of data but instead is only 90MiB.
Now my question is about why those two figures do not match? How can my beloved journald, end up using 770MiB for essentially storing 90MiB of data?
What, next to "the meauring and being confused about the result", have I done to help myself?
I have skimmed the lengthy man journactl but am clearly at a loss with the considerable intricacy and detail it offers, yet could not find an hint as to what might cause the counter-intuitive thing that the binary-compressed-journal-data uses more space than the actuall plain text reproduced log.
I guess there should be an "easy", explanation.
systemd-journald
serverfault.com/questions/898951/â¦
â don_crissti
Aug 13 at 10:29
@don_crissti thanks for pointing to a similar question on serverfault. I checked it and the provided answer, while useful in the investigation, do not provide an answer. It seems thatjournaldwhen allocating disk spaces uses multiple of 8MiB sized files, even when only (as in one case of mine) 56kiB ofjournal -o verboseis stored stored in this file.
â humanityANDpeace
Aug 13 at 11:23
Have a look inside/var/log/journal. That should clarify things a bit for you. ;)
â Mioriin
Aug 13 at 11:57
@Mioriin do you imply this github.com/systemd/systemd/issues/5285 issue? what do you mean.
â humanityANDpeace
Aug 13 at 11:58
journalctlby default only loads the current journal file and not all the archived journals to save on resources. If it were to load all 90 days' worth of logs, you'd be waiting quite a while for anything to show up. To verify that it's actually using ~770MiB, trydu -h /var/log/journal.
â Mioriin
Aug 13 at 14:09
 |Â
show 1 more comment
up vote
1
down vote
favorite
up vote
1
down vote
favorite
TL;DR how comes that journald stores about 770 MiB while displaying the logs only yields 90MiB of data?
#UPDATE#
I did a journalctl -a -o verbose which at least yielded some 530MiB, yet less than the 770MiB the journal uses on disk. Anyway I would have expected that the journal on disk is compressed in some sort, and should be less than the 530MiB and not bigger instead. what is going on?
#UPDATE2#
It seems that journald has an tendency to create mostly empty journal files.
$> cat .system@0000.journal~ | wc -c
8388608
#> cat .system@0000.journal~ | sed 's/x00//g' | wc -c
47181
which might be what the comment by @Mioriin hints at. Is there any reason why there are 8MiB , but almost empy journal files kept by journald?
long version with background
My system is quite chatty, with linux kernel dmesg output, gnome DE, etc. - filling up my /var/log directory via the system logger journald that I sadly use (being still on a standard systemd arch linux).
I looked via du -hs /var/log and found roughly 2GiB with most being of journald journals. I removed the data being older than 90 days via journalctl --vacuum-time=90d and alas with sacrfificing logs older than ~3 month reduced the disk space used for the journald logs to only about 770MiB as checked via
$> journalctl --disk-usage
Archived and active journals take up 768.0M in the file system.
which is great.
However 770MiB of data for 90 days on my arch linux box seems quite a bit so I tried investigating to see what for the 770MiB are used for. I did the imho first logical step to measure initial size of the log output via
$> journalctl | wc
584467 7662317 90227741
and to double check did
$ journalctl | dd of=/dev/null iflag=fullblock bs=1M
86+1 records in
86+1 records out
90244229 bytes (90 MB, 86 MiB) copied, 9.21893 s, 9.8 MB/s
showing that for some reason, reading the "binary-because-saves-space" journald in plain text is not yielding some < 770MiB of data but instead is only 90MiB.
Now my question is about why those two figures do not match? How can my beloved journald, end up using 770MiB for essentially storing 90MiB of data?
What, next to "the meauring and being confused about the result", have I done to help myself?
I have skimmed the lengthy man journactl but am clearly at a loss with the considerable intricacy and detail it offers, yet could not find an hint as to what might cause the counter-intuitive thing that the binary-compressed-journal-data uses more space than the actuall plain text reproduced log.
I guess there should be an "easy", explanation.
systemd-journald
TL;DR how comes that journald stores about 770 MiB while displaying the logs only yields 90MiB of data?
#UPDATE#
I did a journalctl -a -o verbose which at least yielded some 530MiB, yet less than the 770MiB the journal uses on disk. Anyway I would have expected that the journal on disk is compressed in some sort, and should be less than the 530MiB and not bigger instead. what is going on?
#UPDATE2#
It seems that journald has an tendency to create mostly empty journal files.
$> cat .system@0000.journal~ | wc -c
8388608
#> cat .system@0000.journal~ | sed 's/x00//g' | wc -c
47181
which might be what the comment by @Mioriin hints at. Is there any reason why there are 8MiB , but almost empy journal files kept by journald?
long version with background
My system is quite chatty, with linux kernel dmesg output, gnome DE, etc. - filling up my /var/log directory via the system logger journald that I sadly use (being still on a standard systemd arch linux).
I looked via du -hs /var/log and found roughly 2GiB with most being of journald journals. I removed the data being older than 90 days via journalctl --vacuum-time=90d and alas with sacrfificing logs older than ~3 month reduced the disk space used for the journald logs to only about 770MiB as checked via
$> journalctl --disk-usage
Archived and active journals take up 768.0M in the file system.
which is great.
However 770MiB of data for 90 days on my arch linux box seems quite a bit so I tried investigating to see what for the 770MiB are used for. I did the imho first logical step to measure initial size of the log output via
$> journalctl | wc
584467 7662317 90227741
and to double check did
$ journalctl | dd of=/dev/null iflag=fullblock bs=1M
86+1 records in
86+1 records out
90244229 bytes (90 MB, 86 MiB) copied, 9.21893 s, 9.8 MB/s
showing that for some reason, reading the "binary-because-saves-space" journald in plain text is not yielding some < 770MiB of data but instead is only 90MiB.
Now my question is about why those two figures do not match? How can my beloved journald, end up using 770MiB for essentially storing 90MiB of data?
What, next to "the meauring and being confused about the result", have I done to help myself?
I have skimmed the lengthy man journactl but am clearly at a loss with the considerable intricacy and detail it offers, yet could not find an hint as to what might cause the counter-intuitive thing that the binary-compressed-journal-data uses more space than the actuall plain text reproduced log.
I guess there should be an "easy", explanation.
systemd-journald
systemd-journald
edited Aug 13 at 12:06
asked Aug 13 at 10:08
humanityANDpeace
4,59743349
4,59743349
serverfault.com/questions/898951/â¦
â don_crissti
Aug 13 at 10:29
@don_crissti thanks for pointing to a similar question on serverfault. I checked it and the provided answer, while useful in the investigation, do not provide an answer. It seems thatjournaldwhen allocating disk spaces uses multiple of 8MiB sized files, even when only (as in one case of mine) 56kiB ofjournal -o verboseis stored stored in this file.
â humanityANDpeace
Aug 13 at 11:23
Have a look inside/var/log/journal. That should clarify things a bit for you. ;)
â Mioriin
Aug 13 at 11:57
@Mioriin do you imply this github.com/systemd/systemd/issues/5285 issue? what do you mean.
â humanityANDpeace
Aug 13 at 11:58
journalctlby default only loads the current journal file and not all the archived journals to save on resources. If it were to load all 90 days' worth of logs, you'd be waiting quite a while for anything to show up. To verify that it's actually using ~770MiB, trydu -h /var/log/journal.
â Mioriin
Aug 13 at 14:09
 |Â
show 1 more comment
serverfault.com/questions/898951/â¦
â don_crissti
Aug 13 at 10:29
@don_crissti thanks for pointing to a similar question on serverfault. I checked it and the provided answer, while useful in the investigation, do not provide an answer. It seems thatjournaldwhen allocating disk spaces uses multiple of 8MiB sized files, even when only (as in one case of mine) 56kiB ofjournal -o verboseis stored stored in this file.
â humanityANDpeace
Aug 13 at 11:23
Have a look inside/var/log/journal. That should clarify things a bit for you. ;)
â Mioriin
Aug 13 at 11:57
@Mioriin do you imply this github.com/systemd/systemd/issues/5285 issue? what do you mean.
â humanityANDpeace
Aug 13 at 11:58
journalctlby default only loads the current journal file and not all the archived journals to save on resources. If it were to load all 90 days' worth of logs, you'd be waiting quite a while for anything to show up. To verify that it's actually using ~770MiB, trydu -h /var/log/journal.
â Mioriin
Aug 13 at 14:09
serverfault.com/questions/898951/â¦
â don_crissti
Aug 13 at 10:29
serverfault.com/questions/898951/â¦
â don_crissti
Aug 13 at 10:29
@don_crissti thanks for pointing to a similar question on serverfault. I checked it and the provided answer, while useful in the investigation, do not provide an answer. It seems that
journald when allocating disk spaces uses multiple of 8MiB sized files, even when only (as in one case of mine) 56kiB of journal -o verbose is stored stored in this file.â humanityANDpeace
Aug 13 at 11:23
@don_crissti thanks for pointing to a similar question on serverfault. I checked it and the provided answer, while useful in the investigation, do not provide an answer. It seems that
journald when allocating disk spaces uses multiple of 8MiB sized files, even when only (as in one case of mine) 56kiB of journal -o verbose is stored stored in this file.â humanityANDpeace
Aug 13 at 11:23
Have a look inside
/var/log/journal. That should clarify things a bit for you. ;)â Mioriin
Aug 13 at 11:57
Have a look inside
/var/log/journal. That should clarify things a bit for you. ;)â Mioriin
Aug 13 at 11:57
@Mioriin do you imply this github.com/systemd/systemd/issues/5285 issue? what do you mean.
â humanityANDpeace
Aug 13 at 11:58
@Mioriin do you imply this github.com/systemd/systemd/issues/5285 issue? what do you mean.
â humanityANDpeace
Aug 13 at 11:58
journalctl by default only loads the current journal file and not all the archived journals to save on resources. If it were to load all 90 days' worth of logs, you'd be waiting quite a while for anything to show up. To verify that it's actually using ~770MiB, try du -h /var/log/journal.â Mioriin
Aug 13 at 14:09
journalctl by default only loads the current journal file and not all the archived journals to save on resources. If it were to load all 90 days' worth of logs, you'd be waiting quite a while for anything to show up. To verify that it's actually using ~770MiB, try du -h /var/log/journal.â Mioriin
Aug 13 at 14:09
 |Â
show 1 more comment
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f462266%2fwhy-do-journalctl-disk-usage-and-journalctl-wc-not-correspond-well%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
serverfault.com/questions/898951/â¦
â don_crissti
Aug 13 at 10:29
@don_crissti thanks for pointing to a similar question on serverfault. I checked it and the provided answer, while useful in the investigation, do not provide an answer. It seems that
journaldwhen allocating disk spaces uses multiple of 8MiB sized files, even when only (as in one case of mine) 56kiB ofjournal -o verboseis stored stored in this file.â humanityANDpeace
Aug 13 at 11:23
Have a look inside
/var/log/journal. That should clarify things a bit for you. ;)â Mioriin
Aug 13 at 11:57
@Mioriin do you imply this github.com/systemd/systemd/issues/5285 issue? what do you mean.
â humanityANDpeace
Aug 13 at 11:58
journalctlby default only loads the current journal file and not all the archived journals to save on resources. If it were to load all 90 days' worth of logs, you'd be waiting quite a while for anything to show up. To verify that it's actually using ~770MiB, trydu -h /var/log/journal.â Mioriin
Aug 13 at 14:09