mjhjmtu

We have our the logrotate config for a service as below:

Even though compress is not mentioned in the config, the log files are gzipped after each rotation. I believe it is because the compress line is uncommented in the /etc/logrotate.conf file, there by enabling it globally. The questions are:

1. Is there a time delay or interval between the log file is rotated,
   (from debug.log to debug.log-20190315) and when it is
   compressed (from debug.log-20190315 to debug.log-20190315.gz)?




2. If there is a delay, would mentioning compress in the specific log
   rotate config file of the service would compress that log file
   immediately after rotating it to debug.log-20190315 from debug.log ?

I do not see delaycompress mentioned in any of the logrotate config files.

(Background: Our Splunk Indexer seems to be indexing a debug.log-2019xxxx file from this service. We have blacklisted *.gz$ and debug.log$ from going to the Splunk, but it seems that the file debug.log-2019xxxx exists for a few seconds or minutes because of which it gets forwarded to Splunk since during that time it does not match the blacklisted regex - *.gz$ and debug.log$. I know that I can fix this by adding debug.log-[0-9]* to the blacklist, but would like to know what causes the existence of xxxxxx.debug.log-20190315)

The gzip compression step would take a finite amount of time, so it's possible that you're seeing the original file while the compression process is ongoing. If the size of the original file is large enough, it can exist for quite some time before the compression is completed. The time taken for the compression will vary according to the settings passed to gzip (speed & level of compression) as well as the compressibility of the file. gzip will remove the original file only after the compression process is complete.

As a small test for verification , I created a file of size 1 GiB from /dev/urandom and another file of size 1 GiB from /dev/zero and tested the time it took to compress them.

The file containing random data took about 2 minutes & 23 seconds:

[root@testvm1 ~]# time gzip testfile-random.txt

real 2m27.417s
user 2m22.172s
sys 0m2.839s

And the zero file took about 29 seconds:

[root@testvm1 ~]# time gzip testfile-zero.txt

real 0m28.930s
user 0m27.453s
sys 0m0.989s

While the compression was taking place, the original file was visible in both cases:

[root@testvm1 ~]# ls -lh testfile-random.txt*
-rw-r--r--. 1 root root 1.0G Mar 15 17:49 testfile-random.txt
-rw-------. 1 root root 75M Mar 15 17:59 testfile-random.txt.gz

[root@testvm1 ~]# ls -lh testfile-zero.txt*
-rw-r--r--. 1 root root 1.0G Mar 15 18:04 testfile-zero.txt
-rw-------. 1 root root 992K Mar 15 18:05 testfile-zero.txt.gz