mjhjmtu

I have several linux systems running various homemade applications, some web apis, some background data crunching, multiple databases and things I haven't even found yet. These systems were set up by multiple people over several months, and no one really knows how each server is used. Some of the people setting up programs on these were interns or contractors who have since departed. So we don't know how anything is configured.

I'm trying to figure out what log files are being written, especially those being written in weird places (i.e. not under /var/log). I'm also trying to find all log files not being rotated, especially if they are growing rapidly. I found two servers in the past week running up against their disk limit and crashing processes. My goal is to reconfigure each application to do something sane for logging, and eventually send it all to an ELK stack, but for now I just need to figure out what I've got.

So to start I'm trying to find everything that's a log file. That's hard if they're scattered randomly in the system. Some under /home/someuser, one was under /root, some in /tmp and one in /var/lib.

My first thought to find log files was to find any file modified recently.
See this answer: https://askubuntu.com/a/704163/139584

This gets me a lot of noise though. Databases persist things to disk, so they write files, system updates replace binaries so those are modified, and users have modified stuff in their homes.

My next thought was to find by name. Most log files end with .log, but some do not. Maybe some have "log" somewhere in the pathname. See this answer: https://askubuntu.com/a/144703/139584

Once I have a list of logs, I can scan the logrotate rules to find anything matching. That should be easy enough with for and grep.

Does anyone have a better idea of how to enumerate stray log files in an undocumented linux system?

I found this to be an interesting problem, for several reasons:

• I might run into a similar system, and need to get a handle on file / filesystem growth


• Gathering the list of "local" filesystems is not simple


• Determining growth requires multiple checks, with a time delay in-between


• Generically excluding "user home directories" required some care

I've come up with a script that, at a high level, uses find to look for non-executable files that have been modified in the past 7 days; it then sleeps for a minute and then re-scans those files to see if any of them have grown more than 42 bytes.

Of course, all of the arbitrary numbers are freely editable in your own copy of the script:

• amount of time to sleep (wait for log file growth)


• amount of growth to alert on


• how recently-modified the files should be

I gather the list of local filesystems using lsblk, asking it to produce a list, without headings, of only the mount points; because that output includes block devices that aren't necessarily mounted (e.g. whole disks, swap areas, etc), I then filter for mountpoints that contain a /.

Excluding home directories seemed like a good idea, but I didn't want to assume that every home directory was under /home, so I pull the UID_MIN from /etc/login.defs as the starting range for "typical" users, then use awk to extract home directories of such users from /etc/passwd. Those home directories are then excluded from the find.

Caveats

I wanted to use find ... -print0 in combination with readarray -t -d '' to safely capture any and all filenames, but the null-delimited readarray requires a recent bash version (4.4-alpha or later). Instead, I compromised and use find ... -print, with the caveat that any relevant filename that contains a newline in it will cause errors.

The script will not (during any single run) find newly-created log files; it gathers the initial list of potential log files, then loops back over that same list to see which file(s) grew. Newly-created files would only be caught by a subsequent run.

The script

#!/bin/bash

# files that grow by more than this much are interesting; in bytes per second; also, the answer to Life, the Universe, and Everything
rate=42

# how long we'll wait to account for file growth
sleeptime=60

function gethomedirs() (
 uidmin=$(awk '/^UID_MIN/ print $2 ' < /etc/login.defs)
 awk -F: -v umin="$uidmin" '$3 >= umin print $6 ' < /etc/passwd | sort -u
)

function findlogfiles 
 readarray -t homedirs < <(gethomedirs)

 if [ $#homedirs[@] -eq 0 ]
 then
 excludes=()
 elif [ $#homedirs[@] -eq 1 ]
 then
 excludes=("( -path $homedirs[0] ) -prune -o")
 else
 excludes=()
 excludes+=("(")
 excludes+=(" -path $homedirs[0]")
 for((i=1; i < $#homedirs[@]; i++))
 do
 excludes+=(" -o -path $homedirs[i]")
 done
 excludes+=(") -prune -o ")
 fi

 find $(lsblk --list --noheadings --output MOUNTPOINT 

readarray -t files < <(findlogfiles)
declare -A initialsize
for file in "$files[@]"
do
 initialsize["$file"]=$(stat -c %s "$file")
done

#echo Waiting $sleeptime seconds for log files to grow... >&2
sleep $sleeptime

for file in "$files[@]"
do
 # if the file went away, skip it
 [ -f "$file" ] || continue
 size2=$(stat -c %s "$file")
 if (( size2 >= ($initialsize["$file"] + rate * sleeptime) ))
 then
 printf "%sn" "$file"
 fi
done

So I came up with a plan, though I'm not super happy with it.

I used the ideas I already had, but made it less generic. I'm looking for log files in specific places, and only ones larger than a certain size (1M). So if someone has a log in a weird place I may not find it. To exclude files rotated by logrotate, I'm manually triggering a log rotation first. This way those files will be truncated before I run my searches. I'm also assuming log files named *.log, so if someone has a weirdly named log then I won't find it and it will cause problems eventually. I hope to solve that by teaching my team to use an ELK stack in the near future.

Here's my script that I run on each machine:

#!/bin/sh

sudo logrotate -vf /etc/logrotate.conf
sudo find /var/log -type f -mtime -2 -name "*log" -size +1M -exec sudo ls -l ; 
sudo find /home -type f -mtime -2 -name "*log" -size +1M -exec sudo ls -l ; 
sudo find /root -type f -mtime -2 -name "*log" -size +1M -exec sudo ls -l ;