mjhjmtu

On my server, the root partition is 73GB, but the disk says, it is full, although there are only about 6GB used on that device:

# df -h
Filesystem Size Used Avail Use% Mounted on
udev 997M 0 997M 0% /dev
tmpfs 202M 41M 162M 20% /run
/dev/mapper/p22server-root 73G 68G 655M 100% /

(I use LVM on the server: /dev/mapper/p22server-root -> ../dm-0)
If I check with

ncdu -x /

I find out that the total usage is just 5.9GB.

My guess is, that there must be files still open, that are not visible in the file-tree.

How can I debug this? I guess, that a reboot will restore the lost space, but rebooting is not possible right now.

If your suspicion is true, you may have an easier time with lsof.

Look out for "(deleted)" or similar in the output.

There are two possibilities:

1. There are deleted files still open by some procesces

You can see all open files with lsof. For example those are the TYPEs shown in lsof and how often they appear in the output:

# lsof|cut -c50-54|sort|uniq -c
 375 CHR 
 610 DIR 
 211 FIFO 
 32 IPv4 
 17 IPv6 
 40 link 
 419 node 
 152 nown 
6008 REG 
 9 sock 
 598 unix 

This shows only the File Descriptor column (FD):

# lsof|awk -v field="FD" 'NR==1 c = index($0,field) print substr($0,c-1,length(field)+4)'|sort|uniq -c|sort -n|tail

look for files with the value DEL in the output of lsof to get a hint.

2. Files are hidden behinid a mount point

But more probably, the files are just "hidden" due to a mount point, that was not used while large amount of files was stored in the mount-folder and now they are invisible to ncdu.

mount the root directory in another location and analyze this:

mkdir /temp-root
mount --bind / /temp-root
ncdu -x /temp_root