mjhjmtu

Using rsync is surprising fast and simple.

mkdir empty_dir
rsync -a --delete empty_dir/ yourdirectory/

@sarath's answer mentioned another fast choice: Perl! Its benchmarks are faster than rsync -a --delete.

cd yourdirectory
perl -e 'for(<*>)((stat)[9]<(unlink))'

Sources:

1. https://stackoverflow.com/questions/1795370/unix-fast-remove-directory-for-cleaning-up-daily-builds


2. http://www.slashroot.in/which-is-the-fastest-method-to-delete-files-in-linux

Someone on Twitter suggested using -delete instead of -exec rm -f ;

This has improved the efficiency of the command, it still uses recursion to go through everything though.

What about something like:
find /path/to/folder -name "filenamestart*" -type f -print0 | xargs -0rn 20 rm -f

You can limit number of files to delete at once by changing the argument for parameter -n. The file names with blanks are included also.

Expanding on one of the comments, I do not think you're doing what you think you're doing.

First I created a huge amount of files, to simulate your situation:

$ mkdir foo
$ cd foo/
$ for X in $(seq 1 1000);do touch 1..1000_$X; done

Then I tried what I expected to fail, and what it sounds like you're doing in the question:

$ rm -r foo/*
bash: /bin/rm: Argument list too long

But this does work:

$ rm -r foo/
$ ls foo
ls: cannot access foo: No such file or directory

A clever trick:

rsync -a --delete empty/ your_folder/

It's super CPU intensive, but really really fast. See https://web.archive.org/web/20130929001850/http://linuxnote.net/jianingy/en/linux/a-fast-way-to-remove-huge-number-of-files.html

I had the opportunity to test -delete as compared to -exec rm ; and for me -delete was the answer to this problem.

Using -delete deleted the files in a folder of 400,000 files at least 1,000 times faster than rm.

The 'How to delete large number of files in linux' article suggests it is about three time faster, but in my test the difference was much more dramatic.

There are couple of methods that can be used to delete large number of files in linux,. You can use find with delete option, which is faster than exec option. Then you can use perl unlink, then even rsync.
How to delete large number of files in linux

About the -delete option above: I'm using it to remove a large number (1M+ est) files in a temp folder that I created and inadvertently forgot to cleanup nightly. I filled my disk/partition accidentally, and nothing else could remove them but the find . command. It is slow, at first I was using:

find . -ls -exec rm ;

But that was taking an EXTREME amount of time. It started after about 15 mins to remove some of the files, but my guess is that it was removing less than 10 or so per second after it finally started. So, I tried the:

find . -delete

instead, and I'm letting it run right now. It appears to be running faster, though it's EXTREMELY taxing on the CPU which the other command was not. It's been running for like an hour now and I think I'm getting space back on my drive and the partition gradually "slimming down" but it's still taking a very long time. I seriously doubt it's running 1,000 times faster than the other. As in all things, I just wanted to point out the tradeoff in space vs. time. If you have the CPU bandwidth to spare (we do) then run the latter. It's got my CPU running (uptime reports):

10:59:17 up 539 days, 21:21, 3 users, load average: 22.98, 24.10, 22.87

And I've seen the load average go over 30.00 which is not good for a busy system, but for ours which is normally lightly loaded, it's OK for a couple hours. I've checked most other things on the system and they're still responsive so we are OK for now.

Deleting REALLY LARGE directories needs a different approach, as I learned from this site - you'll need to utilize ionice.It ensures (with -c3) that deletes will only be performed when the system has IO-time for it. You systems load will not rise to high and everything stays responsive (though my CPU time for find was quite high at about 50%).

find <dir> -type f -exec ionice -c3 rm ;

Consider using Btrfs volume and simply delete whole volume for such a directory with large number of files.

Alternatively you can create an FS image file then unmount and delete its file to remove everything at once really fast.

Assuming to have GNU parallel installed, I've used this:

parallel rm -rf dir/ ::: `ls -f dir/`


and it was fast enough.

ls -1 | xargs rm -rf 

should work inside the main folder

For Izkata's hint above:

But this does work:

$ rm -r foo/
$ ls foo
ls: cannot access foo: No such file or directory

This almost worked - or would have worked - but I had some problems in permission; files were on a server, but still I don't understand where this permission issue came from. Anyway, Terminal asked for confirmation on every file. Amount of files was around 20 000, so this wasn't an option. After "-r" I added option "-f", so the whole command was "rm -r -f foldername/". Then it seemed to work fine. I'm a novice with Terminal, but I guess this was okay, right? Thanks!

Depending on how well you need to get rid of those files, I'd suggest using shred.

$ shred -zuv folder

if you want to purge the directory, but you can't remove it and recreate it, I suggest moving it and recreating it instantly.

mv folder folder_del
mkdir folder
rm -rf folder_del

this is faster, believe it or not, as only one inode has to be changed. Remember: You can't really parallelize this tast on a multicore computer. It comes down to disk access, which is limited by the RAID or what have you.

If you have millions of files and every solution above gets your system in stress you may try this inspiration:

File nice_delete:

#!/bin/bash

MAX_LOAD=3
FILES=("$@")
BATCH=100

while [ $#FILES[@] -gt 0 ]; do
 DEL=("$FILES[@]:0:$BATCH")
 ionice -c3 rm "$DEL[@]"
 echo -n "#"
 FILES=("$FILES[@]:$BATCH")
 while [[ $(cat /proc/loadavg | awk 'print int($1)') -gt $MAX_LOAD ]]; do
 echo -n "."
 sleep 1
 done
done

And now delete the files:

find /path/to/folder -type f -exec ./nice_delete +

Find will create batches (see getconf ARG_MAX) of some tens thousands of files and pass it to nice_delete. This will create even smaller batches to allow sleeping when overload is detected.