mjhjmtu

I'm running something like this:

find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | while read file ; do
 echo "Encrypting $file..."
 gpg --trust-model always --recipient "me@myself.com" --output "$file.gpg" 
 --encrypt "$file" && rm "$file" 
done

This runs great, but it seems that GPG is not optimized to use multiple cores for an encryption operation. The files I'm encrypting are about 2GB in size and I have quite a bit of them. I'd like to be able to run X jobs in parallel to encrypt the files and then remove them. How can I do this, setting a limit to, say, 8 jobs at a time?

If you install the GNU Parallel tool you can make pretty easy work of what you're trying to accomplish:

$ find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | 
 parallel --gnu -j 8 --workdir $PWD ' 
 echo "Encrypting ..."; 
 gpg --trust-model always 
 --recipient "me@myself.com" --output ".gpg" 
 --encrypt "" && rm "" 
 '

details

The above is taking the output of find and running it through to parallel, and running 8 at a time. Everywhere there's an occurrence of the filenames that are being passed through from find will replace the in those spots.

References

• Running shell script in parallel

You might want to look at gnu parallel and its --semaphore option. From the documentation:

--semaphore

Work as a counting semaphore. --semaphore will cause GNU parallel to start command in the background. When the number of simultaneous jobs is reached, GNU parallel will wait for one of these to complete before starting another command.

You use --jobs 8 to limit the number of jobs to 8. You can pipe the output of sort into parallel like you would do with xargs.
sem is an alias for parallel --semaphore

I have written an easy-to-use Perl script that allows to control the maximum number commands that are run at the same time: https://github.com/matmu/parallelize_cmds

Might be of interest for you.