mjhjmtu

Over time, I've encountered the same pattern again and again: I have some kind of directory structure:

example/
├── a
│   └── c
│   ├── d.txt (120k)
│   └── e.txt (60k)
└── b
 └── f.txt (280k)

And I want to "copy" the files over to another directory, say, example_grepped, applying a command to each as if in place of cp - say, grep ERROR so that say, I end up with a folder with the same structure but with files filtered through the grep.

example_grepped/
├── a
│   └── c
│   ├── d.txt (1k)
│   └── e.txt (0b)
└── b
 └── f.txt (12k)

Same pattern for converting media files (FLACs to MP3s, PNGs to JPGs), and this time when converting different schema formats as part of a build process.

Is there a generic command that I could use? Something like foobar example example_grepped --command 'grep ERROR' or foobar flacs mp3s --command 'ffmpeg -i .mp3'?

An obscure xargs flag perhaps? (a find piped through xargs would almost suffice, but most if not all commands expect the directory structure to already exist.)

Closest answer I can find without separately recreating the directory structure is to use install:

cd example
find . -type f -exec sh -c 'grep ERROR | install -D /dev/stdin /tmp/example_grepped/' ;

Unfortunately the above can only work if your command can throw its result to STDOUT.

Another way to approach this is to use a program that does recursive copies anyway. I checked rsync, but could not find a callback option on a quick glance. But gnu tar has an option --to-command for which you can provide a command to be run that gets the file's input into stdin. But how to get the file created then? Well, the command called finds the current file name in $TAR_FILENAME.

Putting it all together, the basic call is

tar cf - example | tar xf - --to-command="./script example_grepped 'grep-pattern'"

where script could be something like

#!/bin/bash
mkdir -p $(dirname "$1/$TAR_FILENAME")
grep '$2' >"$1/$TAR_FILENAME"
exit 0

Another way to approach this, would be to wrap the tar pipe in a script that gets the command to run on the command line. Yet the escaping for the mkdir ...dirname construct will be a bit challenging.

#!/bin/bash

filter() 

 local target_root="$@: -1"

 target_path=$(sed -E "s/[^/]*/$target_root/" <<< "$1")
 target_dir=$(dirname "$target_path")

 mkdir -p "$target_dir"

 if [[ -f $1 ]]; then
 # do your grep thing here
 grep burger "$1" > "$target_path"
 fi


export -f filter
source_root="example"
target_root="example_grepped"

find "$source_root/" -print0 | xargs -0 -I content bash -c "filter 'content' '$target_root'"

This script also works with directory and file names that contain spaces.

Run this script where the source directory ("example") is located.

Using GNU Parallel you can do something like this:

cd src
find . -type f | parallel 'mkdir -p ../dst///; dostuff --input --output ../dst/'