mjhjmtu

I have a list of files in /tmp/drop directory. I have taken the essential element of my question into test script zz.sh. I have taken a part of the script and put it here. These are the files and the lines listed - and when it given to while loop - how does the variable file gets assigned only the file name and not the various other strings in those lines?

Any help would be appreciated and would be helpful to others too.

machine001:/home/aaa999999> ls -l /tmp/drop | grep "2017-01-29" | egrep '.gz$' | tail -10
+ ls -F -l /tmp/drop
+ tail -10
+ grep 2017-01-29
+ egrep .gz$
-rw-r--r-- 1 user1001 user1001 20 Jan 29 2017 pattern101_worker-2_2017-01-29-21:33:13.888.complete.gz
-rw-r--r-- 1 user1001 user1001 20 Jan 29 2017 pattern101_worker-2_2017-01-29-21:48:14.632.complete.gz
-rw-r--r-- 1 user1001 user1001 20 Jan 29 2017 pattern101_worker-2_2017-01-29-22:03:19.098.complete.gz
-rw-r--r-- 1 user1001 user1001 20 Jan 29 2017 pattern101_worker-2_2017-01-29-22:18:19.416.complete.gz
-rw-r--r-- 1 user1001 user1001 20 Jan 29 2017 pattern101_worker-2_2017-01-29-22:33:19.878.complete.gz
-rw-r--r-- 1 user1001 user1001 20 Jan 29 2017 pattern101_worker-2_2017-01-29-22:48:25.636.complete.gz
-rw-r--r-- 1 user1001 user1001 20 Jan 29 2017 pattern101_worker-2_2017-01-29-23:03:26.515.complete.gz
-rw-r--r-- 1 user1001 user1001 20 Jan 29 2017 pattern101_worker-2_2017-01-29-23:18:28.279.complete.gz
-rw-r--r-- 1 user1001 user1001 20 Jan 29 2017 pattern101_worker-2_2017-01-29-23:33:33.059.complete.gz
-rw-r--r-- 1 user1001 user1001 20 Jan 29 2017 pattern101_worker-2_2017-01-29-23:48:33.841.complete.gz
machine001:/home/aaa999999> cat zz.sh
+ cat zz.sh
#!/bin/bash
set -x

DROP_DIR=/tmp/drop
ARCHIVE_DIR=/tmp/arch
YESTERDAY="2017-01-29"
# move from drop dir
ls -1 $DROP_DIR | grep $YESTERDAY | egrep '.gz$' | while read file; do
 mv $DROP_DIR/$file $ARCHIVE_DIR
 echo 'File name is ' $file
done

ls -1 prints just the filenames. ls -l prints a bunch of other stuff, too. The other is the number one, the other a lowercase letter el.

Though the -1 is unnecessary here, since if the output is redirected to a pipe, the "one-column" mode is implied.

But really, you don't need the ls and the pipe here, at all. The loop can be replaced with just:

for file in "$DROP_DIR/"*"$YESTERDAY"*.gz ; do 
 mv "$file" "$ARCHIVE_DIR"
 echo "File name is $file#$DROP_DIR/"
done

See also: ParsingLs on Greg's Bash wiki.

Suggestion using GNU date:

yesterday=$( date -d yesterday +%F )
drop_dir=/tmp/drop
archive_dir=/tmp/arch

mkdir -p "$archive_dir"

mv -f "$drop_dir"/*"$yesterday"*.gz "$archive_dir"

Here, "$drop_dir"/*"$yesterday"*.gz will expand to all files matching the given pattern. This will work as long as the pattern expand to several thousands of characters, in which case the shell will complain with "Argument list too long".

Or, if you really want to do looping to output each filename as they are moved (only the mv command from above is changed to the following):

for name in "$drop_dir"/*"$yesterday"*.gz; do
 [ -f "$name" ] || continue
 mv "$name" "$archive_dir"
 printf 'Moved "%s"n' "$name"
done

Here, the for loop will iterate over the same names as was used in the previous variation, but one by one. Each name will be held in $name (which includes the directory name). If there were no files matching the pattern, the pattern will remain unexpanded. This is why we do an explicit test to make sure that $name really corresponds to an existing file before moving it.

Or, with find:

find "$drop_dir" -maxdepth 1 -type f -name "*$yesterday*.gz" 
 -print -exec mv "$archive_dir" ';'

Here, find will pick up the names from $drop_dir that corresponds to regular files (-type f) and that matches the pattern given to -name. For each such name, mv is called to move the file into tho destination folder. Before calling mv though, -print will list the filename (including path) on the terminal.

The -maxdepth 1 option stops find from recursing into subfolders of $drop_dir.

Your question isn't too clear to me, but I think what you seem to be looking for is the -o option to your egrep statement:

 egrep -o "[^[:space:]]+.gz$"

The -o option (--only-matching) causes egrep to only output the matching portion of matching lines. The [^[:space:]] matches anything not a space character.