awk file sanity check. awk check for null/missing values in csv files

Clash Royale CLAN TAG#URR8PPP
up vote
1
down vote
favorite
I'm trying to write a simple file sanity check script. I have a directory with dozen CSV files containing id,edname,firstname,lastname,suffix,email.
I like to write a awk script to check if first field contain a number and is not empty. and fields number 3,4 & 6 are not empty and that the file contains 6 fields no more no less than 6, if all of this conditions are true nothing happens but if any of these conditions failed, re-name the file to .bad. here is wha i have but is not picking up missing values in columns 4,6.
for f in *.csv; do
awk -F, '!(NF==6 && $1+0==$1 && $3$4$6!="")f=1; exit ENDexit f' "$f" || mv "$f" "$f".bad;
done
awk csv
add a comment |Â
up vote
1
down vote
favorite
I'm trying to write a simple file sanity check script. I have a directory with dozen CSV files containing id,edname,firstname,lastname,suffix,email.
I like to write a awk script to check if first field contain a number and is not empty. and fields number 3,4 & 6 are not empty and that the file contains 6 fields no more no less than 6, if all of this conditions are true nothing happens but if any of these conditions failed, re-name the file to .bad. here is wha i have but is not picking up missing values in columns 4,6.
for f in *.csv; do
awk -F, '!(NF==6 && $1+0==$1 && $3$4$6!="")f=1; exit ENDexit f' "$f" || mv "$f" "$f".bad;
done
awk csv
1
Surely$3$4$6!=""will return true if at least one of the fields is non-empty. Is that what you really want?
â steeldriver
Dec 7 '17 at 18:36
no. I see. wha I want is, if any of these fields is empty then that's a bad file, because those fields are required/mandatory.
â daniel caceres
Dec 7 '17 at 18:48
add a comment |Â
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I'm trying to write a simple file sanity check script. I have a directory with dozen CSV files containing id,edname,firstname,lastname,suffix,email.
I like to write a awk script to check if first field contain a number and is not empty. and fields number 3,4 & 6 are not empty and that the file contains 6 fields no more no less than 6, if all of this conditions are true nothing happens but if any of these conditions failed, re-name the file to .bad. here is wha i have but is not picking up missing values in columns 4,6.
for f in *.csv; do
awk -F, '!(NF==6 && $1+0==$1 && $3$4$6!="")f=1; exit ENDexit f' "$f" || mv "$f" "$f".bad;
done
awk csv
I'm trying to write a simple file sanity check script. I have a directory with dozen CSV files containing id,edname,firstname,lastname,suffix,email.
I like to write a awk script to check if first field contain a number and is not empty. and fields number 3,4 & 6 are not empty and that the file contains 6 fields no more no less than 6, if all of this conditions are true nothing happens but if any of these conditions failed, re-name the file to .bad. here is wha i have but is not picking up missing values in columns 4,6.
for f in *.csv; do
awk -F, '!(NF==6 && $1+0==$1 && $3$4$6!="")f=1; exit ENDexit f' "$f" || mv "$f" "$f".bad;
done
awk csv
edited Dec 7 '17 at 20:15
Jeff Schaller
32k848109
32k848109
asked Dec 7 '17 at 18:26
daniel caceres
262
262
1
Surely$3$4$6!=""will return true if at least one of the fields is non-empty. Is that what you really want?
â steeldriver
Dec 7 '17 at 18:36
no. I see. wha I want is, if any of these fields is empty then that's a bad file, because those fields are required/mandatory.
â daniel caceres
Dec 7 '17 at 18:48
add a comment |Â
1
Surely$3$4$6!=""will return true if at least one of the fields is non-empty. Is that what you really want?
â steeldriver
Dec 7 '17 at 18:36
no. I see. wha I want is, if any of these fields is empty then that's a bad file, because those fields are required/mandatory.
â daniel caceres
Dec 7 '17 at 18:48
1
1
Surely
$3$4$6!="" will return true if at least one of the fields is non-empty. Is that what you really want?â steeldriver
Dec 7 '17 at 18:36
Surely
$3$4$6!="" will return true if at least one of the fields is non-empty. Is that what you really want?â steeldriver
Dec 7 '17 at 18:36
no. I see. wha I want is, if any of these fields is empty then that's a bad file, because those fields are required/mandatory.
â daniel caceres
Dec 7 '17 at 18:48
no. I see. wha I want is, if any of these fields is empty then that's a bad file, because those fields are required/mandatory.
â daniel caceres
Dec 7 '17 at 18:48
add a comment |Â
2 Answers
2
active
oldest
votes
up vote
2
down vote
As steeldriver pointed out in the comments, your third test will be true if any of the three fields isn't empty. I assume you actually want something like this:
for f in *.csv; do
awk -F, '!(NF==6 && $1+0==$1 && $3!="" && $4!="" && $6!="")f=1; exit
ENDexit f' "$f" || mv "$f" "$f".bad;
done
well thought out. thanks. let me run it.
â daniel caceres
Dec 7 '17 at 20:12
how can I also check if $6 contains an @ sign?
â daniel caceres
Dec 8 '17 at 2:52
add a comment |Â
up vote
1
down vote
No need to do string comparisons if you just want to ensure it's not null. Empty strings are falsy, so:
awk -F, 'BEGIN flag=0 !(NF==6 && $1+0==$1 && $3 && $4 && $6) flag=1 END exit flag'
This fails if fields 3, 4 or 6 are0though. But that's not empty.
â terdonâ¦
Dec 7 '17 at 19:13
I believe it fair to infer that a first name, last name, and email address of0are invalid, but you also raise a fair point.
â DopeGhoti
Dec 7 '17 at 19:23
lol, I had completely missed that point. You're right, a last name of0doesn't seem very likely :) Ah, but you should also check that$1is a number.
â terdonâ¦
Dec 7 '17 at 19:24
yes $1 must be a number, otherwise is a bad file.
â daniel caceres
Dec 7 '17 at 19:36
add a comment |Â
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
As steeldriver pointed out in the comments, your third test will be true if any of the three fields isn't empty. I assume you actually want something like this:
for f in *.csv; do
awk -F, '!(NF==6 && $1+0==$1 && $3!="" && $4!="" && $6!="")f=1; exit
ENDexit f' "$f" || mv "$f" "$f".bad;
done
well thought out. thanks. let me run it.
â daniel caceres
Dec 7 '17 at 20:12
how can I also check if $6 contains an @ sign?
â daniel caceres
Dec 8 '17 at 2:52
add a comment |Â
up vote
2
down vote
As steeldriver pointed out in the comments, your third test will be true if any of the three fields isn't empty. I assume you actually want something like this:
for f in *.csv; do
awk -F, '!(NF==6 && $1+0==$1 && $3!="" && $4!="" && $6!="")f=1; exit
ENDexit f' "$f" || mv "$f" "$f".bad;
done
well thought out. thanks. let me run it.
â daniel caceres
Dec 7 '17 at 20:12
how can I also check if $6 contains an @ sign?
â daniel caceres
Dec 8 '17 at 2:52
add a comment |Â
up vote
2
down vote
up vote
2
down vote
As steeldriver pointed out in the comments, your third test will be true if any of the three fields isn't empty. I assume you actually want something like this:
for f in *.csv; do
awk -F, '!(NF==6 && $1+0==$1 && $3!="" && $4!="" && $6!="")f=1; exit
ENDexit f' "$f" || mv "$f" "$f".bad;
done
As steeldriver pointed out in the comments, your third test will be true if any of the three fields isn't empty. I assume you actually want something like this:
for f in *.csv; do
awk -F, '!(NF==6 && $1+0==$1 && $3!="" && $4!="" && $6!="")f=1; exit
ENDexit f' "$f" || mv "$f" "$f".bad;
done
edited Dec 7 '17 at 20:14
Jeff Schaller
32k848109
32k848109
answered Dec 7 '17 at 19:13
terdonâ¦
122k28230403
122k28230403
well thought out. thanks. let me run it.
â daniel caceres
Dec 7 '17 at 20:12
how can I also check if $6 contains an @ sign?
â daniel caceres
Dec 8 '17 at 2:52
add a comment |Â
well thought out. thanks. let me run it.
â daniel caceres
Dec 7 '17 at 20:12
how can I also check if $6 contains an @ sign?
â daniel caceres
Dec 8 '17 at 2:52
well thought out. thanks. let me run it.
â daniel caceres
Dec 7 '17 at 20:12
well thought out. thanks. let me run it.
â daniel caceres
Dec 7 '17 at 20:12
how can I also check if $6 contains an @ sign?
â daniel caceres
Dec 8 '17 at 2:52
how can I also check if $6 contains an @ sign?
â daniel caceres
Dec 8 '17 at 2:52
add a comment |Â
up vote
1
down vote
No need to do string comparisons if you just want to ensure it's not null. Empty strings are falsy, so:
awk -F, 'BEGIN flag=0 !(NF==6 && $1+0==$1 && $3 && $4 && $6) flag=1 END exit flag'
This fails if fields 3, 4 or 6 are0though. But that's not empty.
â terdonâ¦
Dec 7 '17 at 19:13
I believe it fair to infer that a first name, last name, and email address of0are invalid, but you also raise a fair point.
â DopeGhoti
Dec 7 '17 at 19:23
lol, I had completely missed that point. You're right, a last name of0doesn't seem very likely :) Ah, but you should also check that$1is a number.
â terdonâ¦
Dec 7 '17 at 19:24
yes $1 must be a number, otherwise is a bad file.
â daniel caceres
Dec 7 '17 at 19:36
add a comment |Â
up vote
1
down vote
No need to do string comparisons if you just want to ensure it's not null. Empty strings are falsy, so:
awk -F, 'BEGIN flag=0 !(NF==6 && $1+0==$1 && $3 && $4 && $6) flag=1 END exit flag'
This fails if fields 3, 4 or 6 are0though. But that's not empty.
â terdonâ¦
Dec 7 '17 at 19:13
I believe it fair to infer that a first name, last name, and email address of0are invalid, but you also raise a fair point.
â DopeGhoti
Dec 7 '17 at 19:23
lol, I had completely missed that point. You're right, a last name of0doesn't seem very likely :) Ah, but you should also check that$1is a number.
â terdonâ¦
Dec 7 '17 at 19:24
yes $1 must be a number, otherwise is a bad file.
â daniel caceres
Dec 7 '17 at 19:36
add a comment |Â
up vote
1
down vote
up vote
1
down vote
No need to do string comparisons if you just want to ensure it's not null. Empty strings are falsy, so:
awk -F, 'BEGIN flag=0 !(NF==6 && $1+0==$1 && $3 && $4 && $6) flag=1 END exit flag'
No need to do string comparisons if you just want to ensure it's not null. Empty strings are falsy, so:
awk -F, 'BEGIN flag=0 !(NF==6 && $1+0==$1 && $3 && $4 && $6) flag=1 END exit flag'
edited Dec 7 '17 at 20:01
answered Dec 7 '17 at 19:11
DopeGhoti
40.6k54979
40.6k54979
This fails if fields 3, 4 or 6 are0though. But that's not empty.
â terdonâ¦
Dec 7 '17 at 19:13
I believe it fair to infer that a first name, last name, and email address of0are invalid, but you also raise a fair point.
â DopeGhoti
Dec 7 '17 at 19:23
lol, I had completely missed that point. You're right, a last name of0doesn't seem very likely :) Ah, but you should also check that$1is a number.
â terdonâ¦
Dec 7 '17 at 19:24
yes $1 must be a number, otherwise is a bad file.
â daniel caceres
Dec 7 '17 at 19:36
add a comment |Â
This fails if fields 3, 4 or 6 are0though. But that's not empty.
â terdonâ¦
Dec 7 '17 at 19:13
I believe it fair to infer that a first name, last name, and email address of0are invalid, but you also raise a fair point.
â DopeGhoti
Dec 7 '17 at 19:23
lol, I had completely missed that point. You're right, a last name of0doesn't seem very likely :) Ah, but you should also check that$1is a number.
â terdonâ¦
Dec 7 '17 at 19:24
yes $1 must be a number, otherwise is a bad file.
â daniel caceres
Dec 7 '17 at 19:36
This fails if fields 3, 4 or 6 are
0 though. But that's not empty.â terdonâ¦
Dec 7 '17 at 19:13
This fails if fields 3, 4 or 6 are
0 though. But that's not empty.â terdonâ¦
Dec 7 '17 at 19:13
I believe it fair to infer that a first name, last name, and email address of
0 are invalid, but you also raise a fair point.â DopeGhoti
Dec 7 '17 at 19:23
I believe it fair to infer that a first name, last name, and email address of
0 are invalid, but you also raise a fair point.â DopeGhoti
Dec 7 '17 at 19:23
lol, I had completely missed that point. You're right, a last name of
0 doesn't seem very likely :) Ah, but you should also check that $1 is a number.â terdonâ¦
Dec 7 '17 at 19:24
lol, I had completely missed that point. You're right, a last name of
0 doesn't seem very likely :) Ah, but you should also check that $1 is a number.â terdonâ¦
Dec 7 '17 at 19:24
yes $1 must be a number, otherwise is a bad file.
â daniel caceres
Dec 7 '17 at 19:36
yes $1 must be a number, otherwise is a bad file.
â daniel caceres
Dec 7 '17 at 19:36
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f409543%2fawk-file-sanity-check-awk-check-for-null-missing-values-in-csv-files%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
1
Surely
$3$4$6!=""will return true if at least one of the fields is non-empty. Is that what you really want?â steeldriver
Dec 7 '17 at 18:36
no. I see. wha I want is, if any of these fields is empty then that's a bad file, because those fields are required/mandatory.
â daniel caceres
Dec 7 '17 at 18:48