awk file sanity check. awk check for null/missing values in csv files

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
1
down vote

favorite












I'm trying to write a simple file sanity check script. I have a directory with dozen CSV files containing id,edname,firstname,lastname,suffix,email.



I like to write a awk script to check if first field contain a number and is not empty. and fields number 3,4 & 6 are not empty and that the file contains 6 fields no more no less than 6, if all of this conditions are true nothing happens but if any of these conditions failed, re-name the file to .bad. here is wha i have but is not picking up missing values in columns 4,6.



for f in *.csv; do 
awk -F, '!(NF==6 && $1+0==$1 && $3$4$6!="")f=1; exit ENDexit f' "$f" || mv "$f" "$f".bad;
done






share|improve this question


















  • 1




    Surely $3$4$6!="" will return true if at least one of the fields is non-empty. Is that what you really want?
    – steeldriver
    Dec 7 '17 at 18:36










  • no. I see. wha I want is, if any of these fields is empty then that's a bad file, because those fields are required/mandatory.
    – daniel caceres
    Dec 7 '17 at 18:48














up vote
1
down vote

favorite












I'm trying to write a simple file sanity check script. I have a directory with dozen CSV files containing id,edname,firstname,lastname,suffix,email.



I like to write a awk script to check if first field contain a number and is not empty. and fields number 3,4 & 6 are not empty and that the file contains 6 fields no more no less than 6, if all of this conditions are true nothing happens but if any of these conditions failed, re-name the file to .bad. here is wha i have but is not picking up missing values in columns 4,6.



for f in *.csv; do 
awk -F, '!(NF==6 && $1+0==$1 && $3$4$6!="")f=1; exit ENDexit f' "$f" || mv "$f" "$f".bad;
done






share|improve this question


















  • 1




    Surely $3$4$6!="" will return true if at least one of the fields is non-empty. Is that what you really want?
    – steeldriver
    Dec 7 '17 at 18:36










  • no. I see. wha I want is, if any of these fields is empty then that's a bad file, because those fields are required/mandatory.
    – daniel caceres
    Dec 7 '17 at 18:48












up vote
1
down vote

favorite









up vote
1
down vote

favorite











I'm trying to write a simple file sanity check script. I have a directory with dozen CSV files containing id,edname,firstname,lastname,suffix,email.



I like to write a awk script to check if first field contain a number and is not empty. and fields number 3,4 & 6 are not empty and that the file contains 6 fields no more no less than 6, if all of this conditions are true nothing happens but if any of these conditions failed, re-name the file to .bad. here is wha i have but is not picking up missing values in columns 4,6.



for f in *.csv; do 
awk -F, '!(NF==6 && $1+0==$1 && $3$4$6!="")f=1; exit ENDexit f' "$f" || mv "$f" "$f".bad;
done






share|improve this question














I'm trying to write a simple file sanity check script. I have a directory with dozen CSV files containing id,edname,firstname,lastname,suffix,email.



I like to write a awk script to check if first field contain a number and is not empty. and fields number 3,4 & 6 are not empty and that the file contains 6 fields no more no less than 6, if all of this conditions are true nothing happens but if any of these conditions failed, re-name the file to .bad. here is wha i have but is not picking up missing values in columns 4,6.



for f in *.csv; do 
awk -F, '!(NF==6 && $1+0==$1 && $3$4$6!="")f=1; exit ENDexit f' "$f" || mv "$f" "$f".bad;
done








share|improve this question













share|improve this question




share|improve this question








edited Dec 7 '17 at 20:15









Jeff Schaller

32k848109




32k848109










asked Dec 7 '17 at 18:26









daniel caceres

262




262







  • 1




    Surely $3$4$6!="" will return true if at least one of the fields is non-empty. Is that what you really want?
    – steeldriver
    Dec 7 '17 at 18:36










  • no. I see. wha I want is, if any of these fields is empty then that's a bad file, because those fields are required/mandatory.
    – daniel caceres
    Dec 7 '17 at 18:48












  • 1




    Surely $3$4$6!="" will return true if at least one of the fields is non-empty. Is that what you really want?
    – steeldriver
    Dec 7 '17 at 18:36










  • no. I see. wha I want is, if any of these fields is empty then that's a bad file, because those fields are required/mandatory.
    – daniel caceres
    Dec 7 '17 at 18:48







1




1




Surely $3$4$6!="" will return true if at least one of the fields is non-empty. Is that what you really want?
– steeldriver
Dec 7 '17 at 18:36




Surely $3$4$6!="" will return true if at least one of the fields is non-empty. Is that what you really want?
– steeldriver
Dec 7 '17 at 18:36












no. I see. wha I want is, if any of these fields is empty then that's a bad file, because those fields are required/mandatory.
– daniel caceres
Dec 7 '17 at 18:48




no. I see. wha I want is, if any of these fields is empty then that's a bad file, because those fields are required/mandatory.
– daniel caceres
Dec 7 '17 at 18:48










2 Answers
2






active

oldest

votes

















up vote
2
down vote













As steeldriver pointed out in the comments, your third test will be true if any of the three fields isn't empty. I assume you actually want something like this:



for f in *.csv; do 
awk -F, '!(NF==6 && $1+0==$1 && $3!="" && $4!="" && $6!="")f=1; exit
ENDexit f' "$f" || mv "$f" "$f".bad;
done





share|improve this answer






















  • well thought out. thanks. let me run it.
    – daniel caceres
    Dec 7 '17 at 20:12










  • how can I also check if $6 contains an @ sign?
    – daniel caceres
    Dec 8 '17 at 2:52

















up vote
1
down vote













No need to do string comparisons if you just want to ensure it's not null. Empty strings are falsy, so:



awk -F, 'BEGIN flag=0 !(NF==6 && $1+0==$1 && $3 && $4 && $6) flag=1 END exit flag'





share|improve this answer






















  • This fails if fields 3, 4 or 6 are 0 though. But that's not empty.
    – terdon♦
    Dec 7 '17 at 19:13










  • I believe it fair to infer that a first name, last name, and email address of 0 are invalid, but you also raise a fair point.
    – DopeGhoti
    Dec 7 '17 at 19:23











  • lol, I had completely missed that point. You're right, a last name of 0 doesn't seem very likely :) Ah, but you should also check that $1 is a number.
    – terdon♦
    Dec 7 '17 at 19:24











  • yes $1 must be a number, otherwise is a bad file.
    – daniel caceres
    Dec 7 '17 at 19:36










Your Answer







StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f409543%2fawk-file-sanity-check-awk-check-for-null-missing-values-in-csv-files%23new-answer', 'question_page');

);

Post as a guest






























2 Answers
2






active

oldest

votes








2 Answers
2






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
2
down vote













As steeldriver pointed out in the comments, your third test will be true if any of the three fields isn't empty. I assume you actually want something like this:



for f in *.csv; do 
awk -F, '!(NF==6 && $1+0==$1 && $3!="" && $4!="" && $6!="")f=1; exit
ENDexit f' "$f" || mv "$f" "$f".bad;
done





share|improve this answer






















  • well thought out. thanks. let me run it.
    – daniel caceres
    Dec 7 '17 at 20:12










  • how can I also check if $6 contains an @ sign?
    – daniel caceres
    Dec 8 '17 at 2:52














up vote
2
down vote













As steeldriver pointed out in the comments, your third test will be true if any of the three fields isn't empty. I assume you actually want something like this:



for f in *.csv; do 
awk -F, '!(NF==6 && $1+0==$1 && $3!="" && $4!="" && $6!="")f=1; exit
ENDexit f' "$f" || mv "$f" "$f".bad;
done





share|improve this answer






















  • well thought out. thanks. let me run it.
    – daniel caceres
    Dec 7 '17 at 20:12










  • how can I also check if $6 contains an @ sign?
    – daniel caceres
    Dec 8 '17 at 2:52












up vote
2
down vote










up vote
2
down vote









As steeldriver pointed out in the comments, your third test will be true if any of the three fields isn't empty. I assume you actually want something like this:



for f in *.csv; do 
awk -F, '!(NF==6 && $1+0==$1 && $3!="" && $4!="" && $6!="")f=1; exit
ENDexit f' "$f" || mv "$f" "$f".bad;
done





share|improve this answer














As steeldriver pointed out in the comments, your third test will be true if any of the three fields isn't empty. I assume you actually want something like this:



for f in *.csv; do 
awk -F, '!(NF==6 && $1+0==$1 && $3!="" && $4!="" && $6!="")f=1; exit
ENDexit f' "$f" || mv "$f" "$f".bad;
done






share|improve this answer














share|improve this answer



share|improve this answer








edited Dec 7 '17 at 20:14









Jeff Schaller

32k848109




32k848109










answered Dec 7 '17 at 19:13









terdon♦

122k28230403




122k28230403











  • well thought out. thanks. let me run it.
    – daniel caceres
    Dec 7 '17 at 20:12










  • how can I also check if $6 contains an @ sign?
    – daniel caceres
    Dec 8 '17 at 2:52
















  • well thought out. thanks. let me run it.
    – daniel caceres
    Dec 7 '17 at 20:12










  • how can I also check if $6 contains an @ sign?
    – daniel caceres
    Dec 8 '17 at 2:52















well thought out. thanks. let me run it.
– daniel caceres
Dec 7 '17 at 20:12




well thought out. thanks. let me run it.
– daniel caceres
Dec 7 '17 at 20:12












how can I also check if $6 contains an @ sign?
– daniel caceres
Dec 8 '17 at 2:52




how can I also check if $6 contains an @ sign?
– daniel caceres
Dec 8 '17 at 2:52












up vote
1
down vote













No need to do string comparisons if you just want to ensure it's not null. Empty strings are falsy, so:



awk -F, 'BEGIN flag=0 !(NF==6 && $1+0==$1 && $3 && $4 && $6) flag=1 END exit flag'





share|improve this answer






















  • This fails if fields 3, 4 or 6 are 0 though. But that's not empty.
    – terdon♦
    Dec 7 '17 at 19:13










  • I believe it fair to infer that a first name, last name, and email address of 0 are invalid, but you also raise a fair point.
    – DopeGhoti
    Dec 7 '17 at 19:23











  • lol, I had completely missed that point. You're right, a last name of 0 doesn't seem very likely :) Ah, but you should also check that $1 is a number.
    – terdon♦
    Dec 7 '17 at 19:24











  • yes $1 must be a number, otherwise is a bad file.
    – daniel caceres
    Dec 7 '17 at 19:36














up vote
1
down vote













No need to do string comparisons if you just want to ensure it's not null. Empty strings are falsy, so:



awk -F, 'BEGIN flag=0 !(NF==6 && $1+0==$1 && $3 && $4 && $6) flag=1 END exit flag'





share|improve this answer






















  • This fails if fields 3, 4 or 6 are 0 though. But that's not empty.
    – terdon♦
    Dec 7 '17 at 19:13










  • I believe it fair to infer that a first name, last name, and email address of 0 are invalid, but you also raise a fair point.
    – DopeGhoti
    Dec 7 '17 at 19:23











  • lol, I had completely missed that point. You're right, a last name of 0 doesn't seem very likely :) Ah, but you should also check that $1 is a number.
    – terdon♦
    Dec 7 '17 at 19:24











  • yes $1 must be a number, otherwise is a bad file.
    – daniel caceres
    Dec 7 '17 at 19:36












up vote
1
down vote










up vote
1
down vote









No need to do string comparisons if you just want to ensure it's not null. Empty strings are falsy, so:



awk -F, 'BEGIN flag=0 !(NF==6 && $1+0==$1 && $3 && $4 && $6) flag=1 END exit flag'





share|improve this answer














No need to do string comparisons if you just want to ensure it's not null. Empty strings are falsy, so:



awk -F, 'BEGIN flag=0 !(NF==6 && $1+0==$1 && $3 && $4 && $6) flag=1 END exit flag'






share|improve this answer














share|improve this answer



share|improve this answer








edited Dec 7 '17 at 20:01

























answered Dec 7 '17 at 19:11









DopeGhoti

40.6k54979




40.6k54979











  • This fails if fields 3, 4 or 6 are 0 though. But that's not empty.
    – terdon♦
    Dec 7 '17 at 19:13










  • I believe it fair to infer that a first name, last name, and email address of 0 are invalid, but you also raise a fair point.
    – DopeGhoti
    Dec 7 '17 at 19:23











  • lol, I had completely missed that point. You're right, a last name of 0 doesn't seem very likely :) Ah, but you should also check that $1 is a number.
    – terdon♦
    Dec 7 '17 at 19:24











  • yes $1 must be a number, otherwise is a bad file.
    – daniel caceres
    Dec 7 '17 at 19:36
















  • This fails if fields 3, 4 or 6 are 0 though. But that's not empty.
    – terdon♦
    Dec 7 '17 at 19:13










  • I believe it fair to infer that a first name, last name, and email address of 0 are invalid, but you also raise a fair point.
    – DopeGhoti
    Dec 7 '17 at 19:23











  • lol, I had completely missed that point. You're right, a last name of 0 doesn't seem very likely :) Ah, but you should also check that $1 is a number.
    – terdon♦
    Dec 7 '17 at 19:24











  • yes $1 must be a number, otherwise is a bad file.
    – daniel caceres
    Dec 7 '17 at 19:36















This fails if fields 3, 4 or 6 are 0 though. But that's not empty.
– terdon♦
Dec 7 '17 at 19:13




This fails if fields 3, 4 or 6 are 0 though. But that's not empty.
– terdon♦
Dec 7 '17 at 19:13












I believe it fair to infer that a first name, last name, and email address of 0 are invalid, but you also raise a fair point.
– DopeGhoti
Dec 7 '17 at 19:23





I believe it fair to infer that a first name, last name, and email address of 0 are invalid, but you also raise a fair point.
– DopeGhoti
Dec 7 '17 at 19:23













lol, I had completely missed that point. You're right, a last name of 0 doesn't seem very likely :) Ah, but you should also check that $1 is a number.
– terdon♦
Dec 7 '17 at 19:24





lol, I had completely missed that point. You're right, a last name of 0 doesn't seem very likely :) Ah, but you should also check that $1 is a number.
– terdon♦
Dec 7 '17 at 19:24













yes $1 must be a number, otherwise is a bad file.
– daniel caceres
Dec 7 '17 at 19:36




yes $1 must be a number, otherwise is a bad file.
– daniel caceres
Dec 7 '17 at 19:36

















 

draft saved


draft discarded















































 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f409543%2fawk-file-sanity-check-awk-check-for-null-missing-values-in-csv-files%23new-answer', 'question_page');

);

Post as a guest













































































Popular posts from this blog

Peggy Mitchell

Palaiologos

The Forum (Inglewood, California)