How can my postfix/spamcop learn from Gmail 421 rejections?

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
1
down vote

favorite












I have a number of vanity catch-all email domains. Back in the day, that was a good idea, and now it's too late to change for my friends and family.



I do not relay out, only serving incoming domains. Those incoming messages then get forwarded using mail aliases rules that each user locally configures. I am receiving mail and forwarding to user gmail inboxes using postfix. To make sure I filter out 90%+ of the spam, I run spamassassin with auto-update, as well as two RBL blocking lists and SPF records. Good mail does get through to Google, which is great!



Bad mail that still slips through the net ends up with a 421 temporary denial from Google. Typically Google will say "this is spam" or "this contains bad links" in the reject message, which is good as far as it goes, but I don't read the logs every hour and check every message.



Currently, I run a command that flushes the deferred queue once a day, so that I don't re-try the same spam too often. This is somewhat fragile, because a single message that arrives right before the flush, and then gets deferred once for some technical reason (TCP timeout etc) would also get deleted without delivery. Not great!



So, how can I go about training my spanassassin based on the messages received back from Google?
For now, I'm thinking of something that wakes up every 10 minutes, tails the mail.log file, and looks for 421 messages, extracts the message ID using regex, then runs postcat on that message, and feeds it to sa-learn for training.



First: Is something like this already available?



Second: Can you find anything wrong, missed assumption, etc, in my reasoning above that I should correct?






postfix







share|improve this question



















  • 1




    This is the kind of thing that perl's File::Tail module is good for. use File::Tail to monitor mail.log for 421 codes on delivery to google. extract the queue-id and postcat -bh the deferred mail to /usr/bin/sa-learn --spam .... I use sa-ham and sa-spam aliases for training spamassasin (which i run from amavisd-new)...note: postfix runs | aliases as user nobody by default, so you'll need a second alias file owned by the user you run spamassassin as (in my case, that's user amavis)
    – cas
    Aug 6 '16 at 6:19











  • alternatively, don't use an alias, run sa-learn directly from the File::Tail monitoring script, and run that script with the same uid that you use to run spamassassin.
    – cas
    Aug 6 '16 at 6:23










  • btw, I also use formail (from the procmail package) to strip Resent-From etc headers (I usually bounce spam that gets through my filters to sa-spam). so my sa-spam alias looks like this: sa-spam: "|/usr/bin/formail -I Resent-From -I Resent-Date -I Resent-Message-ID -I Resent-To -s | /usr/bin/sa-learn --spam --dbpath /var/lib/amavis/.spamassassin/"
    – cas
    Aug 6 '16 at 6:25











  • finally, File::Tail seemed appropriate for a quick answer I wrote last night. it's just pseudocode, but it's still a useful example: unix.stackexchange.com/a/301545/7696
    – cas
    Aug 6 '16 at 6:31










  • Thanks for the suggestions. It sounds like you're basically doing what I had come up with! (But as a persistent process.) I have written perl before, and there are other languages I'd probably use if that's the way I go. (Including C -- nothing wrong with that :-)
    – Jon Watte
    Aug 7 '16 at 4:35














up vote
1
down vote

favorite












I have a number of vanity catch-all email domains. Back in the day, that was a good idea, and now it's too late to change for my friends and family.



I do not relay out, only serving incoming domains. Those incoming messages then get forwarded using mail aliases rules that each user locally configures. I am receiving mail and forwarding to user gmail inboxes using postfix. To make sure I filter out 90%+ of the spam, I run spamassassin with auto-update, as well as two RBL blocking lists and SPF records. Good mail does get through to Google, which is great!



Bad mail that still slips through the net ends up with a 421 temporary denial from Google. Typically Google will say "this is spam" or "this contains bad links" in the reject message, which is good as far as it goes, but I don't read the logs every hour and check every message.



Currently, I run a command that flushes the deferred queue once a day, so that I don't re-try the same spam too often. This is somewhat fragile, because a single message that arrives right before the flush, and then gets deferred once for some technical reason (TCP timeout etc) would also get deleted without delivery. Not great!



So, how can I go about training my spanassassin based on the messages received back from Google?
For now, I'm thinking of something that wakes up every 10 minutes, tails the mail.log file, and looks for 421 messages, extracts the message ID using regex, then runs postcat on that message, and feeds it to sa-learn for training.



First: Is something like this already available?



Second: Can you find anything wrong, missed assumption, etc, in my reasoning above that I should correct?






postfix







share|improve this question



















  • 1




    This is the kind of thing that perl's File::Tail module is good for. use File::Tail to monitor mail.log for 421 codes on delivery to google. extract the queue-id and postcat -bh the deferred mail to /usr/bin/sa-learn --spam .... I use sa-ham and sa-spam aliases for training spamassasin (which i run from amavisd-new)...note: postfix runs | aliases as user nobody by default, so you'll need a second alias file owned by the user you run spamassassin as (in my case, that's user amavis)
    – cas
    Aug 6 '16 at 6:19











  • alternatively, don't use an alias, run sa-learn directly from the File::Tail monitoring script, and run that script with the same uid that you use to run spamassassin.
    – cas
    Aug 6 '16 at 6:23










  • btw, I also use formail (from the procmail package) to strip Resent-From etc headers (I usually bounce spam that gets through my filters to sa-spam). so my sa-spam alias looks like this: sa-spam: "|/usr/bin/formail -I Resent-From -I Resent-Date -I Resent-Message-ID -I Resent-To -s | /usr/bin/sa-learn --spam --dbpath /var/lib/amavis/.spamassassin/"
    – cas
    Aug 6 '16 at 6:25











  • finally, File::Tail seemed appropriate for a quick answer I wrote last night. it's just pseudocode, but it's still a useful example: unix.stackexchange.com/a/301545/7696
    – cas
    Aug 6 '16 at 6:31










  • Thanks for the suggestions. It sounds like you're basically doing what I had come up with! (But as a persistent process.) I have written perl before, and there are other languages I'd probably use if that's the way I go. (Including C -- nothing wrong with that :-)
    – Jon Watte
    Aug 7 '16 at 4:35












up vote
1
down vote

favorite









up vote
1
down vote

favorite











I have a number of vanity catch-all email domains. Back in the day, that was a good idea, and now it's too late to change for my friends and family.



I do not relay out, only serving incoming domains. Those incoming messages then get forwarded using mail aliases rules that each user locally configures. I am receiving mail and forwarding to user gmail inboxes using postfix. To make sure I filter out 90%+ of the spam, I run spamassassin with auto-update, as well as two RBL blocking lists and SPF records. Good mail does get through to Google, which is great!



Bad mail that still slips through the net ends up with a 421 temporary denial from Google. Typically Google will say "this is spam" or "this contains bad links" in the reject message, which is good as far as it goes, but I don't read the logs every hour and check every message.



Currently, I run a command that flushes the deferred queue once a day, so that I don't re-try the same spam too often. This is somewhat fragile, because a single message that arrives right before the flush, and then gets deferred once for some technical reason (TCP timeout etc) would also get deleted without delivery. Not great!



So, how can I go about training my spanassassin based on the messages received back from Google?
For now, I'm thinking of something that wakes up every 10 minutes, tails the mail.log file, and looks for 421 messages, extracts the message ID using regex, then runs postcat on that message, and feeds it to sa-learn for training.



First: Is something like this already available?



Second: Can you find anything wrong, missed assumption, etc, in my reasoning above that I should correct?






postfix







share|improve this question















I have a number of vanity catch-all email domains. Back in the day, that was a good idea, and now it's too late to change for my friends and family.



I do not relay out, only serving incoming domains. Those incoming messages then get forwarded using mail aliases rules that each user locally configures. I am receiving mail and forwarding to user gmail inboxes using postfix. To make sure I filter out 90%+ of the spam, I run spamassassin with auto-update, as well as two RBL blocking lists and SPF records. Good mail does get through to Google, which is great!



Bad mail that still slips through the net ends up with a 421 temporary denial from Google. Typically Google will say "this is spam" or "this contains bad links" in the reject message, which is good as far as it goes, but I don't read the logs every hour and check every message.



Currently, I run a command that flushes the deferred queue once a day, so that I don't re-try the same spam too often. This is somewhat fragile, because a single message that arrives right before the flush, and then gets deferred once for some technical reason (TCP timeout etc) would also get deleted without delivery. Not great!



So, how can I go about training my spanassassin based on the messages received back from Google?
For now, I'm thinking of something that wakes up every 10 minutes, tails the mail.log file, and looks for 421 messages, extracts the message ID using regex, then runs postcat on that message, and feeds it to sa-learn for training.



First: Is something like this already available?



Second: Can you find anything wrong, missed assumption, etc, in my reasoning above that I should correct?






postfix




postfix






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 7 mins ago









Rui F Ribeiro

37.4k1374118




37.4k1374118










asked Aug 5 '16 at 16:31









Jon Watte

10814




10814







  • 1




    This is the kind of thing that perl's File::Tail module is good for. use File::Tail to monitor mail.log for 421 codes on delivery to google. extract the queue-id and postcat -bh the deferred mail to /usr/bin/sa-learn --spam .... I use sa-ham and sa-spam aliases for training spamassasin (which i run from amavisd-new)...note: postfix runs | aliases as user nobody by default, so you'll need a second alias file owned by the user you run spamassassin as (in my case, that's user amavis)
    – cas
    Aug 6 '16 at 6:19











  • alternatively, don't use an alias, run sa-learn directly from the File::Tail monitoring script, and run that script with the same uid that you use to run spamassassin.
    – cas
    Aug 6 '16 at 6:23










  • btw, I also use formail (from the procmail package) to strip Resent-From etc headers (I usually bounce spam that gets through my filters to sa-spam). so my sa-spam alias looks like this: sa-spam: "|/usr/bin/formail -I Resent-From -I Resent-Date -I Resent-Message-ID -I Resent-To -s | /usr/bin/sa-learn --spam --dbpath /var/lib/amavis/.spamassassin/"
    – cas
    Aug 6 '16 at 6:25











  • finally, File::Tail seemed appropriate for a quick answer I wrote last night. it's just pseudocode, but it's still a useful example: unix.stackexchange.com/a/301545/7696
    – cas
    Aug 6 '16 at 6:31










  • Thanks for the suggestions. It sounds like you're basically doing what I had come up with! (But as a persistent process.) I have written perl before, and there are other languages I'd probably use if that's the way I go. (Including C -- nothing wrong with that :-)
    – Jon Watte
    Aug 7 '16 at 4:35












  • 1




    This is the kind of thing that perl's File::Tail module is good for. use File::Tail to monitor mail.log for 421 codes on delivery to google. extract the queue-id and postcat -bh the deferred mail to /usr/bin/sa-learn --spam .... I use sa-ham and sa-spam aliases for training spamassasin (which i run from amavisd-new)...note: postfix runs | aliases as user nobody by default, so you'll need a second alias file owned by the user you run spamassassin as (in my case, that's user amavis)
    – cas
    Aug 6 '16 at 6:19











  • alternatively, don't use an alias, run sa-learn directly from the File::Tail monitoring script, and run that script with the same uid that you use to run spamassassin.
    – cas
    Aug 6 '16 at 6:23










  • btw, I also use formail (from the procmail package) to strip Resent-From etc headers (I usually bounce spam that gets through my filters to sa-spam). so my sa-spam alias looks like this: sa-spam: "|/usr/bin/formail -I Resent-From -I Resent-Date -I Resent-Message-ID -I Resent-To -s | /usr/bin/sa-learn --spam --dbpath /var/lib/amavis/.spamassassin/"
    – cas
    Aug 6 '16 at 6:25











  • finally, File::Tail seemed appropriate for a quick answer I wrote last night. it's just pseudocode, but it's still a useful example: unix.stackexchange.com/a/301545/7696
    – cas
    Aug 6 '16 at 6:31










  • Thanks for the suggestions. It sounds like you're basically doing what I had come up with! (But as a persistent process.) I have written perl before, and there are other languages I'd probably use if that's the way I go. (Including C -- nothing wrong with that :-)
    – Jon Watte
    Aug 7 '16 at 4:35







1




1




This is the kind of thing that perl's File::Tail module is good for. use File::Tail to monitor mail.log for 421 codes on delivery to google. extract the queue-id and postcat -bh the deferred mail to /usr/bin/sa-learn --spam .... I use sa-ham and sa-spam aliases for training spamassasin (which i run from amavisd-new)...note: postfix runs | aliases as user nobody by default, so you'll need a second alias file owned by the user you run spamassassin as (in my case, that's user amavis)
– cas
Aug 6 '16 at 6:19





This is the kind of thing that perl's File::Tail module is good for. use File::Tail to monitor mail.log for 421 codes on delivery to google. extract the queue-id and postcat -bh the deferred mail to /usr/bin/sa-learn --spam .... I use sa-ham and sa-spam aliases for training spamassasin (which i run from amavisd-new)...note: postfix runs | aliases as user nobody by default, so you'll need a second alias file owned by the user you run spamassassin as (in my case, that's user amavis)
– cas
Aug 6 '16 at 6:19













alternatively, don't use an alias, run sa-learn directly from the File::Tail monitoring script, and run that script with the same uid that you use to run spamassassin.
– cas
Aug 6 '16 at 6:23




alternatively, don't use an alias, run sa-learn directly from the File::Tail monitoring script, and run that script with the same uid that you use to run spamassassin.
– cas
Aug 6 '16 at 6:23












btw, I also use formail (from the procmail package) to strip Resent-From etc headers (I usually bounce spam that gets through my filters to sa-spam). so my sa-spam alias looks like this: sa-spam: "|/usr/bin/formail -I Resent-From -I Resent-Date -I Resent-Message-ID -I Resent-To -s | /usr/bin/sa-learn --spam --dbpath /var/lib/amavis/.spamassassin/"
– cas
Aug 6 '16 at 6:25





btw, I also use formail (from the procmail package) to strip Resent-From etc headers (I usually bounce spam that gets through my filters to sa-spam). so my sa-spam alias looks like this: sa-spam: "|/usr/bin/formail -I Resent-From -I Resent-Date -I Resent-Message-ID -I Resent-To -s | /usr/bin/sa-learn --spam --dbpath /var/lib/amavis/.spamassassin/"
– cas
Aug 6 '16 at 6:25













finally, File::Tail seemed appropriate for a quick answer I wrote last night. it's just pseudocode, but it's still a useful example: unix.stackexchange.com/a/301545/7696
– cas
Aug 6 '16 at 6:31




finally, File::Tail seemed appropriate for a quick answer I wrote last night. it's just pseudocode, but it's still a useful example: unix.stackexchange.com/a/301545/7696
– cas
Aug 6 '16 at 6:31












Thanks for the suggestions. It sounds like you're basically doing what I had come up with! (But as a persistent process.) I have written perl before, and there are other languages I'd probably use if that's the way I go. (Including C -- nothing wrong with that :-)
– Jon Watte
Aug 7 '16 at 4:35




Thanks for the suggestions. It sounds like you're basically doing what I had come up with! (But as a persistent process.) I have written perl before, and there are other languages I'd probably use if that's the way I go. (Including C -- nothing wrong with that :-)
– Jon Watte
Aug 7 '16 at 4:35















active

oldest

votes











Your Answer







StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f301563%2fhow-can-my-postfix-spamcop-learn-from-gmail-421-rejections%23new-answer', 'question_page');

);

Post as a guest






















active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes















 

draft saved


draft discarded















































 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f301563%2fhow-can-my-postfix-spamcop-learn-from-gmail-421-rejections%23new-answer', 'question_page');

);

Post as a guest
































































-

Popular posts from this blog

Peggy Mitchell

Image
EastEnders character This article is about the soap opera character. For the British tennis player, see Peggy Michell. For the author, see Margaret Mitchell. Peggy Mitchell Barbara Windsor as Peggy Mitchell (2008) EastEnders character Portrayed by Jo Warne (1991) Barbara Windsor (1994–2016) Duration 1991, 1994–2010, 2013–2016 First appearance Episode 650 30 April 1991  ( 1991-04-30 ) Last appearance Episode 5286 17 May 2016  ( 2016-05-17 ) (appearance) Episode 5287 19 May 2016 (voiceover) Introduced by Michael Ferguson (1991) Barbara Emile (1994) Louise Berridge (2004) Kate Harwood (2005) Lorraine Newman (2013) Dominic Treadwell-Collins (2014) Spin-off appearances The Mitchells - Naked Truths (1998) Classification Former; regular Profile Other names Peggy Butcher Occupation Barmaid Businesswoman Pub landlady Jo Warne as Peggy Mitchell (1991) Family Family Mitchell Father Jack Martin Mother Lily Martin Sisters Aunt ...
Read more

The Forum (Inglewood, California)

Image
The Forum "LA Forum" Prairie Ave. façade of The Forum in 2014 Full name The Forum, presented by Chase Former names The Forum (1967–1988) Great Western Forum (1988–2003) Address 3900 W. Manchester Blvd Location Inglewood, California Coordinates Coordinates: 33°57′29″N 118°20′31″W  /  33.95806°N 118.34194°W  / 33.95806; -118.34194 Public transit     Downtown Inglewood station Owner The Madison Square Garden Company Operator MSG Entertainment Seating type Reserved Capacity 17,500 Half-bowl: 8,000 Construction Broke ground July 1, 1966  ( 1966-07-01 ) Opened December 30, 1967  ( 1967-12-30 ) Renovated 1988, 2012–2014 Construction cost $16 million Renovation: 2014: $76.5 million Architect Charles Luckman Associates (original) Brisbin Brook Beynon (renovation) Structural engineer Johnson & Nielsen Associates (original) Severud Associates (renovation) Genera...
Read more

Palaiologos

Image
Palaiologos Παλαιολόγος Palaeologus, Palaeologue Imperial Family Double-headed eagle with the family cypher Country Byzantine Empire, Duchy of Montferrat (remaining lineage after Empire annexed by Ottomans) Founded 11th century  ( 11th century ) Founder Nikephoros Palaiologos (first known) Final ruler Constantine XI Palaiologos (Byzantine Empire) Margaret Paleologa (Montferrat) Titles Byzantine Emperor Marquess of Montferrat Style(s) "Augustus" "Basileus" "Autokrator" Traditions Greek Orthodoxy (predominantly) Roman Catholicism (remaining lineage in Montferrat) Dissolution 1533  ( 1533 ) Cadet branches Claimants: House of Kastrioti (defunct) House of Rurik (defunct) Palaiologan dynasty Chronology Michael VIII 1259–1282 with Andronikos II as co-emperor, 1261–1282 Andronikos II 1282–1328 with Michael IX (1294–1320) and Andronikos III (1321–1328) as co-emperors Andronikos III 1328–1341 John...
Read more