Bounce rule not picking up bounced emails.

Advanced Questions & Problems

Bounce rule not picking up bounced emails.

Postby lawsonium » 1:17am, Sun 01 Jul, 2012

I noticed recently that I wasn't processing bounces as well as I thought so I ran some tests and tracked the results:

I created a fake user email and sent a message from phplist to the fake user.
The message bounced as expected and sat in my bounced-mail@mydomain.com folder.
I created a simple rule to match exactly, word for word, the error returned in the bounced email and made sure it was active.
I clicked 'process bounces' and it reported 10 of 16 processed (which incidentally is the exact same message I got after I processed bounces after my last newsletter was sent).
I got an email saying one bounce processed.
But the database showed that the bounced message was an 'unidentified bounce' and 'not processed'.
I checked the user and they still had a bounce count of 0 (zero).

I've trawled the forums and tried loads of possible solutions and tested different rules for hours and I keep getting the same result.

I am completely at a dead end and my sender score goes down each month because I'm sending emails to dead accounts.

Why isn't it working?
What can I check or try to fix this?

Any help much appreciated.

Thanks, Matt
lawsonium
phpLister
 
Posts: 10
Joined: 6:38pm, Fri 11 Nov, 2011

Re: Bounce rule not picking up bounced emails.

Postby duncanc » 7:50am, Sun 01 Jul, 2012

The status 'unidentified bounce' suggests that neither of the headers X-MessageId or X-Message were in the bounce response message. phplist adds them to the outgoing message, and relies on the bounce response message including the bounced message to identify the message id, and also the user id.

Can you check whether that is the case and copy the whole bounce response message here?
duncanc
Moderator
 
Posts: 2440
Joined: 6:34am, Sat 08 May, 2010
Location: London

Re: Bounce rule not picking up bounced emails.

Postby lawsonium » 10:24am, Sun 01 Jul, 2012

Hi, thanks for getting back to me. Firstly, I'm sorry, I think posted this in the wrong place accidentally (it was late and I was very tired!). Thanks for moving it.

I read about the 'X' header entries you described on another post, they described their problem as having these header entries stripped form their message and I wonder if that is my problem also, I think it is... Unfortunately the other post had no replies at all :-(

An example of an email that doesn't get processed (hMailServer is the mail server I use):
Return-Path:
Content-Type: text/plain; charset="utf-8"
Return-Path: <>
Message-ID: <12C211A3-5891-43E8-8AB5-61D9B76B3CEB@mail.<my_domain>.co.uk>
Date: Thu, 28 Jun 2012 23:48:42 +0100
From: mailer-daemon@mail.<my_domain>.co.uk
To: listbounces@<my_domain>.co.uk
Subject: Undeliverable: My Newsletter - Issue No. 10
Content-Transfer-Encoding: quoted-printable
X-hMailServer-LoopCount: 1

Your message did not reach some or all of the intended recipients.

Sent: Thu, 28 Jun 2012 23:48:38 +0100
Subject: My Newsletter - Issue No. 10

The following recipient(s) could not be reached:

dead.email@address.com
Error Type: SMTP
Remote server (173.194.67.27) issued an error.
hMailServer sent: RCPT TO:<dead.email@address.com>
Remote server replied: 550 5.2.1 The email account that you tried to reach is disabled. fy6si1501581wib.33

hMailServer

The data BLOB in 'phplist_bounce' for this email:
Your message did not reach some or all of the intended recipients.

Sent: Thu, 28 Jun 2012 23:48:38 +0100
Subject: My Newsletter - Issue No. 10

The following recipient(s) could not be reached:

bounced.email@ntlworld.com
Error Type: SMTP
Remote server (173.194.67.27) issued an error.
hMailServer sent: RCPT TO:<bounced.email@ntlworld.com>
Remote server replied: 550 5.2.1 The email account that you tried to reach is disabled. fy6si1501581wib.33

hMailServer



Interestingly. An email that was successful DOES include the 'X' headers, so I guess that is the reason they were processed. This is an example (they are highlighted red):
--=_1339812776_7877_1726_=
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

This message was created automatically by mail delivery software.

A message you sent could not be delivered to the following recipients:
bounced.email@dwac.demon.co.uk

The message has not been collected after 30 days

--=_1339812776_7877_1726_=
Content-Type: message/delivery-status

Reporting-MTA: dns; punt-3.mail.demon.net

Final-Recipient: rfc822; bounced.email@dwac.demon.co.uk
Action: failed
Status: 5.4.7

--=_1339812776_7877_1726_=
Content-Type: text/rfc822-headers; charset=us-ascii
Content-Transfer-Encoding: 7bit

Received: from punt3.mail.demon.net by mailstore
for bounced.email@dwac.demon.co.uk id 1SUlfv-2eYknw-07-FHI;
Wed, 16 May 2012 21:22:27 +0000
Received: from [194.217.242.72] (lhlo=anchor-hub.mail.demon.net)
by punt3.mail.demon.net with lmtp id 1SUlfv-2eYknw-07
for bounced.email@dwac.demon.co.uk; Wed, 16 May 2012 21:22:27 +0000
Received: from [89.200.138.74] (helo=mail.<my_domain>.co.uk)
by anchor-hub.mail.demon.net with smtp id 1SUlft-0001GM-7Q
for bounced.email@dwac.demon.co.uk; Wed, 16 May 2012 21:22:27 +0000
dkim-signature: v=1; a=rsa-sha1; d=<my_domain>.co.uk; s=ddkey;
c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:MIME-Version:Content-Type;
bh=Aq+IzkEEMd25yfqsAsZDVQjPWFk=;
b=YW5dcPBu7gF136Yp54bREM1bpQ1A4Tk2E7Dr6WLlu9Sy8RMrfLuV02WiEUlEzGzXFGKosz8XTAzm8wgCrZM5HM4/99c3k2WhxHmvmwQNv2lWZIbIDT6H8LeO3+bD77S0XsTeZZqDLTq2LAeZMv7py37mFPXfkfxb5F4UZGIg7Qk=
Received: from newsletters.<my_domain>.co.uk ([89.200.138.74])
by mail.<my_domain>.co.uk
; Wed, 16 May 2012 22:21:48 +0100
Date: Wed, 16 May 2012 22:21:48 +0100
To: bounced.email@dwac.demon.co.uk
From: My Newsletter <newsletter@<my_domain>.co.uk>
Subject: My Newsletter - Issue No.9
Message-ID: <f52b6dad93dba85bb606b8c629cca513@newsletters.<my_domain>.co.uk>
X-Priority: 3
X-Mailer: PHPMailer [version 1.73]
X-Mailer: phplist v2.10.17
X-MessageID: 95
X-ListMember: bounced.email@dwac.demon.co.uk

Errors-To: listbounces@<my_domain>.co.uk
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="b1_f52b6dad93dba85bb606b8c629cca513"
X-CNFS-Analysis: v=1.0 c=1 a=4G709EXKR1AA:10 a=mhcXdlOaJxsA:10 a=3j4BkbkPAAAA:8 a=6ks0QRl_AAAA:8 a=zDpSE3HdAAAA:8 a=UXOulrRAAAAA:8 a=HGfbgkoRFPpP6t8lN6AA:9 a=dm73Bg9ww8T19h5LHe8A:7 a=jbv8BIeG27gA:10 a=1mI2R1nvXRMA:10 a=Cyq7oN_fprin0DW1:21 a=KBZ5FFFzf6UPCU7E:21 a=SSmOFEACAAAA:8 a=09OksP0NAAAA:8 a=JqEG_dyiAAAA:8 a=9arimXYhAAAA:20 a=p27Z0iBnAAAA:8 a=A_3UDLNHxzfMdhCdnaYA:9 a=0jASoS6KhH-8HG29VuwA:7 a=_W_S_7VecoQA:10 a=frz4AuCg-hUA:10 a=tXsnliwV7b4A:10 a=uohbgR4oxNIA:10 a=QxmF7B006AgA:10 a=vONAHOggY6gA:10 a=ReLCYSF_eAcA:10 a=jHZKhQ3zu0_p9IS7:21 a=TIJKfMYH-9TvhO4i:21


--=_1339812776_7877_1726_=--



I am thinking that, emails bounced back by other email servers are returning with the FULL header, whereas emails that fail to send from my server after so many attempts (5 in my case I think), are having most of the header stripped out before being put into my 'bounced-email' forlder (listbounces@<my_domain>.co.uk).

At least I have something to go on now but no idea why this is happening. Any thoughts?

I just did a quick check on my database and some addresses seem to have bounced 8 or more times without being processed :-(
Is there a script or, to save me working it out with my limited SQL knowledge, do you have some SQL that could unconfirm all users that have 3 or more 'unidentified bounce' entries in 'phplist_bounce'?
I'm guessing it would have to make use of WHERE data LIKE '%<email_address>%' but link that email address to the user table and where the count in 'phplist_bounce' was 3 or more.
My SQL isn't quite that good.

Any help appreciated.

Thanks, Matt.

EDIT:
Now I think I know what I'm looking for, I found the following:

This post: http://forums.phplist.com/viewtopic.php?f=6&t=36516
Lead to this post: http://forums.phplist.com/viewtopic.php?f=24&t=35707&p=84184&hilit=bounce+mailer+daemon#p84184
Which suggests this fix (in hMailServer): http://www.hmailserver.com/forum/viewtopic.php?f=20&t=18029

I will have a go at applying the described fix for custome 'X' headers in hMailServer and report back.
lawsonium
phpLister
 
Posts: 10
Joined: 6:38pm, Fri 11 Nov, 2011

Re: Bounce rule not picking up bounced emails.

Postby duncanc » 6:05pm, Sun 01 Jul, 2012

Yes, the underlying problem is with the hmail server. If you are able to make the change to the server itself then that should enable future bounces to be processed by phplist.
duncanc
Moderator
 
Posts: 2440
Joined: 6:34am, Sat 08 May, 2010
Location: London

Re: Bounce rule not picking up bounced emails.

Postby lawsonium » 6:16pm, Sun 01 Jul, 2012

Hi,

I have applied the script to my hMailServer and I'm now getting the custom 'X' headers so phpList is able to identify the bounces. There is a small problem that hMailServer also still routes the incorrect bounce as well as the new amended one so I still get an entry in the bounce table with an unidentified bounce. I will post on the hMail forums about that though.

This is good except it has thrown another error up. That is, if the rule that traps the bounce is, for example: "unconfirm user and delete bounce", it removes the bounced email entry from the bounce table but if you click on the rule and view the details, at the bottom there is an option to see "related bounces". Clicking on the bounce id brings up an error: "Fatal Error: No such Record"...?
This appears to be because phpList isn't removing the record from the table: 'phplist_bounceregex_bounce' (I think). Manually deleting this record will clear it from the list.
What do I need to edit in the code to fix this?

Thanks, Matt.
lawsonium
phpLister
 
Posts: 10
Joined: 6:38pm, Fri 11 Nov, 2011

Re: Bounce rule not picking up bounced emails.

Postby duncanc » 7:22pm, Sun 01 Jul, 2012

The main processing is in file admin/processbounces.php

But the processing for "unconfirm user and delete bounce" does appear to delete the record from table phplist_bounceregex_bounce. See function deleteBounce() in file admin/lib.php.
duncanc
Moderator
 
Posts: 2440
Joined: 6:34am, Sat 08 May, 2010
Location: London

Re: Bounce rule not picking up bounced emails.

Postby lawsonium » 8:11pm, Sun 01 Jul, 2012

Hi,

I had looked in there and agree that this does show all three tables as being deleted. Code as follows:
Code: Select all
function deleteBounce($id = 0) {
  if (!$id) return;
  $id = sprintf('%d',$id);
  Sql_query(sprintf('delete from %s where id = %d',$GLOBALS['tables']['bounce'],$id));
  Sql_query(sprintf('delete from %s where bounce = %d',$GLOBALS['tables']['user_message_bounce'],$id));
  Sql_query(sprintf('delete from %s where bounce = %d',$GLOBALS['tables']['bounceregex_bounce'],$id));
}



However, I did find this link: https://mantis.phplist.com/view.php?id=9629
which appears to describe my problem (no solutions though).

From the description I checked the code file: 'bounce.php'
and it contains the following code:
Code: Select all
  if ($deletebounce) {
    print sprintf($GLOBALS['I18N']->get('DeletingB').'\n', $id);
    Sql_query("delete from {$tables["bounce"]} where id = $id");
    print $GLOBALS['I18N']->get('DoneAndLoading')."<br /><hr><br />\n";
    print PageLink2("bounces",$GLOBALS['I18N']->get('BackToBList'));
    $next = Sql_Fetch_Row_query(sprintf('select id from %s where id > %d',$tables["bounce"],$id));
    $id = $next[0];
    if (!$id) {
      $next = Sql_Fetch_Row_query(sprintf('select id from %s order by id desc limit 0,5',$tables["bounce"],$id));
      $id = $next[0];
    }


The original poster of that bug report seemed to think that this function should use the deleteBounce() function in lib.php which, as we can see, it doesn't.
It just has this SQL call:
Code: Select all
Sql_query("delete from {$tables["bounce"]} where id = $id");


I have tried to add a line beneath this that looked like:
Code: Select all
Sql_query("delete from {$tables["phplist_bounceregex_bounce"]} where bounce = $id");

but this didn't seem to fix the problem.

I don't know my way around the phpList code well enough to know if that bug report is pointing me in the right direction or not. What do you think?
If it is, what would I need to do to the code in 'bounce.php' to use the deleteBounce() function in lib.php?
Is it as simple as changing the line of code above to read:
Code: Select all
deleteBounce($row['bounce']);

exchanging the parameter '$row['bounce']' for '$id'?

I don't want to just hack away experimenting without running it by someone with a bit more knowledge of the code/system first.

In case it's relevant, I am running version 2.10.17 of phpList.

Thanks, Matt.
lawsonium
phpLister
 
Posts: 10
Joined: 6:38pm, Fri 11 Nov, 2011

Re: Bounce rule not picking up bounced emails.

Postby duncanc » 6:50am, Mon 02 Jul, 2012

The problem is in processbounces.php, these lines, 444-447, are in the wrong place
Code: Select all
      Sql_Query(sprintf('update %s set count = count + 1 where id = %d',
        $GLOBALS['tables']['bounceregex'],$rule['id']));
      Sql_Query(sprintf('insert ignore into %s (regex,bounce) values(%d,%d)',
        $GLOBALS['tables']['bounceregex_bounce'],$rule['id'],$row['bounce']));

They should be immediately before the switch statement on line 386.

Currently the row in bounceregex_bounce is created after the bounce has been deleted. If you move the statements then it will be created, then deleted within deleteBounce().

I have added this problem to Mantis, https://mantis.phplist.com/view.php?id=16653
and re-added the old issue that you noted https://mantis.phplist.com/view.php?id=16654
duncanc
Moderator
 
Posts: 2440
Joined: 6:34am, Sat 08 May, 2010
Location: London

Re: Bounce rule not picking up bounced emails.

Postby lawsonium » 5:43pm, Mon 02 Jul, 2012

Hi duncanc, that is excellent. Thank you and well spotted.
Thank you also for logging the bug.

I ran a small test and it displayed the correct behaviour (removing the bounce).

I think that covers the required solution to this particular issue. I am, however, still working on an SQL statement to consolidate the mass of unidentified bounces in the 'phplist_bounce' table. I think I have cracked it and will post in this ticket for completeness.

Thanks again.

Matt.
lawsonium
phpLister
 
Posts: 10
Joined: 6:38pm, Fri 11 Nov, 2011


Return to Advanced Questions & Problems

Who is online

Users browsing this forum: No registered users and 1 guest