[mod] re-sending soft bounces with advanced bounce handling

3rd party code for phpList

[mod] re-sending soft bounces with advanced bounce handling

Postby CS » 2:05pm, Mon 15 Sep, 2008

I am testing out PHPlist to see how well it will work for my company's needs. So far, everything looks great; however, I have been able to get PHPlist to automatically re-send messages when the delivery failed. The only way I have found is to process bounces then manually run a query to remove the users who bounced from being in the "sent" table for that message. At that point, PHPlist will re-send the message. However, that is obviously very clunky and I'm sure there is a better way to do it.

Any help would be greatly appreciated!
Last edited by CS on 1:56pm, Tue 30 Sep, 2008, edited 1 time in total.
CS
 

Postby Ernesto » 8:15pm, Mon 15 Sep, 2008

If I understood correctly you want to re-send a message to email addresses that bounced the message in a previous attempt. This is not possible with the current code I'm afraid.

Assuming the causes for the bounces were permanent (e.g. the email address doesn't exist anymore), then it would not seem like a good idea to retry sending to that address. On the other hand, in the case of 'soft' bounces with temporary causes (e.g. mailbox full) it might be interesting to be able to retry sending a message.

While 'advanced bounce handling' can distinguish between 'soft' and 'hard' bounces, it does not have an option to resend a bounced message. Might be an idea to file a feature request.
Search first, Ask questions later...
Ernesto
PL Freak!
 
Posts: 592
Joined: 2:25am, Thu 08 Jun, 2006

Postby CS » 2:04pm, Tue 16 Sep, 2008

Thanks for the response! You understood my request correctly, I want to resend messages when PHPlist receives a 'soft' bounce. I have seen other users report messages like this in their posts:

Code: Select all
1 emails failed (will retry later)


Which is what made me think it already had this feature in some form. If this is not what it is doing, then what does it consider a 'failed' message?

I believe I could probably 'hack' the code to make it perform this function (by checking the bounce type then sending an query to remove the user from the 'sent' table for that message), but a built-in feature would obviously be preferable.
CS
 

Postby Ernesto » 4:53pm, Tue 16 Sep, 2008

CS wrote:what does it consider a 'failed' message?
Good question. I'm not quite sure, but I think 'failed' messages are labelled as such during queue processing, and might be caused by anomalies/time-outs(?) during the sending process, or by email addresses that phplist didn't detect as invalid even though they are. You could take a look at processqueue.php to check this out. See also: http://forums.phplist.com/viewtopic.php?p=53652#53652

Bounced messages, on the other hand, wont be detected during queue processing. It may take some time for bounced emails to accumulate in the bounce account you defined in config.php. Only when you process bounces, will phplist retrieve them from the bounce account and be able to detect and act upon them.

Standard bounce processing makes no distinction between soft and hard bounces: a bounce is a bounce. Every bounce for a given email address will increase its bounce count. If the bounce count reaches the threshold you defined in config.php, the email address will be marked as 'unconfirmed' (or 'blacklisted', not quite sure if that was changed recently).

Advanced bounce processing, is better suited as a starting point for your project, because it will try to distinguish between soft and hard bounces, through regular expressions. The bouncing rules you define, will allow you to act differently for each category of bounces.
Since the options that are available to act on a bounce do not yet include re-sending the same message, you might consider concentrating your modifications there.
I suppose it might be easiest to add an advanced bounce handling option that will unmark the message's 'sent' status for a particular user, and at the same time not increase the bounce count for that user.

I expect you would then only need to requeue the message and it should be sent to all users that do not have the 'sent' status for that message, i.e. users that had soft bounces, as well as new subscribers.

BTW, 'advanced bounce handling' is enabled in the online demo, if you want to get an idea of currently available options. See http://demo.phplist.com/lists/admin/?page=bouncemgt
Search first, Ask questions later...
Ernesto
PL Freak!
 
Posts: 592
Joined: 2:25am, Thu 08 Jun, 2006

Postby CS » 6:37pm, Tue 16 Sep, 2008

Thanks again for your help. I dug around a bit in the code and it generates a 'failed message' error if there is some error that actually prevents it from sending the email in the first place.

I'll hack on it a bit and if I come up with something useful I'll post back here with my results.
CS
 

Postby CS » 9:57pm, Mon 29 Sep, 2008

By the way, I have figured this out. If anyone is interested, I did it this way:

1) I added a new config setting in config.php called $soft_bounce_threshold that determines how many times it will resend a message on a soft bounce.
2) In admin/lib.php search for the "bouncerulesactions" array (near the top) and added a new line so it looks like this (I added the 'resend' action):
Code: Select all
$GLOBALS['bounceruleactions'] = array(
  'deleteuser' => $GLOBALS['I18N']->get('delete user'),
  'unconfirmuser' => $GLOBALS['I18N']->get('unconfirm user'),
  'blacklistuser' => $GLOBALS['I18N']->get('blacklist user'),
  'deleteuserandbounce' => $GLOBALS['I18N']->get('delete user and bounce'),
  'unconfirmuseranddeletebounce' => $GLOBALS['I18N']->get('unconfirm user and delete bounce'),
  'blacklistuseranddeletebounce' => $GLOBALS['I18N']->get('blacklist user and delete bounce'),
  'deletebounce' => $GLOBALS['I18N']->get('delete bounce'),
  'resend' => $GLOBALS['I18N']->get('Resend message to user and delete bounce'),
); // This is a custom bounce rule

3) In admin/processbounces.php, look around line 400 (about two thirds down) for a large switch and add the following case:
Code: Select all
         case 'resend':
          # This is a custom bounce rule for 'soft bounces' which will modify the database to make it appear as if the message has not yet been sent to the user and also makes sure the message still has the status "inprocess". 
          # In order to actually resend the message, the queue will have to be processed again.
          # $soft_bounce_threshhold is a new variable and should be set in the config file along with a comment explaining what it is for.
          # It defines how many times PHPlist will attempt to send the same message to the same user. 
            Sql_Query(sprintf('select * from %s where user = %s and message = %s',$tables["user_message_bounce"],$userdata['id'],$row['message']));
            if (!isset($soft_bounce_threshold)) $soft_bounce_threshold = 5;
            if (mysql_affected_rows() <= $soft_bounce_threshold)
            {
             logEvent('User '.$userdata['email'].' message resent by bounce rule '.PageLink2('bouncerule&id='.$rule['id'],$rule['id']));
             $advanced_report .= 'Message resent to user '.$userdata['email'].' by bounce rule '.$rule['id']."\n";
             $advanced_report .= 'User: http://'.$GLOBALS['domain'].$GLOBALS['adminpages'].'/?page=user&id='.$userdata['id']."\n";
             $advanced_report .= 'Rule: http://'.$GLOBALS['domain'].$GLOBALS['adminpages'].'/?page=bouncerule&id='.$rule['id']."\n";
             Sql_Query(sprintf('delete from %s where messageid = %s and userid = %s',$tables["usermessage"],$row['message'],$userdata['id']));
                  Sql_Query(sprintf('update %s set status = "inprocess" where id = %s',$tables['message'],$row['message']));
            }
          deleteBounce($row['bounce']);
            break;


In order to actually use this, you will need to process bounces and then process queue again. If the message has already finished sending, it will re-queue it so it can be processed again.

You will also need to define bounce rules that use this action.

Hopefully somebody will find this information useful and a similar process can be used to add other custom bounce rules, as well.
CS
 

Postby Ernesto » 10:14pm, Mon 29 Sep, 2008

Nice hack!

Have you already used it on your production install? Just wondering what your experiences are with using your mod in
combination with 'regular expressions' for soft bounces.

I agree that this should certainly opens up the way for additional bounce rules.

Cheers!
Search first, Ask questions later...
Ernesto
PL Freak!
 
Posts: 592
Joined: 2:25am, Thu 08 Jun, 2006

Postby CS » 1:06pm, Tue 30 Sep, 2008

Thanks. I haven't used it in production yet. I'll be using PHPlist for several clients, some of whom have very large lists, so I'm doing extensive testing prior to production, though I expect to start limited production soon.
CS
 

Postby H2B2 » 10:26pm, Tue 30 Sep, 2008

CS, Thanks for sharing your mod!

I took the liberty of changing the title of the first post (Getting PHPlist to retry sending 'failed' messages) to one that (IMO) provides
a better description of the actual contents of this thread. Please change my alternative title if you feel you have a better one.

Thread moved to the contributions section.

EDIT: BTW, I just discovered a mod that seems somewhat related, which is available at mantis: http://mantis.phplist.com/view.php?id=8499
You could add a link to this thread in the above mentioned mantis report if you like.
H2B2
Moderator
 
Posts: 7188
Joined: 1:51am, Wed 15 Mar, 2006

Postby CS » 1:16pm, Wed 01 Oct, 2008

H2B2:

Thanks for the Mantis link. It looks like that mod does essentially the same thing as mine except that it decrements the bounce count (I recommend adding that bit if you use my mod), but it doesn't automatically re-queue the message or have a re-send limit. I think I will link to this thread in the Mantis ticket.

Btw, I like the idea mentioned in the thread of being able to export and import bounce rules. It seems like it would be a relatively simple matter to export the bounceregex table as a tab-delimited or CSV file, then have another script that would read the file and create the appropriate queries to update the table for import.

It would be a low priority for me at the moment, but I might work on it when I have time.
CS
 

Postby CS » 9:52pm, Thu 06 Nov, 2008

In some testing, I've discovered a change that needs to be made to my code. In the "resend" case, delete the line that says:
Code: Select all
deleteBounce($row['bounce']);
and replace it with the line:
Code: Select all
Sql_query(sprintf('delete from %s where id = %d',$GLOBALS['tables']['bounce'],$row['bounce']));
Otherwise it will resend the message indefinitely as the deleteBounce() function also removes the entry from the user_message_bounce table, which is used to keep track of the number of times the message was re-sent.

I also suggest adding the following to lines to prevent this mod from affecting the 'sent' and 'bounced' numbers in the message stats:
Code: Select all
            Sql_Query(sprintf('update %s set processed = processed - 1 where id = %s',$tables['message'],$row['message']));
            Sql_Query(sprintf('update %s set bouncecount = bouncecount - 1 where id = %s',$tables['message'],$row['message']));
CS
 

Re: [mod] re-sending soft bounces with advanced bounce handling

Postby bugs31 » 2:44pm, Thu 14 Jan, 2010

Hello
I want to apply this mod to my processbounces.php and lib.php.
However i don't know how to aplly this mod to my processbounces.php
I do like this

Code: Select all
      case 'resend':
         Sql_Query(sprintf('select * from %s where user = %s and message = %s',$tables["user_message_bounce"],$userdata['id'],$row['message']));
         if (!isset($soft_bounce_threshold)) $soft_bounce_threshold = 5;
         if (mysql_affected_rows() <= $soft_bounce_threshold)
            {
               logEvent('User '.$userdata['email'].' message resent by bounce rule '.PageLink2('bouncerule&id='.$rule['id'],$rule['id']));
               $advanced_report .= 'Message resent to user '.$userdata['email'].' by bounce rule '.$rule['id']."\n";
               $advanced_report .= 'User: http://'.$GLOBALS['domain'].$GLOBALS['adminpages'].'/?page=user&id='.$userdata['id']."\n";
               $advanced_report .= 'Rule: http://'.$GLOBALS['domain'].$GLOBALS['adminpages'].'/?page=bouncerule&id='.$rule['id']."\n";
               Sql_Query(sprintf('delete from %s where messageid = %s and userid = %s',$tables["usermessage"],$row['message'],$userdata['id']));
               Sql_Query(sprintf('update %s set status = "inprocess" where id = %s',$tables['message'],$row['message']));
            }
            Sql_query(sprintf('delete from %s where id = %d',$GLOBALS['tables']['bounce'],$row['bounce']));
            Sql_Query(sprintf('update %s set processed = processed - 1 where id = %s',$tables['message'],$row['message']));
            Sql_Query(sprintf('update %s set bouncecount = bouncecount - 1 where id = %s',$tables['message'],$row['message']));
         break;


or like this

Code: Select all
      case 'resend':
         Sql_Query(sprintf('select * from %s where user = %s and message = %s',$tables["user_message_bounce"],$userdata['id'],$row['message']));
         if (!isset($soft_bounce_threshold)) $soft_bounce_threshold = 5;
         if (mysql_affected_rows() <= $soft_bounce_threshold)
            {
               logEvent('User '.$userdata['email'].' message resent by bounce rule '.PageLink2('bouncerule&id='.$rule['id'],$rule['id']));
               $advanced_report .= 'Message resent to user '.$userdata['email'].' by bounce rule '.$rule['id']."\n";
               $advanced_report .= 'User: http://'.$GLOBALS['domain'].$GLOBALS['adminpages'].'/?page=user&id='.$userdata['id']."\n";
               $advanced_report .= 'Rule: http://'.$GLOBALS['domain'].$GLOBALS['adminpages'].'/?page=bouncerule&id='.$rule['id']."\n";
               Sql_Query(sprintf('delete from %s where messageid = %s and userid = %s',$tables["usermessage"],$row['message'],$userdata['id']));
               Sql_Query(sprintf('update %s set status = "inprocess" where id = %s',$tables['message'],$row['message']));
               Sql_Query(sprintf('update %s set processed = processed - 1 where id = %s',$tables['message'],$row['message']));
               Sql_Query(sprintf('update %s set bouncecount = bouncecount - 1 where id = %s',$tables['message'],$row['message']));
            }
            Sql_query(sprintf('delete from %s where id = %d',$GLOBALS['tables']['bounce'],$row['bounce']));
         break;


My problem is with these two line

Code: Select all
            Sql_Query(sprintf('update %s set processed = processed - 1 where id = %s',$tables['message'],$row['message']));
            Sql_Query(sprintf('update %s set bouncecount = bouncecount - 1 where id = %s',$tables['message'],$row['message']));


Where can i put these two line ?

Thanks
bugs31
phpList newbie
 
Posts: 2
Joined: 2:34pm, Thu 14 Jan, 2010

Re: [mod] re-sending soft bounces with advanced bounce handling

Postby CS2 » 3:46pm, Thu 14 Jan, 2010

They should probably be included like this:
Code: Select all
if (mysql_affected_rows() < $soft_bounce_threshold)
{
  Sql_Query(sprintf('update %s set processed = processed - 1 where id = %s',$tables['message'],$row['message']));
  Sql_Query(sprintf('update %s set bouncecount = bouncecount - 1 where id = %s',$tables['message'],$row['message']));
}


That way, if the user reaches the full bounce threshold, it will show the correct number.
CS2
PL Master
 
Posts: 216
Joined: 2:20am, Wed 04 Feb, 2009

Re: [mod] re-sending soft bounces with advanced bounce handling

Postby bugs31 » 6:21pm, Thu 14 Jan, 2010

Thank you
bugs31
phpList newbie
 
Posts: 2
Joined: 2:34pm, Thu 14 Jan, 2010

Re: [mod] re-sending soft bounces with advanced bounce handling

Postby RodrigoDC » 4:27pm, Fri 04 Jun, 2010

Hello, thanks for this mod. However, I'm either too stupid or the instructions to implement it are bit confusing, because I'm not able to execute it.

:oops:

Here's a step by step of what I did:

1. Modified config.php as follows: added
Code: Select all
# Allows re-sending messages to soft-bounced e-mails
# Define how many times it will resend a message on a soft bounce.
$soft_bounce_threshold = "2";


2. Modified admin lib.php by adding
Code: Select all
 'resend' => $GLOBALS['I18N']->get('Resend message to user and delete bounce'), // This is a custom bounce rule


after
Code: Select all
'deletebounce' => $GLOBALS['I18N']->get('delete bounce'),


3. Modified admin/processbounces.php by adding
Code: Select all
      case 'resend':
              # This is a custom bounce rule for 'soft bounces' which will modify the database to make it appear as if the message has not yet been sent to the user and also makes sure the message still has the status "inprocess".
              # In order to actually resend the message, the queue will have to be processed again.
              # $soft_bounce_threshhold is a new variable and should be set in the config file along with a comment explaining what it is for.
              # It defines how many times PHPlist will attempt to send the same message to the same user.
                Sql_Query(sprintf('select * from %s where user = %s and message = %s',$tables["user_message_bounce"],$userdata['id'],$row['message']));
                if (!isset($soft_bounce_threshold)) $soft_bounce_threshold = 5;
                if (mysql_affected_rows() <= $soft_bounce_threshold)
                {
                 logEvent('User '.$userdata['email'].' message resent by bounce rule '.PageLink2('bouncerule&id='.$rule['id'],$rule['id']));
                 $advanced_report .= 'Message resent to user '.$userdata['email'].' by bounce rule '.$rule['id']."\n";
                 $advanced_report .= 'User: http://'.$GLOBALS['domain'].$GLOBALS['adminpages'].'/?page=user&id='.$userdata['id']."\n";
                 $advanced_report .= 'Rule: http://'.$GLOBALS['domain'].$GLOBALS['adminpages'].'/?page=bouncerule&id='.$rule['id']."\n";
                 Sql_Query(sprintf('delete from %s where messageid = %s and userid = %s',$tables["usermessage"],$row['message'],$userdata['id']));
                 Sql_Query(sprintf('update %s set status = "inprocess" where id = %s',$tables['message'],$row['message']));
                }
              Sql_query(sprintf('delete from %s where id = %d',$GLOBALS['tables']['bounce'],$row['bounce']));
                break;


after
Code: Select all
        case 'unconfirmuser':
          logEvent('User '.$userdata['email'].' unconfirmed by bounce rule '.PageLink2('bouncerule&id='.$rule['id'],$rule['id']));
          Sql_Query(sprintf('update %s set confirmed = 0 where id = %d',$GLOBALS['tables']['user'],$row['user']));
          $advanced_report .= 'User '.$userdata['email'].' made unconfirmed by bounce rule '.$rule['id']."\n";
          $advanced_report .= 'User: http://'.$GLOBALS['domain'].$GLOBALS['adminpages'].'/?page=user&id='.$userdata['id']."\n";
          $advanced_report .= 'Rule: http://'.$GLOBALS['domain'].$GLOBALS['adminpages'].'/?page=bouncerule&id='.$rule['id']."\n";
          addUserHistory($userdata['email'],$GLOBALS['I18N']->get("Auto Unsubscribed"),$GLOBALS['I18N']->get("User auto unsubscribed for")." ".$GLOBALS['I18N']->get("bounce rule").' '.$rule['id']);
          addSubscriberStatistics('auto unsubscribe',1);
          break;


4. My particular soft bounce was due to the fact that my host forgot to increase the max e-mails per hour allowed, and hence many bounced. I thus went to Admin > Manage Bounces > generate bounce rules, but since none of the candidate rules applied, I created a new rule, namely:

Domain xxxxx.com has exceeded the max emails per hour (100) allowed. Message discarded


Selected the new action: "Resend message to user and delete bounce" and activated it.

5. I clicked on Admin > Manage Bounces > generate bounce rules again, and then click on Admin > process queue, but nothing happened (actually I got the "Finished, Nothing to do" message).

Am I off the mark? How do I actually manage to resend a previous e-mail to all those folks who bounced simply because the host forgot to increase the limit? (My host has since increased the limit).

Any help to point me in the right direction would be greatly appreciate it.

Thanks,
Rodrigo
RodrigoDC
phpList newbie
 
Posts: 4
Joined: 4:52am, Fri 25 Feb, 2005

Next

Return to Contributions: Plug-ins, Add-ons, Mods

Who is online

Users browsing this forum: No registered users and 2 guests