Bounced Messages not appearing.

[billy4eggs]

Hi,

I have installed and tested my installation of PHPlist and my system configuration is:-

phplist version: 2.10.19
PHP version: 5.2.17
Browser: Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20100101 Firefox/17.0
Webserver: Apache
Website: borneodream.com/lists
Mysql Info: 5.5.27-28.1-log
PHP Modules (Modules are also confirmed here http://borneodream.com/i.php ):-

date
libxml
openssl
pcre
zlib
bcmath
bz2
calendar
ctype
curl
dba
dom
hash
filter
ftp
gd
gettext
session
iconv
standard
json
ldap
mbstring
mcrypt
mhash
mysql
SimpleXML
ncurses
pcntl
SPL
PDO
pdo_mysql
pdo_pgsql
pdo_sqlite
pgsql
posix
pspell
readline
Reflection
imap
shmop
mysqli
soap
sockets
SQLite
exif
tidy
tokenizer
wddx
xml
xmlreader
xmlrpc
xmlwriter
xsl
zip
cgi
bz2_filter
http
mailparse
memcache
sdo
stats
xattr
SourceGuardian
ionCube Loader
Zend Optimizer

And have set-up a bounce email address of listbounces@borneodream.com which I can send to via a standard desktop email program and also by clicking on 'process bounces' it retrieves the message I sent.

But, on sending messages from PHPlist to a list containing known bogus emails non are sent to the bounced account even though they are bogus.

When you set-up a message and send it against a working list that may have a few rouge emails within it you see the bounce codes in the event log but still nothing is sent to list-bounce account.

I look forward to your comments and help.

Thanks
Billy

[JacobN]

Hey Billy,

You mentioned that you set up your bounce email address already, did you just create this email account with your web host, or did you also remember to un-comment and add it in your /config/config.php file?

By default, phpList is going to set the From: header of your messages to what you have entered in the From Line: field on the send a message page.

When an email bounces, it's going to try to bounce to the Return-Path: address in your headers. I'm guessing if you take a look at the messages you're currently sending out, this is set to username@server.WebHost.tld. This is because with the way PHP handles email, it authenticates via the web server user instead of with an actual email account. Now if you're on a host that runs cPanel or similar type control panels, you might be able to log directly into webmail with your main cPanel username (not an email account) and see where all your bounces have been ending up.

You could just keep checking it that way manually if it works for you, but more than likely you'll want to edit your /config/config.php file to reflect your listbounces@ address, and then also setup bounce processing so phpList can handle the bounces for you.

Open up your /config/config.php script and look for this line:

Code: Select all

# $message_envelope = 'listbounces@yourdomain';

Replace it with:

Code: Select all

$message_envelope = 'listbounces@borneodream.com';

Making sure to take the pound # symbol off of the front to un-comment the line.

That should allow you to see the bounce backs by checking that email account directly. But if you'd like to be able to click on the view bounces link in phpList and have process bounces work properly you'll also want to replace this section in your config file:

Code: Select all

$bounce_mailbox_host = 'localhost';
$bounce_mailbox_user = 'popuser';
$bounce_mailbox_password = 'password';

With something along these lines:

Code: Select all

$bounce_mailbox_host = 'mail.borneodream.com';
$bounce_mailbox_user = 'listbounces@borneodream.com';
$bounce_mailbox_password = 'YourPasswordHere';

You can even setup a cronjob to automate bounce processing for you, or change from using POP authentication to Mbox if it doesn't work for you. I'd recommend taking a look at the full phpList documentation on configuring bounces for a clearer picture of how it all ties together.

Hope that works out for you, let me know how it goes!

[JacobN]

Hey Billy,

Sorry I read your question again and I might have misunderstood you the first time. I missed the part where you said you are able to click Process Bounces. So it sounds like you might already have your /config/config.php script setup with:

Code: Select all

$message_envelope = 'listbounces@borneodream.com';

You're probably also using:

Code: Select all

$bounce_protocol = 'pop';

Finally also:

Code: Select all

$bounce_mailbox_host = 'mail.borneodream.com';
$bounce_mailbox_user = 'listbounces@borneodream.com';
$bounce_mailbox_password = 'YourPasswordHere'

When you click on Process Bounces in phpList, it makes a POP connection to your listbounces@borneodream.com email account, and it downloads the messages off of the email server. So if you're trying to go check the email account directly, there is nothing there.

If neither of these cases are helpful to you, please clarify a bit what exactly you have setup in terms of your bounce settings, and I can try to lead you in the right direction.

[billy4eggs]

Hi Jacob,

Many thanks for answering my post.

What I do know or what works:-

Using standard desktop email program such as MS Outlook I can send and receive a email to my listbounces@borneodream.com account. Also, I can send an email to this account from MS Outlook and collect with PHPlist using the 'process bounces' link.

During a recent mailing of 3000 email addresses I saw in the event log about 86 bounces and undeliverable emails but not one was passed to the listbounces@borneodream.com account for PHPlist to process.

I have tested this by sending a message using PHPlist to a test list of users where all the email accounts are bogus and not one is sent to the listbounces@borneodream.com account

Code: Select all

=========================================================================

Settings for handling bounces

=========================================================================

*/

# Message envelope. This is the email that system messages come from
# it is useful to make this one where you can process the bounces on
# you will probably get a X-Authentication-Warning in your message
# when using this with sendmail
# NOTE: this is *very* different from the From: line in a message
# to use this feature, uncomment the following line, and change the email address
# to some existing account on your system
# requires PHP version > "4.0.5" and "4.3.1+" without safe_mode
$message_envelope = 'listbounces@borneodream.com';

# Handling bounces. Check README.bounces for more info
# This can be 'pop' or 'mbox'
$bounce_protocol = 'pop';

# set this to 0, if you set up a cron to download bounces regularly by using the
# commandline option. If this is 0, users cannot run the page from the web
# frontend. Read README.commandline to find out how to set it up on the
# commandline
define ("MANUALLY_PROCESS_BOUNCES",1);

# when the protocol is pop, specify these three
$bounce_mailbox_host = 'mail.borneodream.com';
$bounce_mailbox_user = 'listbounces@borneodream.com';
$bounce_mailbox_password = 'XXXXXXX';

# the "port" is the remote port of the connection to retrieve the emails
# the default should be fine but if it doesn't work, you can try the second
# one. To do that, add a # before the first line and take off the one before the
# second line

$bounce_mailbox_port = "110/pop3/notls";
#$bounce_mailbox_port = "110/pop3";

# when the protocol is mbox specify this one
# it needs to be a local file in mbox format, accessible to your webserver user
$bounce_mailbox = '/var/spool/mail/listbounces';

# set this to 0 if you want to keep your messages in the mailbox. this is potentially
# a problem, because bounces will be counted multiple times, so only do this if you are
# testing things.
$bounce_mailbox_purge = 1;

# set this to 0 if you want to keep unprocessed messages in the mailbox. Unprocessed
# messages are messages that could not be matched with a user in the system
# messages are still downloaded into PHPlist, so it is safe to delete them from
# the mailbox and view them in PHPlist
$bounce_mailbox_purge_unprocessed = 1;

# how many bounces in a row need to have occurred for a user to be marked unconfirmed
$bounce_unsubscribe_threshold = 5;


/*

The above is taken from my config.php file.

And this is what the headers on my messages look like when sent from PHPlist:-

Code: Select all

To: billy4eggs@gmail.com
Subject: Borneo Dream
Date: Tue, 1 Jan 2013 01:17:23 -0600
From: "noreply@borneodream.com" <noreply@borneodream.com>
Message-ID: <363038da87f7bee9eb6b13bd2034cdce@borneodream.com>
X-Priority: 3
X-Mailer: PHPMailer [version 1.73]
X-Mailer: phplist v2.10.19
X-MessageID: 24
X-ListMember: billy4eggs@gmail.com
Errors-To: listbounces@borneodream.com
MIME-Version: 1.0
Content-Type: multipart/related;

I hope that helps

Thanks
Billy

[JacobN]

Hello again Billy,

No problem at all with the help.

I see that you do have a Errors-To: header, but seem to still be lacking the Return-path: or envelope-from ones.

Depending on how your web host has PHP, and their mail server configured, this is probably why you are still not getting any bounce back messages into the listbounces@ address.

It's mentioned in the notes in the config.php file right above where you are setting the $message_envelope value, that PHP might need to have safe_mode disabled in order to add that header.

So my best guess would be that, or the way your web host has their email server configured. Can you request for them to take a look at the mail logs for you and see where the bounces might ultimately be trying to end up currently?

I noticed you are on a cPanel host, so more than likely you should be able to login directly to your webmail via the URL http://borneodream.com/webmail. Login with your cPanel username and password instead of an email account. You might get a warning in cPanel about being logged in as the cPanel user, but go ahead and click on one of your webmail clients. You should hopefully be able to see all the bounce backs there.

If you aren't seeing any bounce backs there, then you'd definitely want to reach out to your web host for an explanation of what's going on.

Hope that helps!

- Jacob

[billy4eggs]

Hi,

Yes all the bounced emails are ending up in the default account to the cPanel account which is scubadi2@borneodream.com that can not be accessed by POP only via WebMail.

I have contacted the web server support team for guidance on this Return-Path: statement and whether they can suggest alternatives or allow PHPlist to modify the email headers to add this statement.

Thanks
Billy

[billy4eggs]

Hi,

I have had some feedback from my hosting company SiteGround.com and they want to know exactly how and what PHPlist adds to the email headers so that they can better understand what is stopping the Return-Path: statement from being inserted.

Thanks
Billy

[billy4eggs]

Hi,

Sorry another odd question... Even though the From: address is valid why is the bounced email not being return to there?

Thanks
Billy

[JacobN]

Hey Billy,

That's good that you're getting the bounce backs in your main cPanel account. I believe if worse comes to worse, and you can't get the headers properly functioning, you might be able to setup a cronjob to just move them over into the correct mailbox.

This would be a last ditch resort, and I haven't tested this theory, but I believe it would look something like this:

Code: Select all

1 */4 * * * mv /home/scubadi2/mail/new/* /home/scubadi2/mail/borneodream.com/listbounces/

Essentially that will run a cron job every 4 hours and move any new messages from the main cPanel account's mail folder, into your listbounces accounts so that then phpList can check it via POP.

Of course that is, if your web host is running the newer maildir mailbox format. If they are still using mbox, then I think you might even be able to just update your config to point to the cPanel user's mbox file with the $bounce_mailbox setting. Also you'd want to make sure to change $bounce_protocol from pop to mbox.

In regards to how phpList appends the Return-Path header, it seems to be happening at the sendmail level. When I review my Exim mail logs on the server this is the full transaction I see taking place:

Code: Select all

cwd=/home/userna5/public_html/list/admin 6 args: /usr/sbin/sendmail -t -i -oi -f listbounces@domain.tld
1TqEHn-0006Jx-V8 <= listbounces@domain.tld T="Test" for a@domain.tld
1TqEHn-0006Jx-V8 ** a@domain.tld R=virtual_aliases: No Such User Here"
1TqEHo-0006KC-5k <= <> T="Mail delivery failed: returning message to sender" for listbounces@domain.tld

cwd=/home/userna5/public_html/list/admin 6 args: /usr/sbin/sendmail -t -i -oi -f listbounces@domain.tld
1TqEHo-0006KC-5k => listbounces <listbounces@domain.tld> R=virtual_user T=virtual_userdelivery

Those first 4 lines are showing the current working directory is my /list/admin/ directory, and then it's calling sendmail with the important argument being -f listbounces@domain.tld so that the Return-Path and envelope-from headers match up. Next you'll notice <= listbounces@domain.tld, this is Exim actually seeing the delivery as coming from this email user, instead of <= scubadi2@server.WebHost.tld which you're probably seeing in your mail logs at the moment.

phpList by default is using the PHP mail() function, which calls sendmail. If there is a problem modifying the headers this way on the particular server you're on, you can also try to use the PHPmailer options in your config.php script:

Code: Select all

# To use a SMTP please give your server hostname here, leave it blank to use the standard
# PHP mail() command.
define("PHPMAILERHOST",'');

Just fill in the blank with mail.borneodream.com, then move onto the section below:

Code: Select all

# if you want to use smtp authentication when sending the email uncomment the following
# two lines and set the username and password to be the correct ones
#$phpmailer_smtpuser = 'smtpuser';
#$phpmailer_smtppassword = 'smtppassword';

I'm sure you can probably guess what to do here, un-comment the two lines by removing the pound # symbols, then enter in your mail credentials.

The From: header in email is very different then the actual account on the server responsible for originating that message. It can be easily spoofed or faked, so the Return-Path: header gives a more accurate picture of where a message really came from.

It's just the way that email is setup to operate, so that when a server encounters a bounce it doesn't rely on weak From: header information and start reverse spamming someone. For example if I spoofed my From: header to be president@whitehouse.gov, and then purposely sent to a million bogus email accounts, if those mail servers just bounced right back to the From: header this would give anyone the power to spam blast the president all day long.

Hopefully with one of those methods it will start working for you. Good luck, and let me know if any of them click!

- Jacob