Throttled Sending Instead of Batches

[drubarth]

I've used other mailing programs that allow throttling of messages, which I prefer over the idea of batches in phpList. I love phpList, except for the batch processing.

So for those who are interested, here is a mod that allows you to throttle sending of messages.

Add to config/config.php file in the batch configuration section

# Throttle sending messages; indicates the number of milliseconds to pause
# between each message (millisecond is one-millionth of a second)
define("MAILQUEUE_PROCESSING_THROTTLE",300000);

You'll probably want to turn off the batch processing.

Modify admin/processqueue.php

find the following:

output("Sending $messageid to $useremail");
sendEmail($messageid,$useremail,$userhash,$htmlpref,$rssitems);

add after this the following line:

usleep(MAILQUEUE_PROCESSING_THROTTLE);

Be forewarned that this has not been tested much. I'm only using this in the command line interface mode, so be aware that it may have an impact if you're running the Process Queue from a browser due to the browser window needing to be open longer.

Please let me know if this works for you, or doesn't. I'm posting it here first to get some feedback, and if it looks good I'll join the developers list and propose the code as an added feature to the application.

[cecsystems.net]

Will this fire off from the process queue on the admin interface? I can't get cron to work to save my skin. edited phplist and everythig just keep getting No input file specified?

[drubarth]

Yes, this should work from the admin interface, but you may still have to deal with the browser window timing out, as seems to be the problem for a lot of people. Please let me know if you do have an issue or not. If you do, you may want to set the batch size as well so that the server keeps communicating with the browser.

[cecsystems.net]

I don't suppose I could bother you for your cron example that works?

[drubarth]

Here's what I execute for my cron job.

Code: Select all

/usr/bin/php /html/lists/admin/index.php -pprocessqueue -c/html/lists/config/config.php


It requires that you have the php-cli version, not cgi, and that your executing user is defined in your config file.

Hope that helps. It took me quite a while to get it to work, but that's in part because I was setting it up to execute for multiple domains sequentially and to turn off my outbound mailscanner so it wouldn't create such a load on the server.

Good Luck!

[aaronvegh]

Hi Daryl,
I've been working through your solution to send out a campaign. With your throttling code it seems to run a lot faster, so thanks very much! I look forward to testing it with my next campaign in earnest.

However, I'm looking at the code you show above for executing from cron. Can you please confirm that's correct? The -c switch isn't working. However, it does execute the admin/index.php file no problem, and runs through the list in no time.

THanks,
Aaron.

[drubarth]

The "/html/lists/" part of the command will be different on your system. Make sure the paths are correct for the user running the cron job. Are you running your cron job as root or as some other user? For testing, you can just execute the command at the shell prompt.

Also, did you validate that you're running the php-cli version?

[aaronvegh]

Hi there,
Yes to all. I'm actually not trying it in crontab until I have the command working. The path is correct and the command executes, but it doesn't like the -c switch. If I understand BASH right, the -c switch is being applied by PHP, right? And it'd be looking for the php.ini file at the path set by the -c switch.

Elsewhere, I was instructed to give options to the admin/index.php file differently. Here's what I have:

Code: Select all

php /path/to/my/listserver/lists/admin/index.php page=processqueue
-c /path/to/my/listserver/lists/config/config.php login=***** password=*****


My real question, I guess, is do we really need to attach the config file to this? Doesn't PHPList get that configuration anyway? Or am I missing something?

Cheers!
Aaron.

[drubarth]

The code you show above is if you're running PHP in cgi mode from the command line. This confuses many people, because you have to be running the cli version of PHP, and many people by default have the cgi version installed. At the command line, type "php -v" to see which version you have. It should look something like this:

PHP 4.3.10 (cli) (built: Dec 22 2004 13:36:20)

Most likely you have the cgi version and it's ignoring the config file parameter. I believe you only need it when you're running the cli version. Also with the cli version, you don't need to specify the login info, instead it compares who you are as a unix user to the config file to see if you have permission.

[Daniela]

OK guys, now I'm stuck.
I tried following all the discussions you have on the subject, and even compiled them.
See my post http://www.phplist.com/forums/viewtopic.php?t=2232

But my job stops, after more or less 2300 e-mails, 2 hours.

I keep getting the following info in the event log.

6430 del processqueue 2005-01-18 19:07:08 2355 messages sent and 2282 messages skipped
6431 del processqueue 2005-01-18 19:07:08 Finished this run
6432 del processqueue 2005-01-18 19:07:08 Reload required
6429 del processqueue 2005-01-18 17:36:04 Found them: 68537 to process
6425 del processqueue 2005-01-18 17:36:00 Processing has started, 1 message(s) to process.
6426 del processqueue 2005-01-18 17:36:00 It is safe to click your stop button now, report will be sent by email to daniela.maini@usa.net
6427 del processqueue 2005-01-18 17:36:00 Processing message 20
6428 del processqueue 2005-01-18 17:36:00 Looking for users
6422 del processqueue 2005-01-18 17:21:16 2284 messages sent and 0 messages skipped
6423 del processqueue 2005-01-18 17:21:16 Finished this run
6424 del processqueue 2005-01-18 17:21:16 Reload required
6421 del processqueue 2005-01-18 15:52:04 Found them: 68537 to process
6417 del processqueue 2005-01-18 15:52:00 Processing has started, 1 message(s) to process.
6418 del processqueue 2005-01-18 15:52:00 It is safe to click your stop button now, report will be sent by email to daniela.maini@usa.net
6419 del processqueue 2005-01-18 15:52:00 Processing message 20
6420 del processqueue 2005-01-18 15:52:00 Looking for users

And I have the following info in the message page, in the sent tab

Message info Status Action
From: Mail Publicitario noreply@mailpublicitario.com
Subject: No pierda su oportunidad, descuento promocional.
Entered: 2005-01-18 15:50:32
Embargo: 2005-01-18 15:00:00
inprocess

view
4637 done
edit
delete

It looks like it is in process, but it won't continue.

Can anybody help?

My test e-mail databse had 2000 e-mails, but this one has 68,000 and growing.
I don't seem to find the error.

[drubarth]

You are either hitting a batch limit or your browser is timing out. You need to add to the URL on the parameter "reload=n" where 'n' increments with each batch. Don't ask me why. After all the messages have been sent, you'll need one more for the message to move out of "inprocess".

For more info search this forum for "reload" and "batch processing" and you should find other posts.

good luck!

[Daniela]

Hey Daryl.
Thanks for the fast response.
No I'm not using batch (or at least, that's what I believe).
I'm actually using your procedure to throttle the messages.
Works pretty neat, except, it hangs every 2300 messages more or less (aproximately 2 hours).
I did use your sugestion to have cron jobs reloaded every two hours, and it is working.
But still is half a patch, sin I have +68,000 subscribers (big right?) I had to create 35 reloads (a couple extra just in case).
It is still much better that the batch mode. Really much better.
I checked with my hosting company, and the server doesn't even feel it. I'm sending a message every two secconds. I guess I'll have the server working for 3 days.
If you have any other sugestion, please let me know.
Daniela

[drubarth]

Daniela,

I think you can reduce the number of seconds to a half second delay and probably still be fine, possibly even lower depending on whether your server is scanning outbound messages.

If you have the cli version of PHP, then you'd be better off running it via the command line. To see if you have the cli version type "php -v" at the command line prompt. It will either be 'cgi' or 'cli'. If you can run in the cli mode, then you shouldn't have to do any reloads at all. This is the way I've setup my server.

[MattF]

well, it looks like this may have sent OK. I just tried it on a few messages.

However, I know that my server is limited on the amount I can send per hour...so I don't know if I'm going to get bouncebacks or not...would tweak the delays, but hate to see a browser timeout happen as well...(still haven't got the cron job to work right)

[drubarth]

This works much better when run in the php-cli mode because you don't have to worry about timeouts. I found that aspect very frustrating and am very happy that I now have it working. Working out the throttling is a big added bonus to me since I can better control my server load.