Throttled Sending Instead of Batches

Solutions for other advanced phplisters

at wit's end

Postby smoghat » 7:45am, Tue 04 Oct, 2005

ok, i've been trying to deal with this for 12 hours now!! this is no good. my server throttles at 500 emails an hour. officially anything about receives an unresolvable domain errors.

i've patched the code as suggested. looks good!

i've set it up to send an email once every 7.5 seconds or 8 emails per minute, 480 per hour. indeed, phplist agrees (more or less) saying that it sending roughly 455 msgs/hr. after a time, the messages stop incrementing. i go into message view, suspend sending, requeue and then process quoue.

everything seems fine until, after some delay, messages begin pouring back. yes, i should do a control and find out at what point they come pouring back, but when i have an email box stuffed with 100s of incoming messages, i've got other worries to deal with!

any ideas of what I can do? this is a most unpleasant debug, involving hitting some members of the list repeatedly AND the thing takes hours at a time to check... grr...
smoghat
phpList newbie
 
Posts: 1
Joined: 7:34am, Tue 04 Oct, 2005

Hope for clarification

Postby Jinson » 3:09am, Sun 25 Dec, 2005

jotazzu wrote:Hi,
PHPlist is able to handle such parallel invocations of processes to some extend. If a newly started process finds a running process it waits up for 20 seconds.
You will see messages like:
- A process for this page is already running and it was still alive xxx seconds ago
- Sleeping for 20 seconds, aborting will now quit

If still the previous process is running after 220s the new one finishes itself.
You will see a message like:
- We've been waiting too long, I guess the other script is still going ok

But when the previous process started more than 1200s ago and still is marked as running, the old one is killed without notice and the new one starts to process. This is the programming for v2.8.12 as you can find it at function getPageLock() in the file lib.php.

As you describe your scenario I assume that you are using throttling. The amount of 1000 emails you are sending in 5 hours is quite low. If your email is not bigger than 30kB any method which may be used with cron (PHP-cli, PHP-cgi, lynx, wget, GET, ...) should still be active after that time.

When the new cron job starts after 5 hours it will kill the previous job because the old one started more than 1200s ago. The new one takes over the processing of the queue. This is a completely new start and it continues to send the emails where the previous was stopped. If you added a new message meanwhile this is handled as if both messages are new. So there should be no difference to start both messages at the same time or to add the second later.

But don't use MAILQUEUE_BATCH_SIZE > 0 with your scenario. Only the first batch period will be executed before the process finishes.



Hi jotazzu and drubarth, I'm abit confused on this batch and throttle issue. My setup is described here. And based on your above post, I've changed my settings to

Code: Select all
define("MAILQUEUE_BATCH_SIZE",0);
define("MAILQUEUE_BATCH_PERIOD",0);
define('MAILQUEUE_THROTTLE',6);


Note that my webhost has a limit of 600 emails per hour, I only have the CGI version of php available, safe mode is off, max_execution_time is set to 30. I setup this cron to run every 15 minutes.

Code: Select all
php /home/accountname/public_html/phplist/admin/index.php page=processqueue login=username password=password reload=0 >/dev/null



So with my new settings, batch mode disabled and throttling on for 6 seconds between each mail, I'm assuming that this is what will happen.....
------------
1. I create a newsletter to send to multiple lists totalling 50,000 members.

2. I submit it to the send messages queue on Monday 7:50am.

3. The cron job at 8am starts work and starts sending the messages.

4. By 8:15am, that process of php invoked by the 8am cron job has only
managed to send 150 messages. So, 49,850 messages left to go.

5. At 8:15am, another process called by the 8:15am cron starts up. Because of this line
Code: Select all
if ($running_res[0] > 600)

in the pagelock function for 2.10.2, it'll terminate that old 8am cron job's process because that old one was started more than 10 minutes ago and then the new process will go to work.

6. That new process will then continue sending emails starting from the 49,850th and continue until 8:30am, leaving 49,700 for the next cron job, and the process repeats.

7. I decide to make and add another newsletter to be sent to 5000 out of that 50,000 subscribers. I add the new message to be sent to the queue at 8:50am. The 9am cron job picks up the addition 5k message and adds them to the then 49,600 messages totalling 54,600 messages to send.

8. My emails will be completely sent out after ~ 4 RL days.

==============

My Question
I'm guessing that the process queue cron job might terminate after 10 minutes (600 seconds), instead of continuing until the next cron job terminates it because processqueue.php sets this -> set_time_limit(600); for the message processing?
Jinson
PL Nut
 
Posts: 35
Joined: 8:54am, Sun 13 Nov, 2005
Location: Singapore

Postby jotazzu » 11:17pm, Mon 26 Dec, 2005

This post relates to version 2.10.2 and is also valid for v2.8.12 with minor changes. I assume that the CGI interface of PHP or a text browser like lynx or wget are used from a cron. Also the batch mode of PHPlist may not be enabled.

Hi Jinson,
I suppose that you read the value of max_execution_time (30s) from the output of the phpinfo() function. That is the default max_execution_time a script uses. But if safe_mode is off the script can change this value and PHPlist changes it to 600s as you have written.
This is only the execution time the code of the script needs by itself. Time for throttling and time for system calls like hard disc access or calling the mail system does not count.

Assuming that one mail needs half a second of pure script execution time ( which is quite much), about 1200 mails will be sent from one process. Adding 0.25s for time spend in the system and with a throttling of 6s the whole process will live (1200 * 0.75s) + (1200mails * 6s) = 8100 seconds. It is sufficient to schedule cron jobs every two and a half hour.
You may check the time of the 'Maillist Processing info' mails which PHPlist sends to you after every cron schedule. The dates in the mail cover start and end time of that schedule.

Code: Select all
if ($running_res[0] > 600)
relates to the time since the last modification of the SQL record which monitors a PHPlist queue process. Each processing of an email calls the function keepLock() which increases the 'alive' counter and implicit updates the modification date. Thus this time mirrors inactivity of the process, not the date when it was started.

If you schedule a new cron job every 15 minutes most of them will be killed because the first job is still running and active. If you change >/dev/null at the end of your cron command to >snapshot.html and open this file in a javascript enabled browser 220s after the start of follow crons you should see messages like
- We've been waiting too long, I guess the other script is still going ok

The last question is, if it is better to schedule the new cron before the old is done or to wait to be sure the previous cron has finished.
Basically there is a risk that the processing timeout hits exactly between the short moment when the mail is sent but the SQL database is still not updated. This is a very short moment but with a lot of schedules this will happen from time to time. As a result one receiver will get the mail two times.

If you schedule new crons before the old is done your follow cron will be killed after 220s if started to early or may be right there, but nevertheless will get to work only when the old one HAS FINISHED. Both does not solve the rare problem of doubled mails. So it is probably the best way to be sure that no process is running before starting the next. If started to early the time until the next regular cron schedule might become long. And to start crons every 5 minutes does not sound very efficient.

FYI: the script loops through each message to send in the order when they were entered. Thus at first your 50.000 mails from the first message are sent, then the second starts to follow about 80 hours later.

As batch mode is off, you may delete reload=0 in the cron command.

Best regards
Jochen
jotazzu
PL Master
 
Posts: 183
Joined: 6:31pm, Wed 13 Jul, 2005
Location: Hamburg, Germany

Postby Balitas » 4:00pm, Fri 29 Feb, 2008

jotazzu wrote:Why not use a slighlty changed patch?

Add to .../phpList/config/config.php:
Code: Select all
# Throttle sending messages; indicates the number of seconds to pause
# between each message (fractions are allowed, e.g. 0.3)
define("MAILQUEUE_PROCESSING_THROTTLE",0.5);


Add to .../phpList/admin/processqueue.php
Find the following at about line 420:
Code: Select all
$status = Sql_query("update {$tables['message']} set processed = processed +1 where id = $messageid");


Insert after this line:
Code: Select all
usleep((MAILQUEUE_PROCESSING_THROTTLE)*1000000);


Your suggestions works fine! but you forget to mention ed this:

Replace in .../phpList/config/config.php this:

Code: Select all
# to avoid overloading the server that sends your email, you can add a little delay
# between messages that will spread the load of sending
# you will need to find a good value for your own server
# value is in seconds (or you can play with the autothrottle below)
define('MAILQUEUE_THROTTLE',0);


by this

Code: Select all
# Throttle sending messages; indicates the number of seconds to pause
# between each message (fractions are allowed, e.g. 0.3)
define("MAILQUEUE_PROCESSING_THROTTLE",0.5);


And works! regards :D
Need a website? Enter to www.balitas.com
Balitas
PL Geek
 
Posts: 82
Joined: 2:17am, Mon 13 Feb, 2006

Re: Throttled Sending Instead of Batches

Postby snyperthugs » 11:51pm, Mon 24 May, 2010

OK I really need help!! lol I Need phplist to send out 1 email per minute. (my hosts rules are under 30 email per minute, and under 100 emails per hour) I thought I set all my settings right in the config.php and I also set it for cronjobs. But after sending a test message to a small group I get this message....

Started
Sending in batches of 85 emails
Processing has started, 1 message(s) to process.
Processing message 1
Looking for users
Found them: 18 to process
Processed 18 out of 18 users
Script stage: 5
18 messages sent in 19.14 seconds (3385 msgs/hr)


So does this mean it would send 3385 email per hour??? and 18 message sent in 19sec.??? I need that to be more like 15 - 18min. not sec.
snyperthugs
phpLister
 
Posts: 6
Joined: 6:58am, Tue 13 May, 2008

Re: Throttled Sending Instead of Batches

Postby Balitas » 10:38pm, Thu 04 Nov, 2010

snyperthugs wrote:OK I really need help!! lol I Need phplist to send out 1 email per minute. (my hosts rules are under 30 email per minute, and under 100 emails per hour) I thought I set all my settings right in the config.php and I also set it for cronjobs. But after sending a test message to a small group I get this message....

Started
Sending in batches of 85 emails
Processing has started, 1 message(s) to process.
Processing message 1
Looking for users
Found them: 18 to process
Processed 18 out of 18 users
Script stage: 5
18 messages sent in 19.14 seconds (3385 msgs/hr)


So does this mean it would send 3385 email per hour??? and 18 message sent in 19sec.??? I need that to be more like 15 - 18min. not sec.


the answer of your question is yes, the average speed was 3385 msgs/hr so your phplist send mail to you 18 users in 19.14 seconds....
Need a website? Enter to www.balitas.com
Balitas
PL Geek
 
Posts: 82
Joined: 2:17am, Mon 13 Feb, 2006

Re: Throttled Sending Instead of Batches

Postby Balitas » 2:39pm, Fri 12 Nov, 2010

I noticed in the last version of phplist this hack not works....the speed increase without control and the phplist stop after 1 hour....please advice....
Need a website? Enter to www.balitas.com
Balitas
PL Geek
 
Posts: 82
Joined: 2:17am, Mon 13 Feb, 2006

Re: Throttled Sending Instead of Batches

Postby Aaminah1978 » 12:56pm, Thu 15 Sep, 2011

As I am actually busy in organising a system dealing with simutaneous Cron Jobs for sending I just wonder whether or not this board is still viewed at ?
Aaminah
Aaminah1978
phpLister
 
Posts: 5
Joined: 12:03pm, Wed 10 Aug, 2011

Re:

Postby Aaminah1978 » 3:08pm, Thu 15 Sep, 2011

SamIAM wrote:Thanks for the great info.

I wonder, if i can change that 1200s number to make it much much larger, like 5 hours worth. Or maybe, i can make it where it will only start the new message send ONLY after the first one finished.

It takes me that long, becuase my host has a 200 emails/ hour.

Thanks again for the info.




and now ... after a bit of time ... how many are you submitting nowadays ;-) ?
Have a nice day Aaminah
Aaminah1978
phpLister
 
Posts: 5
Joined: 12:03pm, Wed 10 Aug, 2011

Previous

Return to Advanced Answers, Howtos, Tips & Tricks

Who is online

Users browsing this forum: No registered users and 1 guest