Throttled Sending Instead of Batches

[drubarth]

So did the messages all get sent correctly?

[silverbytes]

No...
And don't understand that either. Eventlog say 1 message was sent at time if I understand right and it seems to be executing every 5 mins...

But I wonder if upgrading to 2.9 would help. It seems to have a domain throttle feature, anybody tried it?

[waznu]

I am still only getting the first 200 messages to process with this set up(followed from posts) and an inprocess message stuck and never moving. please help!?!

thank you, Deborah

[jotazzu]

Hi,

I am working for some days on this topic too. PHPlist 2.8.12 has a problem with batch sending due to a tuning feature in processqueue.php which makes trouble in different situation. This feature might only be necessary if sending 5000+ emails in total.

Today or tomorrow I will post a summarize of cron handling with three workarounds to this problem and how to disable that feature.

Jochen

[drubarth]

phpList has incorporated my mod (didn't look to see if it was exactly the same). So make sure you're not duplicating.

If you hit a limit such as 200, then either your web host is limiting you, or the batch limit is set, or you're not running this using php-cli and you're hitting a timeout (first two are more likely).

Hope this helps give you some direction.

Cheers,

[jotazzu]

Hi Darryl,

its not the throttling, it's the slicing of the MySQL query which causes some problems in 2.8.12.

In version 2.9.5 Michiel has disabled this slicing, so these problems will not occur in future.

I hope to get the post online within the next 2-3 hours.

Ciao,
Jochen

[drubarth]

Thanks for the info. Slicing must be one of the new features added that I haven't looked at yet. What is the concept of slicing?

Cheers,

[jotazzu]

I've my post published. There is also a separate post which explains the problem in detail.

Three and a half solutions for batch/cron processing

This is a quite long topic. Hope you find some interesting things. It was hard work to put it all together.

Jochen

[orangechicken]

# Throttle sending messages; indicates the number of milliseconds to pause
# between each message (millisecond is one-millionth of a second)
define("MAILQUEUE_PROCESSING_THROTTLE",300000);

Minor correction here. This number is microseconds, not milliseconds. A microsecond is a millionth of a second (while a millisecond is, of course, a thousandth).

Edit: And it pays to read the whole thread before pointing something out that's already been pointed out.

Now to actually get my install of phplist to actually *send* emails....[/i]

[bumbum]

I think the throttle is a great feature. At first I found it a bit difficult to calculate the value I wanted so I've added in a couple of comment lines to help others find a suitable value.

# Throttle sending messages; indicates the number of microseconds to pause
# between each message (microsecond is one-millionth of a second)
# Use this formula to calculate a suitable value.
# ((Number of hours required to send emails x 3600000000)/Number of emails)
# Example: To send out 12000 e-mails in 48 hours the formula would be:
# ((48 x 3600000000)/12000) answer 14400000 microseconds to complete task
define("MAILQUEUE_PROCESSING_THROTTLE",14400000);

In the example shown, the user has 12000 subscribers on their mailing list and wishes to send the emails over a 48 hour period. The result is one e-mail being sent every 14400000 microseconds, or 1 email every 14.4 seconds.
This is particularly helpful if you've got a big list on a shared host server. It will help you to avoid hogging the server's resources and having your account suspended or terminated by the host administrators.
These are the settings I use on my own mailing list.

[jotazzu]

Why not use a slighlty changed patch?

Add to .../phpList/config/config.php:

Code: Select all

# Throttle sending messages; indicates the number of seconds to pause
# between each message (fractions are allowed, e.g. 0.3)
define("MAILQUEUE_PROCESSING_THROTTLE",0.5);


Add to .../phpList/admin/processqueue.php
Find the following at about line 420:

Code: Select all

$status = Sql_query("update {$tables['message']} set processed = processed +1 where id = $messageid");


Insert after this line:

Code: Select all

usleep((MAILQUEUE_PROCESSING_THROTTLE)*1000000);


[SamIAM]

I've got a question, what happens when the cron job excutes again during a cron job still processing ?

for example , its sending 2000 emails at 200 emails an hour, the cron job runs every 5 hours. Now the job will take more than 5 hour, what happens when the cron job excutes again ?

also, lets say during that time, I add another email job, what happens ?

sorry too many questions, i'm a newbie

[jotazzu]

Hi,
PHPlist is able to handle such parallel invocations of processes to some extend. If a newly started process finds a running process it waits up for 20 seconds.
You will see messages like:
- A process for this page is already running and it was still alive xxx seconds ago
- Sleeping for 20 seconds, aborting will now quit

If still the previous process is running after 220s the new one finishes itself.
You will see a message like:
- We've been waiting too long, I guess the other script is still going ok

But when the previous process started more than 1200s ago and still is marked as running, the old one is killed without notice and the new one starts to process. This is the programming for v2.8.12 as you can find it at function getPageLock() in the file lib.php.

As you describe your scenario I assume that you are using throttling. The amount of 1000 emails you are sending in 5 hours is quite low. If your email is not bigger than 30kB any method which may be used with cron (PHP-cli, PHP-cgi, lynx, wget, GET, ...) should still be active after that time.

When the new cron job starts after 5 hours it will kill the previous job because the old one started more than 1200s ago. The new one takes over the processing of the queue. This is a completely new start and it continues to send the emails where the previous was stopped. If you added a new message meanwhile this is handled as if both messages are new. So there should be no difference to start both messages at the same time or to add the second later.

But don't use MAILQUEUE_BATCH_SIZE > 0 with your scenario. Only the first batch period will be executed before the process finishes.

[SamIAM]

Thanks for the great info.

I wonder, if i can change that 1200s number to make it much much larger, like 5 hours worth. Or maybe, i can make it where it will only start the new message send ONLY after the first one finished.

It takes me that long, becuase my host has a 200 emails/ hour.

Thanks again for the info.

[robertp3]

How can I determine which mode I am using when PHPlist is installed on a shared hosting service (in this case IpowerWeb).

I need to add the correct cron job to implement the throtte mod in topic below

http://www.phplist.com/forums/viewtopic.php?t=2142