Throttled Sending Instead of Batches

Solutions for other advanced phplisters

Postby drubarth » 7:56am, Wed 20 Jul, 2005

So did the messages all get sent correctly?
Darryl Rubarth

Web hosting and support for open-source solutions
www.exyst.com
drubarth
PL Master
 
Posts: 188
Joined: 10:14pm, Fri 17 Dec, 2004

Postby silverbytes » 2:36pm, Thu 21 Jul, 2005

No...
And don't understand that either. Eventlog say 1 message was sent at time if I understand right and it seems to be executing every 5 mins...

But I wonder if upgrading to 2.9 would help. It seems to have a domain throttle feature, anybody tried it?
silverbytes
PL Nut
 
Posts: 45
Joined: 4:13pm, Fri 18 Mar, 2005

Postby waznu » 9:24pm, Fri 22 Jul, 2005

I am still only getting the first 200 messages to process with this set up(followed from posts) and an inprocess message stuck and never moving. please help!?!

thank you, Deborah
waznu
phpLister
 
Posts: 14
Joined: 9:26pm, Tue 21 Jun, 2005

Postby jotazzu » 5:26pm, Fri 29 Jul, 2005

Hi,

I am working for some days on this topic too. PHPlist 2.8.12 has a problem with batch sending due to a tuning feature in processqueue.php which makes trouble in different situation. This feature might only be necessary if sending 5000+ emails in total.

Today or tomorrow I will post a summarize of cron handling with three workarounds to this problem and how to disable that feature.

Jochen
jotazzu
PL Master
 
Posts: 183
Joined: 6:31pm, Wed 13 Jul, 2005
Location: Hamburg, Germany

Postby drubarth » 6:52pm, Sat 30 Jul, 2005

phpList has incorporated my mod (didn't look to see if it was exactly the same). So make sure you're not duplicating.

If you hit a limit such as 200, then either your web host is limiting you, or the batch limit is set, or you're not running this using php-cli and you're hitting a timeout (first two are more likely).

Hope this helps give you some direction.

Cheers,
Darryl Rubarth

Web hosting and support for open-source solutions
www.exyst.com
drubarth
PL Master
 
Posts: 188
Joined: 10:14pm, Fri 17 Dec, 2004

Postby jotazzu » 7:19pm, Sat 30 Jul, 2005

Hi Darryl,

its not the throttling, it's the slicing of the MySQL query which causes some problems in 2.8.12.

In version 2.9.5 Michiel has disabled this slicing, so these problems will not occur in future.

I hope to get the post online within the next 2-3 hours.

Ciao,
Jochen
jotazzu
PL Master
 
Posts: 183
Joined: 6:31pm, Wed 13 Jul, 2005
Location: Hamburg, Germany

Postby drubarth » 7:50pm, Sat 30 Jul, 2005

Thanks for the info. Slicing must be one of the new features added that I haven't looked at yet. What is the concept of slicing?

Cheers,
Darryl Rubarth

Web hosting and support for open-source solutions
www.exyst.com
drubarth
PL Master
 
Posts: 188
Joined: 10:14pm, Fri 17 Dec, 2004

Postby jotazzu » 12:07am, Sun 31 Jul, 2005

I've my post published. There is also a separate post which explains the problem in detail.

Three and a half solutions for batch/cron processing

This is a quite long topic. Hope you find some interesting things. It was hard work to put it all together.

Jochen
jotazzu
PL Master
 
Posts: 183
Joined: 6:31pm, Wed 13 Jul, 2005
Location: Hamburg, Germany

Re: Throttled Sending Instead of Batches

Postby orangechicken » 4:52pm, Thu 04 Aug, 2005

drubarth wrote: # Throttle sending messages; indicates the number of milliseconds to pause
# between each message (millisecond is one-millionth of a second)
define("MAILQUEUE_PROCESSING_THROTTLE",300000);



Minor correction here. This number is microseconds, not milliseconds. A microsecond is a millionth of a second (while a millisecond is, of course, a thousandth).

Edit: And it pays to read the whole thread before pointing something out that's already been pointed out.

Now to actually get my install of phplist to actually *send* emails....[/i]
orangechicken
phpLister
 
Posts: 6
Joined: 4:38pm, Thu 04 Aug, 2005

Postby bumbum » 6:28pm, Fri 16 Sep, 2005

I think the throttle is a great feature. At first I found it a bit difficult to calculate the value I wanted so I've added in a couple of comment lines to help others find a suitable value.

# Throttle sending messages; indicates the number of microseconds to pause
# between each message (microsecond is one-millionth of a second)
# Use this formula to calculate a suitable value.
# ((Number of hours required to send emails x 3600000000)/Number of emails)
# Example: To send out 12000 e-mails in 48 hours the formula would be:
# ((48 x 3600000000)/12000) answer 14400000 microseconds to complete task
define("MAILQUEUE_PROCESSING_THROTTLE",14400000);


In the example shown, the user has 12000 subscribers on their mailing list and wishes to send the emails over a 48 hour period. The result is one e-mail being sent every 14400000 microseconds, or 1 email every 14.4 seconds.
This is particularly helpful if you've got a big list on a shared host server. It will help you to avoid hogging the server's resources and having your account suspended or terminated by the host administrators.
These are the settings I use on my own mailing list.
bumbum
phpLister
 
Posts: 6
Joined: 12:09am, Thu 25 Nov, 2004
Location: Dublin, Ireland

Postby jotazzu » 8:20pm, Fri 16 Sep, 2005

Why not use a slighlty changed patch?

Add to .../phpList/config/config.php:
Code: Select all
# Throttle sending messages; indicates the number of seconds to pause
# between each message (fractions are allowed, e.g. 0.3)
define("MAILQUEUE_PROCESSING_THROTTLE",0.5);


Add to .../phpList/admin/processqueue.php
Find the following at about line 420:
Code: Select all
$status = Sql_query("update {$tables['message']} set processed = processed +1 where id = $messageid");


Insert after this line:
Code: Select all
usleep((MAILQUEUE_PROCESSING_THROTTLE)*1000000);
jotazzu
PL Master
 
Posts: 183
Joined: 6:31pm, Wed 13 Jul, 2005
Location: Hamburg, Germany

Postby SamIAM » 5:36pm, Fri 23 Sep, 2005

I've got a question, what happens when the cron job excutes again during a cron job still processing ?

for example , its sending 2000 emails at 200 emails an hour, the cron job runs every 5 hours. Now the job will take more than 5 hour, what happens when the cron job excutes again ?

also, lets say during that time, I add another email job, what happens ?

sorry too many questions, i'm a newbie 8)
SamIAM
PL Nut
 
Posts: 22
Joined: 5:30pm, Fri 23 Sep, 2005
Location: Michigan

Postby jotazzu » 11:59pm, Fri 23 Sep, 2005

Hi,
PHPlist is able to handle such parallel invocations of processes to some extend. If a newly started process finds a running process it waits up for 20 seconds.
You will see messages like:
- A process for this page is already running and it was still alive xxx seconds ago
- Sleeping for 20 seconds, aborting will now quit

If still the previous process is running after 220s the new one finishes itself.
You will see a message like:
- We've been waiting too long, I guess the other script is still going ok

But when the previous process started more than 1200s ago and still is marked as running, the old one is killed without notice and the new one starts to process. This is the programming for v2.8.12 as you can find it at function getPageLock() in the file lib.php.

As you describe your scenario I assume that you are using throttling. The amount of 1000 emails you are sending in 5 hours is quite low. If your email is not bigger than 30kB any method which may be used with cron (PHP-cli, PHP-cgi, lynx, wget, GET, ...) should still be active after that time.

When the new cron job starts after 5 hours it will kill the previous job because the old one started more than 1200s ago. The new one takes over the processing of the queue. This is a completely new start and it continues to send the emails where the previous was stopped. If you added a new message meanwhile this is handled as if both messages are new. So there should be no difference to start both messages at the same time or to add the second later.

But don't use MAILQUEUE_BATCH_SIZE > 0 with your scenario. Only the first batch period will be executed before the process finishes.
jotazzu
PL Master
 
Posts: 183
Joined: 6:31pm, Wed 13 Jul, 2005
Location: Hamburg, Germany

Postby SamIAM » 3:24pm, Sat 24 Sep, 2005

Thanks for the great info.

I wonder, if i can change that 1200s number to make it much much larger, like 5 hours worth. Or maybe, i can make it where it will only start the new message send ONLY after the first one finished.

It takes me that long, becuase my host has a 200 emails/ hour.

Thanks again for the info.
SamIAM
PL Nut
 
Posts: 22
Joined: 5:30pm, Fri 23 Sep, 2005
Location: Michigan

Cli vs cgi mode

Postby robertp3 » 4:45pm, Sat 01 Oct, 2005

How can I determine which mode I am using when PHPlist is installed on a shared hosting service (in this case IpowerWeb).

I need to add the correct cron job to implement the throtte mod in topic below

http://www.phplist.com/forums/viewtopic.php?t=2142
robertp3
phpList newbie
 
Posts: 1
Joined: 4:38pm, Sat 01 Oct, 2005

PreviousNext

Return to Advanced Answers, Howtos, Tips & Tricks

Who is online

Users browsing this forum: No registered users and 1 guest