BATCH PROCESSING HANGING? Read This

Solutions for other advanced phplisters

BATCH PROCESSING HANGING? Read This

Postby RationalRabbit » 7:51am, Wed 02 Sep, 2009

I have been working with a mailing list of 150,000 names, and struggling with very intermittant progress with PHP. I have been having to restart the queue in order to keep the batches running.

Here is what I discovered:
If I make sure the number of emails (MAILQUEUE_BATCH_SIZE in config.php) is set to an obtainable amount within the allotted time period (MAILQUEUE_BATCH_PERIOD) the program happily requeues the next batch. However, if not all of the e-mails get processed within the alloted time period, PHPList will hang.

Right now, I have the batch queue set to process 200 e-mails in a 10 minute period, which seems to work well. Not optimal, but I don't need to babysit it. The only thing that bothers me is the batch runs, then pauses for 10 minutes, so I am only sending out 200 e-mails per 20 minutes. Not good! But there does not seem to be a setting to change that.
RationalRabbit
PL Nut
 
Posts: 15
Joined: 8:40am, Thu 23 Aug, 2007
Location: Seattle, Washington

Re: BATCH PROCESSING HANGING? Read This

Postby Heritage » 1:57pm, Wed 02 Sep, 2009

I believe your issue is because you did not declare the refetch time out.

define('REMOTE_URL_REFETCH_TIMEOUT',3600);

Should be:

define('REMOTE_URL_REFETCH_TIMEOUT',600);

Or 700 since you might want some time in between.

I prefer to use:

define('USE_DOMAIN_THROTTLE',1);
define('DOMAIN_BATCH_SIZE',4);
define('DOMAIN_BATCH_PERIOD',120);

MAILQUEUE_BATCH_SIZE is really for capping your hourly or daily limit with your host.

For best results you should cap domains and that is the domain throttle. See Yahoo only wants 5 emails per session.

If you set the REMOTE_URL_REFETCH_TIMEOUT timeout you should be good to go. That is if you are keeping your web browser open. Even then it can lose connection after a day or so.

The ultimate solution is to run a cron job.

Or you can run a php script that refreshes every ten minutes pulling your process queue in an iframe (note you will need to all the login user and pass within the string in order to get it to stay logged in).

lists/admin/?page=processqueue&login=admin&password=phplist
Heritage
Moderator
 
Posts: 223
Joined: 3:25pm, Fri 23 Jun, 2006
Location: 800 720-7301

Re: BATCH PROCESSING HANGING? Read This

Postby RationalRabbit » 3:33pm, Wed 02 Sep, 2009

Thanks!

I was not aware of the REMOTE_URL_REFETCH_TIMEOUT setting!

I am on my own server, so I am not concerned with host limits. However, if I run the queue straight it trickles out e-mail and may even hang.

I've tried every cron string I've found in this forum, to no avail. Cron is running okay. The string just isn't processing the queue for some reason.

Or you can run a php script that refreshes every ten minutes pulling your process queue in an iframe


Great idea!

Thanks. Your response has been most helpful!
RationalRabbit
PL Nut
 
Posts: 15
Joined: 8:40am, Thu 23 Aug, 2007
Location: Seattle, Washington

Re: BATCH PROCESSING HANGING? Read This

Postby RationalRabbit » 9:53pm, Wed 02 Sep, 2009

Humm -

REMOTE_URL_REFETCH_TIMEOUT

I thought the name of this setting seemed a little strange for the function you described.
the config.php file states that this setting has to do with (as its name implies) fetching a remote web page.

It does not state anything as far as a relationship to processing a normal file.
RationalRabbit
PL Nut
 
Posts: 15
Joined: 8:40am, Thu 23 Aug, 2007
Location: Seattle, Washington

Re: BATCH PROCESSING HANGING? Read This

Postby ThorstenAlbrecht » 1:34pm, Thu 12 Nov, 2009

RationalRabbit wrote:Right now, I have the batch queue set to process 200 e-mails in a 10 minute period, which seems to work well. Not optimal, but I don't need to babysit it. The only thing that bothers me is the batch runs, then pauses for 10 minutes, so I am only sending out 200 e-mails per 20 minutes. Not good! But there does not seem to be a setting to change that.


There is a setting to change that. It's called: MAILQUEUE_BATCH_PERIOD.

Please try the following settings for batch processing which works for me using the web interface:

define("MAILQUEUE_BATCH_SIZE",100);
define("MAILQUEUE_BATCH_PERIOD",1);
define('MAILQUEUE_THROTTLE',0);

This means: phpList will send 100 mails per batch, waits for 1s before reloading the webinterface. No waiting between sending the mails.

BTW The documentation of MAILQUEUE_BATCH_PERIOD is completely wrong: "define the length of one batch processing period, in seconds (3600 is an hour)". I tried for hours before I found out that MAILQUEUE_BATCH_PERIOD has a completely diffent meaning and that using a very small value would turn on the reload process of the web interface to cope with any timeout issues of the hosting provider. The default value of 3600 makes no sense at all.

Thorsten
ThorstenAlbrecht
PL Nut
 
Posts: 44
Joined: 4:08pm, Thu 06 Aug, 2009

Re: BATCH PROCESSING HANGING? Read This

Postby DaveThePaintingGuy » 6:21pm, Sun 20 Dec, 2009

Gentlemen, this stuff makes my head spin.

My ISP (Lunarpages) keeps lowering and changing outgoing mail limits, and recently changed to

400 outgoing per hour, and only 20 per minute.

Current Settings, which may be causing a queue processing hang:
define("MAILQUEUE_BATCH_SIZE",360);
define("MAILQUEUE_BATCH_PERIOD",3600);
define('MAILQUEUE_THROTTLE',20);


and the one I do not understand:
define('REMOTE_URL_REFETCH_TIMEOUT',3600);

What would you suggest I try?

And would you describe the expected result in "plain English", please, for example:
"These settings that I recommend will send x-e-mails per minute, then wait for y-seconds, then send 20 more, and stop when 400 are reached, waiting for z-seconds/minutes to pass. "

Thanks
DaveThePaintingGuy
phpList newbie
 
Posts: 2
Joined: 6:04pm, Sun 20 Dec, 2009

Re: BATCH PROCESSING HANGING? Read This

Postby shomam » 2:45am, Sun 01 May, 2011

I am on my own server, so I am not concerned with host limits. However, if I run the queue straight it trickles out e-mail and may even hang.
shomam
phpList newbie
 
Posts: 1
Joined: 2:43am, Sun 01 May, 2011

Re: BATCH PROCESSING HANGING? Read This

Postby ThorstenAlbrecht » 9:03am, Mon 02 May, 2011

What would you suggest I try?



define("MAILQUEUE_BATCH_SIZE",100);
define("MAILQUEUE_BATCH_PERIOD",1);
define('MAILQUEUE_THROTTLE',0);


These settings that I recommend will send 100 e-mails in some minutes, then wait for 1 seconds, then send 100 more, and stop when everthing is finished.

This works for me in a simple browser which reloads every 100 mails.

Thorsten
ThorstenAlbrecht
PL Nut
 
Posts: 44
Joined: 4:08pm, Thu 06 Aug, 2009

Re: BATCH PROCESSING HANGING? Read This

Postby ThorstenAlbrecht » 9:04am, Mon 02 May, 2011

Just to add: Please have a look to

http://mantis.phplist.com/view.php?id=15365

where the problem is described in detail.

Thorsten
ThorstenAlbrecht
PL Nut
 
Posts: 44
Joined: 4:08pm, Thu 06 Aug, 2009


Return to Advanced Answers, Howtos, Tips & Tricks

Who is online

Users browsing this forum: No registered users and 2 guests

cron