Cron job walkthrough

Solutions for other advanced phplisters

Postby evidenthost » 7:22am, Sun 19 Nov, 2006

Welcome. We will be glad to help not only our hosting customers, buy everybody around.
evidenthost
PL Nut
 
Posts: 27
Joined: 3:58am, Fri 17 Nov, 2006
Location: Canada - West Vancouver

Postby Balitas » 12:14am, Tue 21 Nov, 2006

Hi evidenthost please help me in another related issue.

My phplist send no more than 350 emails to don't have problems with my server provider.

The problem is it sends and average of 300 emails per hour, and then finish (and not send all the emails to my 2600 subscribers)

What's happen? Here a piece of code of config.php I modified:

# define the amount of emails you want to send per period. If 0, batch processing
# is disabled and messages are sent out as fast as possible
define("MAILQUEUE_BATCH_SIZE",300);

# define the length of one batch processing period, in seconds (3600 is an hour)
define("MAILQUEUE_BATCH_PERIOD",3600);

# to avoid overloading the server that sends your email, you can add a little delay
# between messages that will spread the load of sending
# you will need to find a good value for your own server
# value is in seconds (or you can play with the autothrottle below)
define('MAILQUEUE_THROTTLE',10);

and other

define('USE_DOMAIN_THROTTLE',1);
define('DOMAIN_BATCH_SIZE',5);
define('DOMAIN_BATCH_PERIOD',60);

PLEASE HELP PHPLIST GURU :)
Need a website? Enter to www.balitas.com
Balitas
PL Geek
 
Posts: 82
Joined: 2:17am, Mon 13 Feb, 2006

Postby Balitas » 7:34am, Tue 21 Nov, 2006

My Buddy you don't answer me :( but I think I find the solution:

I change the parts in black:

# define the amount of emails you want to send per period. If 0, batch processing
# is disabled and messages are sent out as fast as possible
define("MAILQUEUE_BATCH_SIZE",5000);

# define the length of one batch processing period, in seconds (3600 is an hour)
define("MAILQUEUE_BATCH_PERIOD",60000);

# to avoid overloading the server that sends your email, you can add a little delay
# between messages that will spread the load of sending
# you will need to find a good value for your own server
# value is in seconds (or you can play with the autothrottle below)
define('MAILQUEUE_THROTTLE',10);

and other

define('USE_DOMAIN_THROTTLE',1);
define('DOMAIN_BATCH_SIZE',5);
define('DOMAIN_BATCH_PERIOD',60);

What does it mean? I have 2600 subscribers, so with this will send 5000 subscribers in 60000 seconds, but with domain thorttle it control to send 5 messages per minutes, so I don't pass the 300-350 email limit per hour and never stops (because I have less subscribers than the limit I setup)

Waiting your comments.
Need a website? Enter to www.balitas.com
Balitas
PL Geek
 
Posts: 82
Joined: 2:17am, Mon 13 Feb, 2006

Postby evidenthost » 10:24am, Wed 22 Nov, 2006

The first part with mailqueue settings does look correct and should work. (300 emails per hour)

define("MAILQUEUE_BATCH_SIZE",300);
# define the length of one batch processing period, in seconds (3600 is an hour)
define("MAILQUEUE_BATCH_PERIOD",3600);


I am not familiar with domain throttling. You may want to try disabling that setting and try again.
evidenthost
PL Nut
 
Posts: 27
Joined: 3:58am, Fri 17 Nov, 2006
Location: Canada - West Vancouver

Postby H2B2 » 5:56pm, Wed 22 Nov, 2006

Evidenthost, Balitas,

Balitas wrote:The problem is it sends and average of 300 emails per hour, and then finish (and not send all the emails to my 2600 subscribers)

If the server is running php-cgi, there might be a problem with timeouts (see: http://docs.phplist.com/PhpInterfacesInfo and http://forums.phplist.com/viewtopic.php?p=11369#11369 ).
In that case you could try using the "reload" parameter in the cron command, as described by jotazzu in this example:
Cron job with batch processing (PHP-cgi)

Code: Select all
#=================================================================
# CRON JOB with batch mode on - (PHP-cgi)
# crontab for processing all messages with status 'queued' or 'inprocess'.
# Processing starts at 17 min after midnight and continues for 3 hours with
# a reload after one hour. The hour of the cron schedule as well as the
# reload count has to be increased by 1 for each crontab entry.
# Maximal MAILQUEUE_BATCH_SIZE * NUMBER_OF_CONTRAB_ENTRIES emails will be send.
# In this example is NUMBER_OF_CRONTAB_ENTRIES=4.
#=================================================================
17 0 * * *   wget -q -O - 'http://www.mydomain.com/lists/admin/index.php?page=processqueue&login=myusername&password=mypassword&reload=0' >/dev/null
17 1 * * *   wget -q -O - 'http://www.mydomain.com/lists/admin/index.php?page=processqueue&login=myusername&password=mypassword&reload=1' >/dev/null
17 2 * * *   wget -q -O - 'http://www.mydomain.com/lists/admin/index.php?page=processqueue&login=myusername&password=mypassword&reload=2' >/dev/null
17 3 * * *   wget -q -O - 'http://www.mydomain.com/lists/admin/index.php?page=processqueue&login=myusername&password=mypassword&reload=3' >/dev/null


This cron job uses wget and therefore you have to set one entry for each increasing reload. The number of entries multiplied with MAILQUEUE_BATCH_SIZE must be higher than the number of subscribers to that list (assuming the message will be send only to one list). You have to add additional entries when the list grows. To calculate the time between the schedules see at how to tune MAILQUEUE_BATCH_SIZE. (Each schedule has to be in one line.)

Source: http://docs.phplist.com/CronJobExamples
H2B2
Moderator
 
Posts: 7188
Joined: 1:51am, Wed 15 Mar, 2006

Postby Balitas » 9:15am, Thu 23 Nov, 2006

I don't understand please explain as easy way like evidenthost
Need a website? Enter to www.balitas.com
Balitas
PL Geek
 
Posts: 82
Joined: 2:17am, Mon 13 Feb, 2006

Postby H2B2 » 5:47pm, Thu 23 Nov, 2006

1. Determining your settings in config.php
Your config.php settings need to reflect these requirements:
- You have a total of 2600 messages to send
- You have a mail limit of 300 messages per hour
- You want to space the messages to avoid overloading the server

So, these could be your settings in config.php:
define("MAILQUEUE_BATCH_SIZE",300);
define("MAILQUEUE_BATCH_PERIOD",3600);
define('MAILQUEUE_THROTTLE',4);


2. Testing the cURL command on the commandline
So, assuming your server is running PHP-cgi, you will need to reload the process queue command for each batch.
I'm not familiar with cURL, but I think it's worth a try to see if the you can reload the process queue by adding the parameter reload=n to the command, where n is the number of the batch to reload starting with 0, e.g. &reload=0, &reload=1, &reload=3, etc.

So, on the commandline you could type these commands, while making sure you wait one hour between commands.
Code: Select all
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=0
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=1
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=2
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=3
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=4
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=5
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=6
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=7
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=8

Note
: Obviously, you only need to type the first two commands to test if reloading works correctly, assuming you did not yet send batches 0 and 1:
Code: Select all
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=0
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=1


3. Creating the cron job
If this works you can put these command in your crontab file:
Code: Select all
#==============================================================================
#================ CRON JOB with batch mode on - (PHP-cgi) =====================
#
# crontab for processing all messages with status 'queued' or 'inprocess'.
# Processing starts at 17 min after midnight and continues for 9 hours with
# a reload after one hour.
#
# The hour of the cron schedule and the reload count has to be increased by 1
# for each crontab entry, i.e. for each batch you need to process.
#
# Settings in config.php:
# define("MAILQUEUE_BATCH_SIZE",300);
# define("MAILQUEUE_BATCH_PERIOD",3600);
# define('MAILQUEUE_THROTTLE',4);
#
#==============================================================================
17 0 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=0 >/dev/null
17 1 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=1 >/dev/null
17 2 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=2 >/dev/null
17 3 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=3 >/dev/null
17 4 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=4 >/dev/null
17 5 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=5 >/dev/null
17 6 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=6 >/dev/null
17 7 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=7 >/dev/null
17 8 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=8 >/dev/null
 
H2B2
Moderator
 
Posts: 7188
Joined: 1:51am, Wed 15 Mar, 2006

Postby Balitas » 6:27pm, Thu 23 Nov, 2006

Thanks for you help H2B2 now is more clearly for me.

Two more questions:
FIRST QUESTION: If I want this 24 hours day I need to put

17 8 * * * curl -s http://www.example.com/lists/admin/?page ... D&reload=8 >/dev/null

until

17 23 * * * curl -s http://www.example.com/lists/admin/?page ... D&reload=8 >/dev/null

Right?

SECOND QUESTION:
I was working fine with help of evidenthost with wget, is the same of curl?

Regards.

Regards.
Need a website? Enter to www.balitas.com
Balitas
PL Geek
 
Posts: 82
Joined: 2:17am, Mon 13 Feb, 2006

Postby H2B2 » 8:16pm, Thu 23 Nov, 2006

Balitas wrote:SECOND QUESTION:
I was working fine with help of evidenthost with wget, is the same of curl?
I was under the (wrong) assumption that you had succefully tested curl. I probably misread one of the previous posts.

So, replace all occurences of curl -s with wget -q -O - in the above cron schedule as well as the test commands for the commandline.



Balitas wrote:FIRST QUESTION: If I want this 24 hours day I need to put

17 8 * * * curl -s http://www.example.com/lists/admin/?page ... D&reload=8 >/dev/null

until

17 23 * * * curl -s http://www.example.com/lists/admin/?page ... D&reload=8 >/dev/null

Right?
Well, that depends why you want it to go on sending batches in sequence for 24 hours. If you are sending a total message load of 7200 messages (300 messages x 24 hours) this would be alright. But I'm not sure that's you want to do. It might help if you'd explain that point.
H2B2
Moderator
 
Posts: 7188
Joined: 1:51am, Wed 15 Mar, 2006

Postby Balitas » 4:42am, Tue 05 Dec, 2006

H2B2 I have one more question.

I noticed every line you put reload=

What does it mean?

For example if I use:


17 23 * * * curl -s http://www.example.com/lists/admin/?page ... RD&reload=

The reload= must be the same of the hour I selected?

Regards.
Need a website? Enter to www.balitas.com
Balitas
PL Geek
 
Posts: 82
Joined: 2:17am, Mon 13 Feb, 2006

Postby H2B2 » 5:27am, Tue 05 Dec, 2006

The reload parameter will load the next batch on a php-cgi system. So yes, in this schedule the reload number should be the same as the hour, as long as it takes to send that particular message load.

However, if all batches have been sent (i.e., after the last message of the 2600 messages has been sent), you should stop reloading (non-existent) batches. From than on you could use the same command, but without the reload parameter, as long as the number of emails you send is lower than a batch size of 300. This would take care of sending the re-queued message to new subscribers.

I understand you are using wget (not curl), so you could test this crontab:

Code: Select all
#==============================================================================
#================ CRON JOB with batch mode on - (PHP-cgi) =====================
#
# crontab for processing all messages with status 'queued' or 'inprocess'.
# Processing starts at 17 min after midnight and continues for 9 hours with
# a reload after one hour.
#
# The hour of the cron schedule and the reload count has to be increased by 1
# for each crontab entry, i.e. for each batch you need to process.
#
# a new sequence of batches does not need a reload count as long as their number
# is lower than the batch size.
#
# Settings in config.php:
# define("MAILQUEUE_BATCH_SIZE",300);
# define("MAILQUEUE_BATCH_PERIOD",3600);
# define('MAILQUEUE_THROTTLE',4);
#
#==============================================================================
17 0 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=0 >/dev/null
17 1 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=1 >/dev/null
17 2 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=2 >/dev/null
17 3 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=3 >/dev/null
17 4 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=4 >/dev/null
17 5 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=5 >/dev/null
17 6 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=6 >/dev/null
17 7 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=7 >/dev/null
17 8 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=8 >/dev/null
17 9 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD >/dev/null
17 10 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD >/dev/null
...... etc ....
17 23 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD >/dev/null
H2B2
Moderator
 
Posts: 7188
Joined: 1:51am, Wed 15 Mar, 2006

Postby Balitas » 6:27am, Tue 05 Dec, 2006

One more question before proceed, in case or 0 to 9 hour, Ex: setup 1am I need to put 01 or 1? or its the same?
Need a website? Enter to www.balitas.com
Balitas
PL Geek
 
Posts: 82
Joined: 2:17am, Mon 13 Feb, 2006

Postby H2B2 » 6:50am, Tue 05 Dec, 2006

just 1 is OK.
H2B2
Moderator
 
Posts: 7188
Joined: 1:51am, Wed 15 Mar, 2006

Postby Balitas » 4:41pm, Wed 06 Dec, 2006

H2B2 I follow the steps you mentioned.

If I use in config.php:
define("MAILQUEUE_BATCH_SIZE",300);
define("MAILQUEUE_BATCH_PERIOD",3600);
define('MAILQUEUE_THROTTLE',4);

But always send the first 300 messages and then stops.

Another form to setup cronjobs?

Regards.
Need a website? Enter to www.balitas.com
Balitas
PL Geek
 
Posts: 82
Joined: 2:17am, Mon 13 Feb, 2006

Postby Balitas » 5:03pm, Wed 06 Dec, 2006

But only works if

# define the amount of emails you want to send per period. If 0, batch processing
# is disabled and messages are sent out as fast as possible
define("MAILQUEUE_BATCH_SIZE",5000);

# define the length of one batch processing period, in seconds (3600 is an hour)
define("MAILQUEUE_BATCH_PERIOD",60000);

# to avoid overloading the server that sends your email, you can add a little delay
# between messages that will spread the load of sending
# you will need to find a good value for your own server
# value is in seconds (or you can play with the autothrottle below)
define('MAILQUEUE_THROTTLE',10);

What does it mean? I have 2600 subscribers, so with this will send 5000 subscribers in 60000 seconds, but with domain thorttle it control to send 5 messages per minutes, so I don't pass the 300-350 email limit per hour and never stops (because I have less subscribers than the limit I setup)
Need a website? Enter to www.balitas.com
Balitas
PL Geek
 
Posts: 82
Joined: 2:17am, Mon 13 Feb, 2006

PreviousNext

Return to Advanced Answers, Howtos, Tips & Tricks

Who is online

Users browsing this forum: No registered users and 1 guest