Cron job walkthrough

[evidenthost]

Welcome. We will be glad to help not only our hosting customers, buy everybody around.

[Balitas]

Hi evidenthost please help me in another related issue.

My phplist send no more than 350 emails to don't have problems with my server provider.

The problem is it sends and average of 300 emails per hour, and then finish (and not send all the emails to my 2600 subscribers)

What's happen? Here a piece of code of config.php I modified:

# define the amount of emails you want to send per period. If 0, batch processing
# is disabled and messages are sent out as fast as possible
define("MAILQUEUE_BATCH_SIZE",300);

# define the length of one batch processing period, in seconds (3600 is an hour)
define("MAILQUEUE_BATCH_PERIOD",3600);

# to avoid overloading the server that sends your email, you can add a little delay
# between messages that will spread the load of sending
# you will need to find a good value for your own server
# value is in seconds (or you can play with the autothrottle below)
define('MAILQUEUE_THROTTLE',10);

and other

define('USE_DOMAIN_THROTTLE',1);
define('DOMAIN_BATCH_SIZE',5);
define('DOMAIN_BATCH_PERIOD',60);

PLEASE HELP PHPLIST GURU

[Balitas]

My Buddy you don't answer me but I think I find the solution:

I change the parts in black:

# define the amount of emails you want to send per period. If 0, batch processing
# is disabled and messages are sent out as fast as possible
define("MAILQUEUE_BATCH_SIZE",5000);

# define the length of one batch processing period, in seconds (3600 is an hour)
define("MAILQUEUE_BATCH_PERIOD",60000);

# to avoid overloading the server that sends your email, you can add a little delay
# between messages that will spread the load of sending
# you will need to find a good value for your own server
# value is in seconds (or you can play with the autothrottle below)
define('MAILQUEUE_THROTTLE',10);

and other

define('USE_DOMAIN_THROTTLE',1);
define('DOMAIN_BATCH_SIZE',5);
define('DOMAIN_BATCH_PERIOD',60);

What does it mean? I have 2600 subscribers, so with this will send 5000 subscribers in 60000 seconds, but with domain thorttle it control to send 5 messages per minutes, so I don't pass the 300-350 email limit per hour and never stops (because I have less subscribers than the limit I setup)

Waiting your comments.

[evidenthost]

The first part with mailqueue settings does look correct and should work. (300 emails per hour)

define("MAILQUEUE_BATCH_SIZE",300);
# define the length of one batch processing period, in seconds (3600 is an hour)
define("MAILQUEUE_BATCH_PERIOD",3600);

I am not familiar with domain throttling. You may want to try disabling that setting and try again.

[H2B2]

Evidenthost, Balitas,

The problem is it sends and average of 300 emails per hour, and then finish (and not send all the emails to my 2600 subscribers)

If the server is running php-cgi, there might be a problem with timeouts (see: http://docs.phplist.com/PhpInterfacesInfo and http://forums.phplist.com/viewtopic.php?p=11369#11369 ).
In that case you could try using the "reload" parameter in the cron command, as described by jotazzu in this example:

Cron job with batch processing (PHP-cgi)

Code: Select all

#=================================================================
# CRON JOB with batch mode on - (PHP-cgi)
# crontab for processing all messages with status 'queued' or 'inprocess'.
# Processing starts at 17 min after midnight and continues for 3 hours with
# a reload after one hour. The hour of the cron schedule as well as the
# reload count has to be increased by 1 for each crontab entry.
# Maximal MAILQUEUE_BATCH_SIZE * NUMBER_OF_CONTRAB_ENTRIES emails will be send.
# In this example is NUMBER_OF_CRONTAB_ENTRIES=4.
#=================================================================
17 0 * * *   wget -q -O - 'http://www.mydomain.com/lists/admin/index.php?page=processqueue&login=myusername&password=mypassword&reload=0' >/dev/null
17 1 * * *   wget -q -O - 'http://www.mydomain.com/lists/admin/index.php?page=processqueue&login=myusername&password=mypassword&reload=1' >/dev/null
17 2 * * *   wget -q -O - 'http://www.mydomain.com/lists/admin/index.php?page=processqueue&login=myusername&password=mypassword&reload=2' >/dev/null
17 3 * * *   wget -q -O - 'http://www.mydomain.com/lists/admin/index.php?page=processqueue&login=myusername&password=mypassword&reload=3' >/dev/null

This cron job uses wget and therefore you have to set one entry for each increasing reload. The number of entries multiplied with MAILQUEUE_BATCH_SIZE must be higher than the number of subscribers to that list (assuming the message will be send only to one list). You have to add additional entries when the list grows. To calculate the time between the schedules see at how to tune MAILQUEUE_BATCH_SIZE. (Each schedule has to be in one line.)

Source: http://docs.phplist.com/CronJobExamples

[Balitas]

I don't understand please explain as easy way like evidenthost

[H2B2]

1. Determining your settings in config.php
Your config.php settings need to reflect these requirements:
- You have a total of 2600 messages to send
- You have a mail limit of 300 messages per hour
- You want to space the messages to avoid overloading the server

So, these could be your settings in config.php:
define("MAILQUEUE_BATCH_SIZE",300);
define("MAILQUEUE_BATCH_PERIOD",3600);
define('MAILQUEUE_THROTTLE',4);


2. Testing the cURL command on the commandline
So, assuming your server is running PHP-cgi, you will need to reload the process queue command for each batch.
I'm not familiar with cURL, but I think it's worth a try to see if the you can reload the process queue by adding the parameter reload=n to the command, where n is the number of the batch to reload starting with 0, e.g. &reload=0, &reload=1, &reload=3, etc.

So, on the commandline you could type these commands, while making sure you wait one hour between commands.

Code: Select all

curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=0
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=1
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=2
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=3
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=4
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=5
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=6
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=7
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=8


Note: Obviously, you only need to type the first two commands to test if reloading works correctly, assuming you did not yet send batches 0 and 1:

Code: Select all

curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=0
curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=1

3. Creating the cron job
If this works you can put these command in your crontab file:

Code: Select all

#==============================================================================
#================ CRON JOB with batch mode on - (PHP-cgi) =====================
#
# crontab for processing all messages with status 'queued' or 'inprocess'.
# Processing starts at 17 min after midnight and continues for 9 hours with
# a reload after one hour.
#
# The hour of the cron schedule and the reload count has to be increased by 1
# for each crontab entry, i.e. for each batch you need to process.
#
# Settings in config.php:
# define("MAILQUEUE_BATCH_SIZE",300);
# define("MAILQUEUE_BATCH_PERIOD",3600);
# define('MAILQUEUE_THROTTLE',4);
#
#==============================================================================
17 0 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=0 >/dev/null
17 1 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=1 >/dev/null
17 2 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=2 >/dev/null
17 3 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=3 >/dev/null
17 4 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=4 >/dev/null
17 5 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=5 >/dev/null
17 6 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=6 >/dev/null
17 7 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=7 >/dev/null
17 8 * * *  curl -s http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=8 >/dev/null
 

[Balitas]

Thanks for you help H2B2 now is more clearly for me.

Two more questions:
FIRST QUESTION: If I want this 24 hours day I need to put

17 8 * * * curl -s http://www.example.com/lists/admin/?page ... D&reload=8 >/dev/null

until

17 23 * * * curl -s http://www.example.com/lists/admin/?page ... D&reload=8 >/dev/null

Right?

SECOND QUESTION:
I was working fine with help of evidenthost with wget, is the same of curl?

Regards.

Regards.

[H2B2]

SECOND QUESTION:
I was working fine with help of evidenthost with wget, is the same of curl?

I was under the (wrong) assumption that you had succefully tested curl. I probably misread one of the previous posts.

So, replace all occurences of curl -s with wget -q -O - in the above cron schedule as well as the test commands for the commandline.

FIRST QUESTION: If I want this 24 hours day I need to put

17 8 * * * curl -s http://www.example.com/lists/admin/?page ... D&reload=8 >/dev/null

until

17 23 * * * curl -s http://www.example.com/lists/admin/?page ... D&reload=8 >/dev/null

Right?

Well, that depends why you want it to go on sending batches in sequence for 24 hours. If you are sending a total message load of 7200 messages (300 messages x 24 hours) this would be alright. But I'm not sure that's you want to do. It might help if you'd explain that point.

[Balitas]

H2B2 I have one more question.

I noticed every line you put reload=

What does it mean?

For example if I use:


17 23 * * * curl -s http://www.example.com/lists/admin/?page ... RD&reload=

The reload= must be the same of the hour I selected?

Regards.

[H2B2]

The reload parameter will load the next batch on a php-cgi system. So yes, in this schedule the reload number should be the same as the hour, as long as it takes to send that particular message load.

However, if all batches have been sent (i.e., after the last message of the 2600 messages has been sent), you should stop reloading (non-existent) batches. From than on you could use the same command, but without the reload parameter, as long as the number of emails you send is lower than a batch size of 300. This would take care of sending the re-queued message to new subscribers.

I understand you are using wget (not curl), so you could test this crontab:

Code: Select all

#==============================================================================
#================ CRON JOB with batch mode on - (PHP-cgi) =====================
#
# crontab for processing all messages with status 'queued' or 'inprocess'.
# Processing starts at 17 min after midnight and continues for 9 hours with
# a reload after one hour.
#
# The hour of the cron schedule and the reload count has to be increased by 1
# for each crontab entry, i.e. for each batch you need to process.
#
# a new sequence of batches does not need a reload count as long as their number
# is lower than the batch size.
#
# Settings in config.php:
# define("MAILQUEUE_BATCH_SIZE",300);
# define("MAILQUEUE_BATCH_PERIOD",3600);
# define('MAILQUEUE_THROTTLE',4);
#
#==============================================================================
17 0 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=0 >/dev/null
17 1 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=1 >/dev/null
17 2 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=2 >/dev/null
17 3 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=3 >/dev/null
17 4 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=4 >/dev/null
17 5 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=5 >/dev/null
17 6 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=6 >/dev/null
17 7 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=7 >/dev/null
17 8 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD&reload=8 >/dev/null
17 9 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD >/dev/null
17 10 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD >/dev/null
...... etc ....
17 23 * * *  wget -q -O - http://www.domain.com/lists/admin/?page=processqueue&login=USERNAME&password=PASSWORD >/dev/null

[Balitas]

One more question before proceed, in case or 0 to 9 hour, Ex: setup 1am I need to put 01 or 1? or its the same?

[H2B2]

just 1 is OK.

[Balitas]

H2B2 I follow the steps you mentioned.

If I use in config.php:
define("MAILQUEUE_BATCH_SIZE",300);
define("MAILQUEUE_BATCH_PERIOD",3600);
define('MAILQUEUE_THROTTLE',4);

But always send the first 300 messages and then stops.

Another form to setup cronjobs?

Regards.

[Balitas]

But only works if

# define the amount of emails you want to send per period. If 0, batch processing
# is disabled and messages are sent out as fast as possible
define("MAILQUEUE_BATCH_SIZE",5000);

# define the length of one batch processing period, in seconds (3600 is an hour)
define("MAILQUEUE_BATCH_PERIOD",60000);

# to avoid overloading the server that sends your email, you can add a little delay
# between messages that will spread the load of sending
# you will need to find a good value for your own server
# value is in seconds (or you can play with the autothrottle below)
define('MAILQUEUE_THROTTLE',10);

What does it mean? I have 2600 subscribers, so with this will send 5000 subscribers in 60000 seconds, but with domain thorttle it control to send 5 messages per minutes, so I don't pass the 300-350 email limit per hour and never stops (because I have less subscribers than the limit I setup)